Page 1 of 1
error:FAULTY project:9431
Posted: Sat Dec 23, 2017 10:35 am
by SteveWillis
I have had a series of faulty project errors but only on one of my three machines, always on the same GPU. However somehow I managed to fail to notice that it's always the same project.
This is the latest
Code: Select all
2017-12-23:09:46:24:WU04:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-23:09:46:29:WARNING:WU04:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-23:09:46:29:WU04:FS00:Sending unit results: id:04 state:SEND error:FAULTY project:9431 run:74 clone:1 gen:721 core:0x21 unit:0x0000035eab436c9d586fdd344eef242b
Code: Select all
*********************** Log Started 2017-12-22T20:59:43Z ***********************
2017-12-22:20:59:43:******************* Folding@home Client ********************
2017-12-22:20:59:43: Website: http://folding.stanford.edu/
2017-12-22:20:59:43: Copyright: (c) 2009-2016 Stanford University
2017-12-22:20:59:43: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
2017-12-22:20:59:43: Args: --child --lifeline 1582 /etc/fahclient/config.xml --run-as
2017-12-22:20:59:43: fahclient --pid-file=/var/run/fahclient.pid --daemon
2017-12-22:20:59:43: Config: /etc/fahclient/config.xml
2017-12-22:20:59:43:************************** Build ***************************
2017-12-22:20:59:43: Version: 7.4.16
2017-12-22:20:59:43: Date: Jan 6 2017
2017-12-22:20:59:43: Time: 08:08:33
2017-12-22:20:59:43: Repository: Git
2017-12-22:20:59:43: Revision: e12187cbb0bd6937c067b9749af011374563b7b9
2017-12-22:20:59:43: Branch: master
2017-12-22:20:59:43: Compiler: GNU 4.9.2
2017-12-22:20:59:43: Options: -std=gnu++98 -O3 -funroll-loops -ffast-math -mfpmath=sse
2017-12-22:20:59:43: -fno-unsafe-math-optimizations -msse2
2017-12-22:20:59:43: Platform: linux2 4.8.0-2-amd64
2017-12-22:20:59:43: Bits: 64
2017-12-22:20:59:43: Mode: Release
2017-12-22:20:59:43:************************** System **************************
2017-12-22:20:59:43: CPU: AMD FX(tm)-6300 Six-Core Processor
2017-12-22:20:59:43: CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
2017-12-22:20:59:43: CPUs: 6
2017-12-22:20:59:43: Memory: 11.63GiB
2017-12-22:20:59:43: Free Memory: 11.02GiB
2017-12-22:20:59:43: Threads: POSIX_THREADS
2017-12-22:20:59:43: OS Version: 4.4
2017-12-22:20:59:43: Has Battery: false
2017-12-22:20:59:43: On Battery: false
2017-12-22:20:59:43: UTC Offset: -6
2017-12-22:20:59:43: PID: 1584
2017-12-22:20:59:43: CWD: /var/lib/fahclient
2017-12-22:20:59:43: OS: Linux 4.4.0-53-generic x86_64
2017-12-22:20:59:43: OS Arch: AMD64
2017-12-22:20:59:43: GPUs: 4
2017-12-22:20:59:43: GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1080] 8873
2017-12-22:20:59:43: GPU 1: Bus:8 Slot:0 Func:0 NVIDIA:7 GP102 [GeForce GTX 1080 Ti] 11380
2017-12-22:20:59:43: GPU 2: Bus:9 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1080] 8873
2017-12-22:20:59:43: GPU 3: Bus:10 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1080] 8873
2017-12-22:20:59:43:CUDA Device 0: Platform:0 Device:0 Bus:8 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:20:59:43:CUDA Device 1: Platform:0 Device:1 Bus:1 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:20:59:43:CUDA Device 2: Platform:0 Device:2 Bus:9 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:20:59:43:CUDA Device 3: Platform:0 Device:3 Bus:10 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:20:59:43: OpenCL: Not detected: Failed to open dynamic library 'libOpenCL.so':
2017-12-22:20:59:43: libOpenCL.so: cannot open shared object file: No such file or
2017-12-22:20:59:43: directory
2017-12-22:20:59:43:************************************************************
2017-12-22:20:59:43:<config>
2017-12-22:20:59:43: <!-- Client Control -->
2017-12-22:20:59:43: <fold-anon v='true'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43: <!-- Folding Slot Configuration -->
2017-12-22:20:59:43: <gpu v='false'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43: <!-- Logging -->
2017-12-22:20:59:43: <log-date v='true'/>
2017-12-22:20:59:43: <log-rotate-max v='128'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43: <!-- Network -->
2017-12-22:20:59:43: <proxy v=':8080'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43: <!-- Slot Control -->
2017-12-22:20:59:43: <power v='full'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43: <!-- User Information -->
2017-12-22:20:59:43: <passkey v='********************************'/>
2017-12-22:20:59:43: <team v='224497'/>
2017-12-22:20:59:43: <user v='DarthMouse_ALL_1GD5nCZbh7gNo1SESPLT24xEd2Jsu4rTP9'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43: <!-- Work Unit Control -->
2017-12-22:20:59:43: <next-unit-percentage v='100'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43: <!-- Folding Slots -->
2017-12-22:20:59:43: <slot id='1' type='GPU'>
2017-12-22:20:59:43: <cuda-index v='0'/>
2017-12-22:20:59:43: <gpu-index v='0'/>
2017-12-22:20:59:43: <opencl-index v='0'/>
2017-12-22:20:59:43: </slot>
2017-12-22:20:59:43: <slot id='0' type='GPU'>
2017-12-22:20:59:43: <cuda-index v='1'/>
2017-12-22:20:59:43: <gpu-index v='1'/>
2017-12-22:20:59:43: <opencl-index v='1'/>
2017-12-22:20:59:43: </slot>
2017-12-22:20:59:43: <slot id='2' type='GPU'>
2017-12-22:20:59:43: <cuda-index v='2'/>
2017-12-22:20:59:43: <gpu-index v='2'/>
2017-12-22:20:59:43: <opencl-index v='2'/>
2017-12-22:20:59:43: </slot>
2017-12-22:20:59:43: <slot id='3' type='GPU'>
2017-12-22:20:59:43: <cuda-index v='3'/>
2017-12-22:20:59:43: <gpu-index v='3'/>
2017-12-22:20:59:43: <opencl-index v='3'/>
2017-12-22:20:59:43: </slot>
2017-12-22:20:59:43:</config>
2017-12-22:20:59:43:Switching to user fahclient
2017-12-22:20:59:43:Trying to access database...
2017-12-22:20:59:43:Successfully acquired database lock
2017-12-22:20:59:43:Enabled folding slot 01: READY gpu:0:GP104 [GeForce GTX 1080] 8873
2017-12-22:20:59:43:Enabled folding slot 00: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380
2017-12-22:20:59:43:Enabled folding slot 02: READY gpu:2:GP104 [GeForce GTX 1080] 8873
2017-12-22:20:59:43:Enabled folding slot 03: READY gpu:3:GP104 [GeForce GTX 1080] 8873
2017-12-22:20:59:43:WU02:FS01:Starting
2017-12-22:20:59:43:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 02 -suffix 01 -version 704 -lifeline 1584 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2017-12-22:20:59:43:WU02:FS01:Started FahCore on PID 1594
2017-12-22:20:59:43:WU02:FS01:Core PID:1598
2017-12-22:20:59:43:WU02:FS01:FahCore 0x21 started
2017-12-22:20:59:44:WU04:FS02:Starting
2017-12-22:20:59:44:WU04:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 04 -suffix 01 -version 704 -lifeline 1584 -checkpoint 15 -gpu-vendor nvidia -opencl-device 2 -cuda-device 2 -gpu 2
2017-12-22:20:59:44:WU04:FS02:Started FahCore on PID 1601
2017-12-22:20:59:44:WU04:FS02:Core PID:1605
2017-12-22:20:59:44:WU04:FS02:FahCore 0x21 started
2017-12-22:20:59:44:WU00:FS03:Starting
2017-12-22:20:59:44:WU00:FS03:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1584 -checkpoint 15 -gpu-vendor nvidia -opencl-device 3 -cuda-device 3 -gpu 3
2017-12-22:20:59:44:WU00:FS03:Started FahCore on PID 1606
2017-12-22:20:59:44:WU00:FS03:Core PID:1610
2017-12-22:20:59:44:WU00:FS03:FahCore 0x21 started
2017-12-22:20:59:44:WU01:FS00:Starting
2017-12-22:20:59:44:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1584 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
Assuming it was the GPU I tried underclocking it but that hasn't helped.
Re: error:FAULTY project:9431
Posted: Sat Dec 23, 2017 10:46 am
by SteveWillis
Maybe I should mention that FAH thinks the GPU is a 1080 TI but apparently it is actually a 1080 based on the PPDs for the 4 GPUs
Re: error:FAULTY project:9431
Posted: Sat Dec 23, 2017 11:58 am
by SteveWillis
I think I posted the first hundred lines from the wrong log. Look at this instead. Not sure if it matters though.
Code: Select all
steve@fah01 /var/lib/fahclient/logs $ head -120 log-20171223-102844.txt
*********************** Log Started 2017-12-22T21:11:59Z ***********************
2017-12-22:21:11:59:******************* Folding@home Client ********************
2017-12-22:21:11:59: Website: http://folding.stanford.edu/
2017-12-22:21:11:59: Copyright: (c) 2009-2016 Stanford University
2017-12-22:21:11:59: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
2017-12-22:21:11:59: Args: --child --lifeline 1605 /etc/fahclient/config.xml --run-as
2017-12-22:21:11:59: fahclient --pid-file=/var/run/fahclient.pid --daemon
2017-12-22:21:11:59: Config: /etc/fahclient/config.xml
2017-12-22:21:11:59:************************** Build ***************************
2017-12-22:21:11:59: Version: 7.4.16
2017-12-22:21:11:59: Date: Jan 6 2017
2017-12-22:21:11:59: Time: 08:08:33
2017-12-22:21:11:59: Repository: Git
2017-12-22:21:11:59: Revision: e12187cbb0bd6937c067b9749af011374563b7b9
2017-12-22:21:11:59: Branch: master
2017-12-22:21:11:59: Compiler: GNU 4.9.2
2017-12-22:21:11:59: Options: -std=gnu++98 -O3 -funroll-loops -ffast-math -mfpmath=sse
2017-12-22:21:11:59: -fno-unsafe-math-optimizations -msse2
2017-12-22:21:11:59: Platform: linux2 4.8.0-2-amd64
2017-12-22:21:11:59: Bits: 64
2017-12-22:21:11:59: Mode: Release
2017-12-22:21:11:59:************************** System **************************
2017-12-22:21:11:59: CPU: AMD FX(tm)-6300 Six-Core Processor
2017-12-22:21:11:59: CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
2017-12-22:21:11:59: CPUs: 6
2017-12-22:21:11:59: Memory: 11.63GiB
2017-12-22:21:11:59: Free Memory: 11.03GiB
2017-12-22:21:11:59: Threads: POSIX_THREADS
2017-12-22:21:11:59: OS Version: 4.4
2017-12-22:21:11:59: Has Battery: false
2017-12-22:21:11:59: On Battery: false
2017-12-22:21:11:59: UTC Offset: -6
2017-12-22:21:11:59: PID: 1607
2017-12-22:21:11:59: CWD: /var/lib/fahclient
2017-12-22:21:11:59: OS: Linux 4.4.0-53-generic x86_64
2017-12-22:21:11:59: OS Arch: AMD64
2017-12-22:21:11:59: GPUs: 4
2017-12-22:21:11:59: GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1080] 8873
2017-12-22:21:11:59: GPU 1: Bus:8 Slot:0 Func:0 NVIDIA:7 GP102 [GeForce GTX 1080 Ti] 11380
2017-12-22:21:11:59: GPU 2: Bus:9 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1080] 8873
2017-12-22:21:11:59: GPU 3: Bus:10 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1080] 8873
2017-12-22:21:11:59:CUDA Device 0: Platform:0 Device:0 Bus:8 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:21:11:59:CUDA Device 1: Platform:0 Device:1 Bus:1 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:21:11:59:CUDA Device 2: Platform:0 Device:2 Bus:9 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:21:11:59:CUDA Device 3: Platform:0 Device:3 Bus:10 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:21:11:59: OpenCL: Not detected: Failed to open dynamic library 'libOpenCL.so':
2017-12-22:21:11:59: libOpenCL.so: cannot open shared object file: No such file or
2017-12-22:21:11:59: directory
2017-12-22:21:11:59:************************************************************
2017-12-22:21:11:59:<config>
2017-12-22:21:11:59: <!-- Client Control -->
2017-12-22:21:11:59: <fold-anon v='true'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59: <!-- Folding Slot Configuration -->
2017-12-22:21:11:59: <gpu v='false'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59: <!-- Logging -->
2017-12-22:21:11:59: <log-date v='true'/>
2017-12-22:21:11:59: <log-rotate-max v='128'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59: <!-- Network -->
2017-12-22:21:11:59: <proxy v=':8080'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59: <!-- Slot Control -->
2017-12-22:21:11:59: <power v='full'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59: <!-- User Information -->
2017-12-22:21:11:59: <passkey v='********************************'/>
2017-12-22:21:11:59: <team v='224497'/>
2017-12-22:21:11:59: <user v='DarthMouse_ALL_1GD5nCZbh7gNo1SESPLT24xEd2Jsu4rTP9'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59: <!-- Work Unit Control -->
2017-12-22:21:11:59: <next-unit-percentage v='100'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59: <!-- Folding Slots -->
2017-12-22:21:11:59: <slot id='1' type='GPU'>
2017-12-22:21:11:59: <cuda-index v='0'/>
2017-12-22:21:11:59: <gpu-index v='0'/>
2017-12-22:21:11:59: <opencl-index v='0'/>
2017-12-22:21:11:59: </slot>
2017-12-22:21:11:59: <slot id='0' type='GPU'>
2017-12-22:21:11:59: <cuda-index v='1'/>
2017-12-22:21:11:59: <gpu-index v='1'/>
2017-12-22:21:11:59: <opencl-index v='1'/>
2017-12-22:21:11:59: </slot>
2017-12-22:21:11:59: <slot id='2' type='GPU'>
2017-12-22:21:11:59: <cuda-index v='2'/>
2017-12-22:21:11:59: <gpu-index v='2'/>
2017-12-22:21:11:59: <opencl-index v='2'/>
2017-12-22:21:11:59: </slot>
2017-12-22:21:11:59: <slot id='3' type='GPU'>
2017-12-22:21:11:59: <cuda-index v='3'/>
2017-12-22:21:11:59: <gpu-index v='3'/>
2017-12-22:21:11:59: <opencl-index v='3'/>
2017-12-22:21:11:59: </slot>
2017-12-22:21:11:59:</config>
2017-12-22:21:11:59:Switching to user fahclient
2017-12-22:21:11:59:Trying to access database...
2017-12-22:21:11:59:Successfully acquired database lock
2017-12-22:21:11:59:Enabled folding slot 01: READY gpu:0:GP104 [GeForce GTX 1080] 8873
2017-12-22:21:11:59:Enabled folding slot 00: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380
2017-12-22:21:11:59:Enabled folding slot 02: READY gpu:2:GP104 [GeForce GTX 1080] 8873
2017-12-22:21:11:59:Enabled folding slot 03: READY gpu:3:GP104 [GeForce GTX 1080] 8873
2017-12-22:21:11:59:WU01:FS00:Starting
2017-12-22:21:11:59:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1607 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
2017-12-22:21:11:59:WU01:FS00:Started FahCore on PID 1617
2017-12-22:21:11:59:WU01:FS00:Core PID:1621
2017-12-22:21:11:59:WU01:FS00:FahCore 0x21 started
2017-12-22:21:11:59:WU02:FS01:Starting
2017-12-22:21:11:59:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 02 -suffix 01 -version 704 -lifeline 1607 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2017-12-22:21:11:59:WU02:FS01:Started FahCore on PID 1622
2017-12-22:21:11:59:WU02:FS01:Core PID:1626
2017-12-22:21:11:59:WU02:FS01:FahCore 0x21 started
2017-12-22:21:12:00:WU04:FS02:Starting
2017-12-22:21:12:00:WU04:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 04 -suffix 01 -version 704 -lifeline 1607 -checkpoint 15 -gpu-vendor nvidia -opencl-device 2 -cuda-device 2 -gpu 2
2017-12-22:21:12:00:WU04:FS02:Started FahCore on PID 1629
2017-12-22:21:12:00:WU04:FS02:Core PID:1633
2017-12-22:21:12:00:WU04:FS02:FahCore 0x21 started
2017-12-22:21:12:00:WU00:FS03:Starting
2017-12-22:21:12:00:WU00:FS03:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1607 -checkpoint 15 -gpu-vendor nvidia -opencl-device 3 -cuda-device 3 -gpu 3
Re: error:FAULTY project:9431
Posted: Sat Dec 23, 2017 6:04 pm
by bruce
There is only one report of project:9431 run:74 clone:1 gen:721 being returned, and it wasn't' from someone folding as SteveWillis. Ordinary when a project fails, it is assigned to someone else. (This allows WUs which fail because of bad hardware to be completed with good hardware.)
As far as a GPU which thinks it something else, please capture the output of FAHClient --lspci and we can investigate what hardware you actually have. (It's possible that the client is identifying the GPUs in a different order than you think they are in.) In fact, it probably doesn't matter. The 1080 and the 1080 Ti are functionally equivalent as far as FAH is concerned ... they only differ in speed. Since you have set each of the GPUs index values, maybe that's where the problem started.
Re: error:FAULTY project:9431
Posted: Sun Dec 24, 2017 3:45 am
by SteveWillis
Here is just from my last 8 logs
Code: Select all
*********************** Log Started 2017-12-19T05:46:24Z ***********************
log-20171220-222330.txt
2017-12-19:05:46:24: Date: Jan 6 2017
******************************* Date: 2017-12-19 *******************************
2017-12-19:15:21:48:WU02:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
******************************* Date: 2017-12-19 *******************************
******************************* Date: 2017-12-19 *******************************
******************************* Date: 2017-12-20 *******************************
2017-12-20:11:10:02:WU01:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-20:11:10:07:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-20:11:10:07:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:1055 clone:2 gen:99 core:0x21 unit:0x00000080ab436c9d586fdd3c061ef4ae
2017-12-20:11:11:56:WU02:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-20:11:12:01:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-20:11:12:01:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:9431 run:1940 clone:2 gen:60 core:0x21 unit:0x0000004aab436c9d586fdd4314ee6472
******************************* Date: 2017-12-20 *******************************
2017-12-20:14:46:57:WU01:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
******************************* Date: 2017-12-20 *******************************
2017-12-20:21:37:49:WU01:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-20:21:37:54:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-20:21:37:54:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:103 clone:2 gen:349 core:0x21 unit:0x00000199ab436c9d586fdd34692b55c4
Current log -7
*********************** Log Started 2017-12-20T22:23:30Z ***********************
log-20171221-074007.txt
2017-12-20:22:23:30: Date: Jan 6 2017
2017-12-21:00:20:59:WU01:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
******************************* Date: 2017-12-21 *******************************
2017-12-21:07:00:52:WU04:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-21:07:00:57:WARNING:WU04:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-21:07:00:57:WU04:FS00:Sending unit results: id:04 state:SEND error:FAULTY project:9431 run:1572 clone:2 gen:105 core:0x21 unit:0x0000007eab436c9d586fdd4078687ba0
Current log -6
*********************** Log Started 2017-12-21T07:40:07Z ***********************
log-20171221-220654.txt
2017-12-21:07:40:07: Date: Jan 6 2017
******************************* Date: 2017-12-21 *******************************
******************************* Date: 2017-12-21 *******************************
2017-12-21:21:23:53:WU03:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-21:21:23:58:WARNING:WU03:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-21:21:23:58:WU03:FS00:Sending unit results: id:03 state:SEND error:FAULTY project:9431 run:19 clone:1 gen:770 core:0x21 unit:0x00000384ab436c9d586fdd331058bd97
Current log -5
*********************** Log Started 2017-12-21T22:06:54Z ***********************
log-20171222-010313.txt
2017-12-21:22:06:54: Date: Jan 6 2017
2017-12-22:00:21:55:WU04:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-22:00:22:00:WARNING:WU04:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-22:00:22:00:WU04:FS00:Sending unit results: id:04 state:SEND error:FAULTY project:9431 run:155 clone:4 gen:93 core:0x21 unit:0x00000072ab436c9d586fdd350a385278
2017-12-22:00:34:37:WU02:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-22:00:34:41:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-22:00:34:41:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:9431 run:1650 clone:2 gen:203 core:0x21 unit:0x000000f3ab436c9d586fdd40ac510a71
2017-12-22:00:35:21:WU04:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-22:00:35:25:WARNING:WU04:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-22:00:35:25:WU04:FS00:Sending unit results: id:04 state:SEND error:FAULTY project:9431 run:1016 clone:4 gen:80 core:0x21 unit:0x0000005fab436c9d586fdd3b67e40230
Current log -4
*********************** Log Started 2017-12-22T01:03:13Z ***********************
log-20171222-205943.txt
2017-12-22:01:03:13: Date: Jan 6 2017
2017-12-22:05:17:14:WU01:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
******************************* Date: 2017-12-22 *******************************
******************************* Date: 2017-12-22 *******************************
2017-12-22:18:18:17:WU02:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
******************************* Date: 2017-12-22 *******************************
2017-12-22:20:23:03:WU02:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-22:20:23:07:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-22:20:23:07:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:9431 run:1617 clone:0 gen:518 core:0x21 unit:0x00000260ab436c9d586fdd408979b4f4
Current log -3
*********************** Log Started 2017-12-22T20:59:43Z ***********************
log-20171222-211159.txt
2017-12-22:20:59:43: Date: Jan 6 2017
Current log -2
*********************** Log Started 2017-12-22T21:11:59Z ***********************
log-20171223-102844.txt
2017-12-22:21:11:59: Date: Jan 6 2017
******************************* Date: 2017-12-23 *******************************
******************************* Date: 2017-12-23 *******************************
2017-12-23:09:46:24:WU04:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-23:09:46:29:WARNING:WU04:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-23:09:46:29:WU04:FS00:Sending unit results: id:04 state:SEND error:FAULTY project:9431 run:74 clone:1 gen:721 core:0x21 unit:0x0000035eab436c9d586fdd344eef242b
2017-12-23:10:19:08:WU01:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
Current log -1
*********************** Log Started 2017-12-23T10:28:44Z ***********************
log-20171224-011527.txt
2017-12-23:10:28:44: Date: Jan 6 2017
******************************* Date: 2017-12-23 *******************************
******************************* Date: 2017-12-23 *******************************
2017-12-23:23:31:10:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
2017-12-24:00:30:35:WU01:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-24:00:30:40:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-24:00:30:40:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:32 clone:1 gen:348 core:0x21 unit:0x000001a0ab436c9d586fdd33ce284380
*********************** Log Started 2017-12-24T01:15:27Z ***********************
log.txt
2017-12-24:01:15:27: Date: Jan 6 2017
###########################################################################
Sat Dec 23 19:14:02 CST 2017 Download stuck
21:43:45 up 2:29, 1 user, load average: 4.42, 4.55, 4.35
Sat Dec 23 21:43:45 CST 2017
Re: error:FAULTY project:9431
Posted: Sun Dec 24, 2017 3:48 am
by SteveWillis
I fold as
DarthMouse_ALL_1GD5nCZbh7gNo1SESPLT24xEd2Jsu4rTP9
Re: error:FAULTY project:9431
Posted: Sun Dec 24, 2017 4:08 am
by Joe_H
All but the most recent couple WU's that have failed on your system has been successfully processed by someone else. My guess is that there is a problem of some sort with either the setup for that one GPU slot, or the GPU itself.
Re: error:FAULTY project:9431
Posted: Sun Dec 24, 2017 5:14 am
by SteveWillis
Strange that it's ONLY for that project. The GPU doesn't have problems with other projects
Re: error:FAULTY project:9431
Posted: Sun Dec 24, 2017 5:22 pm
by bruce
My guess is that if you reduce the clock rate on that one CPU, you'll be able to run those projects.
Second guess: Improve the airflow around that GPU.
Re: error:FAULTY project:9431
Posted: Sun Dec 24, 2017 6:47 pm
by SteveWillis
Thanks bruce for your suggestions.
I had already lowered the clock rate to -200 mhz which was the lowest it could go and the temperature never gets over 70C. Your suggestion did make me think to change the GPU from maximum performance to auto so maybe that will help.
It also occurs to me that if FAH is misreporting which GPU corresponds to the affected FS then possibly I'm underclocking the wrong GPU, so I'm underclocking all the GPUs on that machine now as a test. We'll see.
Re: error:FAULTY project:9431
Posted: Mon Dec 25, 2017 5:09 am
by bruce
FAH's methodology of enumerating GPUs is often inconsistent. What works in Linux is different than what works in Windows. and I'm not sure they've ever resolve all the issues that arise. This was never a problem back when most systems had only one GPU.
The only dependable method I've found is to pause the GPUs one-at-a-time and feel which one cools off. Then you can adjust the index values to make it work the way you want.
Re: error:FAULTY project:9431
Posted: Mon Dec 25, 2017 3:11 pm
by SteveWillis
Underclocking all the GPUs as a test did stop the error. Thanks for the suggestion on getting them straightened out.
Re: error:FAULTY project:9431
Posted: Tue Dec 26, 2017 6:10 pm
by SteveWillis
I finished all the WUs, deleted all the FSs, one at a time created each FS, saved and restarted folding before going on to the next one. Now the FSs are in the proper order and the proper GPUs are associated with them.
Re: error:FAULTY project:9431
Posted: Wed Dec 27, 2017 10:15 pm
by toTOW
Until your next reboot ...
Re: error:FAULTY project:9431
Posted: Wed Dec 27, 2017 10:37 pm
by Joe_H
Or driver update...