error:FAULTY project:9431

Moderators: Site Moderators, FAHC Science Team

Post Reply
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

error:FAULTY project:9431

Post by SteveWillis »

I have had a series of faulty project errors but only on one of my three machines, always on the same GPU. However somehow I managed to fail to notice that it's always the same project.

This is the latest

Code: Select all

2017-12-23:09:46:24:WU04:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-23:09:46:29:WARNING:WU04:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-23:09:46:29:WU04:FS00:Sending unit results: id:04 state:SEND error:FAULTY project:9431 run:74 clone:1 gen:721 core:0x21 unit:0x0000035eab436c9d586fdd344eef242b

Code: Select all

*********************** Log Started 2017-12-22T20:59:43Z ***********************
2017-12-22:20:59:43:******************* Folding@home Client ********************
2017-12-22:20:59:43:      Website: http://folding.stanford.edu/
2017-12-22:20:59:43:    Copyright: (c) 2009-2016 Stanford University
2017-12-22:20:59:43:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
2017-12-22:20:59:43:         Args: --child --lifeline 1582 /etc/fahclient/config.xml --run-as
2017-12-22:20:59:43:               fahclient --pid-file=/var/run/fahclient.pid --daemon
2017-12-22:20:59:43:       Config: /etc/fahclient/config.xml
2017-12-22:20:59:43:************************** Build ***************************
2017-12-22:20:59:43:      Version: 7.4.16
2017-12-22:20:59:43:         Date: Jan 6 2017
2017-12-22:20:59:43:         Time: 08:08:33
2017-12-22:20:59:43:   Repository: Git
2017-12-22:20:59:43:     Revision: e12187cbb0bd6937c067b9749af011374563b7b9
2017-12-22:20:59:43:       Branch: master
2017-12-22:20:59:43:     Compiler: GNU 4.9.2
2017-12-22:20:59:43:      Options: -std=gnu++98 -O3 -funroll-loops -ffast-math -mfpmath=sse
2017-12-22:20:59:43:               -fno-unsafe-math-optimizations -msse2
2017-12-22:20:59:43:     Platform: linux2 4.8.0-2-amd64
2017-12-22:20:59:43:         Bits: 64
2017-12-22:20:59:43:         Mode: Release
2017-12-22:20:59:43:************************** System **************************
2017-12-22:20:59:43:          CPU: AMD FX(tm)-6300 Six-Core Processor
2017-12-22:20:59:43:       CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
2017-12-22:20:59:43:         CPUs: 6
2017-12-22:20:59:43:       Memory: 11.63GiB
2017-12-22:20:59:43:  Free Memory: 11.02GiB
2017-12-22:20:59:43:      Threads: POSIX_THREADS
2017-12-22:20:59:43:   OS Version: 4.4
2017-12-22:20:59:43:  Has Battery: false
2017-12-22:20:59:43:   On Battery: false
2017-12-22:20:59:43:   UTC Offset: -6
2017-12-22:20:59:43:          PID: 1584
2017-12-22:20:59:43:          CWD: /var/lib/fahclient
2017-12-22:20:59:43:           OS: Linux 4.4.0-53-generic x86_64
2017-12-22:20:59:43:      OS Arch: AMD64
2017-12-22:20:59:43:         GPUs: 4
2017-12-22:20:59:43:        GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1080] 8873
2017-12-22:20:59:43:        GPU 1: Bus:8 Slot:0 Func:0 NVIDIA:7 GP102 [GeForce GTX 1080 Ti] 11380
2017-12-22:20:59:43:        GPU 2: Bus:9 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1080] 8873
2017-12-22:20:59:43:        GPU 3: Bus:10 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1080] 8873
2017-12-22:20:59:43:CUDA Device 0: Platform:0 Device:0 Bus:8 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:20:59:43:CUDA Device 1: Platform:0 Device:1 Bus:1 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:20:59:43:CUDA Device 2: Platform:0 Device:2 Bus:9 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:20:59:43:CUDA Device 3: Platform:0 Device:3 Bus:10 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:20:59:43:       OpenCL: Not detected: Failed to open dynamic library 'libOpenCL.so':
2017-12-22:20:59:43:               libOpenCL.so: cannot open shared object file: No such file or
2017-12-22:20:59:43:               directory
2017-12-22:20:59:43:************************************************************
2017-12-22:20:59:43:<config>
2017-12-22:20:59:43:  <!-- Client Control -->
2017-12-22:20:59:43:  <fold-anon v='true'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43:  <!-- Folding Slot Configuration -->
2017-12-22:20:59:43:  <gpu v='false'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43:  <!-- Logging -->
2017-12-22:20:59:43:  <log-date v='true'/>
2017-12-22:20:59:43:  <log-rotate-max v='128'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43:  <!-- Network -->
2017-12-22:20:59:43:  <proxy v=':8080'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43:  <!-- Slot Control -->
2017-12-22:20:59:43:  <power v='full'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43:  <!-- User Information -->
2017-12-22:20:59:43:  <passkey v='********************************'/>
2017-12-22:20:59:43:  <team v='224497'/>
2017-12-22:20:59:43:  <user v='DarthMouse_ALL_1GD5nCZbh7gNo1SESPLT24xEd2Jsu4rTP9'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43:  <!-- Work Unit Control -->
2017-12-22:20:59:43:  <next-unit-percentage v='100'/>
2017-12-22:20:59:43:
2017-12-22:20:59:43:  <!-- Folding Slots -->
2017-12-22:20:59:43:  <slot id='1' type='GPU'>
2017-12-22:20:59:43:    <cuda-index v='0'/>
2017-12-22:20:59:43:    <gpu-index v='0'/>
2017-12-22:20:59:43:    <opencl-index v='0'/>
2017-12-22:20:59:43:  </slot>
2017-12-22:20:59:43:  <slot id='0' type='GPU'>
2017-12-22:20:59:43:    <cuda-index v='1'/>
2017-12-22:20:59:43:    <gpu-index v='1'/>
2017-12-22:20:59:43:    <opencl-index v='1'/>
2017-12-22:20:59:43:  </slot>
2017-12-22:20:59:43:  <slot id='2' type='GPU'>
2017-12-22:20:59:43:    <cuda-index v='2'/>
2017-12-22:20:59:43:    <gpu-index v='2'/>
2017-12-22:20:59:43:    <opencl-index v='2'/>
2017-12-22:20:59:43:  </slot>
2017-12-22:20:59:43:  <slot id='3' type='GPU'>
2017-12-22:20:59:43:    <cuda-index v='3'/>
2017-12-22:20:59:43:    <gpu-index v='3'/>
2017-12-22:20:59:43:    <opencl-index v='3'/>
2017-12-22:20:59:43:  </slot>
2017-12-22:20:59:43:</config>
2017-12-22:20:59:43:Switching to user fahclient
2017-12-22:20:59:43:Trying to access database...
2017-12-22:20:59:43:Successfully acquired database lock
2017-12-22:20:59:43:Enabled folding slot 01: READY gpu:0:GP104 [GeForce GTX 1080] 8873
2017-12-22:20:59:43:Enabled folding slot 00: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380
2017-12-22:20:59:43:Enabled folding slot 02: READY gpu:2:GP104 [GeForce GTX 1080] 8873
2017-12-22:20:59:43:Enabled folding slot 03: READY gpu:3:GP104 [GeForce GTX 1080] 8873
2017-12-22:20:59:43:WU02:FS01:Starting
2017-12-22:20:59:43:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 02 -suffix 01 -version 704 -lifeline 1584 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2017-12-22:20:59:43:WU02:FS01:Started FahCore on PID 1594
2017-12-22:20:59:43:WU02:FS01:Core PID:1598
2017-12-22:20:59:43:WU02:FS01:FahCore 0x21 started
2017-12-22:20:59:44:WU04:FS02:Starting
2017-12-22:20:59:44:WU04:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 04 -suffix 01 -version 704 -lifeline 1584 -checkpoint 15 -gpu-vendor nvidia -opencl-device 2 -cuda-device 2 -gpu 2
2017-12-22:20:59:44:WU04:FS02:Started FahCore on PID 1601
2017-12-22:20:59:44:WU04:FS02:Core PID:1605
2017-12-22:20:59:44:WU04:FS02:FahCore 0x21 started
2017-12-22:20:59:44:WU00:FS03:Starting
2017-12-22:20:59:44:WU00:FS03:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1584 -checkpoint 15 -gpu-vendor nvidia -opencl-device 3 -cuda-device 3 -gpu 3
2017-12-22:20:59:44:WU00:FS03:Started FahCore on PID 1606
2017-12-22:20:59:44:WU00:FS03:Core PID:1610
2017-12-22:20:59:44:WU00:FS03:FahCore 0x21 started
2017-12-22:20:59:44:WU01:FS00:Starting
2017-12-22:20:59:44:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1584 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
Assuming it was the GPU I tried underclocking it but that hasn't helped.
Image

1080 and 1080TI GPUs on Linux Mint
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: error:FAULTY project:9431

Post by SteveWillis »

Maybe I should mention that FAH thinks the GPU is a 1080 TI but apparently it is actually a 1080 based on the PPDs for the 4 GPUs
Image

1080 and 1080TI GPUs on Linux Mint
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: error:FAULTY project:9431

Post by SteveWillis »

I think I posted the first hundred lines from the wrong log. Look at this instead. Not sure if it matters though.

Code: Select all

steve@fah01 /var/lib/fahclient/logs $ head -120 log-20171223-102844.txt
*********************** Log Started 2017-12-22T21:11:59Z ***********************
2017-12-22:21:11:59:******************* Folding@home Client ********************
2017-12-22:21:11:59:      Website: http://folding.stanford.edu/
2017-12-22:21:11:59:    Copyright: (c) 2009-2016 Stanford University
2017-12-22:21:11:59:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
2017-12-22:21:11:59:         Args: --child --lifeline 1605 /etc/fahclient/config.xml --run-as
2017-12-22:21:11:59:               fahclient --pid-file=/var/run/fahclient.pid --daemon
2017-12-22:21:11:59:       Config: /etc/fahclient/config.xml
2017-12-22:21:11:59:************************** Build ***************************
2017-12-22:21:11:59:      Version: 7.4.16
2017-12-22:21:11:59:         Date: Jan 6 2017
2017-12-22:21:11:59:         Time: 08:08:33
2017-12-22:21:11:59:   Repository: Git
2017-12-22:21:11:59:     Revision: e12187cbb0bd6937c067b9749af011374563b7b9
2017-12-22:21:11:59:       Branch: master
2017-12-22:21:11:59:     Compiler: GNU 4.9.2
2017-12-22:21:11:59:      Options: -std=gnu++98 -O3 -funroll-loops -ffast-math -mfpmath=sse
2017-12-22:21:11:59:               -fno-unsafe-math-optimizations -msse2
2017-12-22:21:11:59:     Platform: linux2 4.8.0-2-amd64
2017-12-22:21:11:59:         Bits: 64
2017-12-22:21:11:59:         Mode: Release
2017-12-22:21:11:59:************************** System **************************
2017-12-22:21:11:59:          CPU: AMD FX(tm)-6300 Six-Core Processor
2017-12-22:21:11:59:       CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
2017-12-22:21:11:59:         CPUs: 6
2017-12-22:21:11:59:       Memory: 11.63GiB
2017-12-22:21:11:59:  Free Memory: 11.03GiB
2017-12-22:21:11:59:      Threads: POSIX_THREADS
2017-12-22:21:11:59:   OS Version: 4.4
2017-12-22:21:11:59:  Has Battery: false
2017-12-22:21:11:59:   On Battery: false
2017-12-22:21:11:59:   UTC Offset: -6
2017-12-22:21:11:59:          PID: 1607
2017-12-22:21:11:59:          CWD: /var/lib/fahclient
2017-12-22:21:11:59:           OS: Linux 4.4.0-53-generic x86_64
2017-12-22:21:11:59:      OS Arch: AMD64
2017-12-22:21:11:59:         GPUs: 4
2017-12-22:21:11:59:        GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1080] 8873
2017-12-22:21:11:59:        GPU 1: Bus:8 Slot:0 Func:0 NVIDIA:7 GP102 [GeForce GTX 1080 Ti] 11380
2017-12-22:21:11:59:        GPU 2: Bus:9 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1080] 8873
2017-12-22:21:11:59:        GPU 3: Bus:10 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1080] 8873
2017-12-22:21:11:59:CUDA Device 0: Platform:0 Device:0 Bus:8 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:21:11:59:CUDA Device 1: Platform:0 Device:1 Bus:1 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:21:11:59:CUDA Device 2: Platform:0 Device:2 Bus:9 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:21:11:59:CUDA Device 3: Platform:0 Device:3 Bus:10 Slot:0 Compute:6.1 Driver:9.0
2017-12-22:21:11:59:       OpenCL: Not detected: Failed to open dynamic library 'libOpenCL.so':
2017-12-22:21:11:59:               libOpenCL.so: cannot open shared object file: No such file or
2017-12-22:21:11:59:               directory
2017-12-22:21:11:59:************************************************************
2017-12-22:21:11:59:<config>
2017-12-22:21:11:59:  <!-- Client Control -->
2017-12-22:21:11:59:  <fold-anon v='true'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59:  <!-- Folding Slot Configuration -->
2017-12-22:21:11:59:  <gpu v='false'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59:  <!-- Logging -->
2017-12-22:21:11:59:  <log-date v='true'/>
2017-12-22:21:11:59:  <log-rotate-max v='128'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59:  <!-- Network -->
2017-12-22:21:11:59:  <proxy v=':8080'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59:  <!-- Slot Control -->
2017-12-22:21:11:59:  <power v='full'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59:  <!-- User Information -->
2017-12-22:21:11:59:  <passkey v='********************************'/>
2017-12-22:21:11:59:  <team v='224497'/>
2017-12-22:21:11:59:  <user v='DarthMouse_ALL_1GD5nCZbh7gNo1SESPLT24xEd2Jsu4rTP9'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59:  <!-- Work Unit Control -->
2017-12-22:21:11:59:  <next-unit-percentage v='100'/>
2017-12-22:21:11:59:
2017-12-22:21:11:59:  <!-- Folding Slots -->
2017-12-22:21:11:59:  <slot id='1' type='GPU'>
2017-12-22:21:11:59:    <cuda-index v='0'/>
2017-12-22:21:11:59:    <gpu-index v='0'/>
2017-12-22:21:11:59:    <opencl-index v='0'/>
2017-12-22:21:11:59:  </slot>
2017-12-22:21:11:59:  <slot id='0' type='GPU'>
2017-12-22:21:11:59:    <cuda-index v='1'/>
2017-12-22:21:11:59:    <gpu-index v='1'/>
2017-12-22:21:11:59:    <opencl-index v='1'/>
2017-12-22:21:11:59:  </slot>
2017-12-22:21:11:59:  <slot id='2' type='GPU'>
2017-12-22:21:11:59:    <cuda-index v='2'/>
2017-12-22:21:11:59:    <gpu-index v='2'/>
2017-12-22:21:11:59:    <opencl-index v='2'/>
2017-12-22:21:11:59:  </slot>
2017-12-22:21:11:59:  <slot id='3' type='GPU'>
2017-12-22:21:11:59:    <cuda-index v='3'/>
2017-12-22:21:11:59:    <gpu-index v='3'/>
2017-12-22:21:11:59:    <opencl-index v='3'/>
2017-12-22:21:11:59:  </slot>
2017-12-22:21:11:59:</config>
2017-12-22:21:11:59:Switching to user fahclient
2017-12-22:21:11:59:Trying to access database...
2017-12-22:21:11:59:Successfully acquired database lock
2017-12-22:21:11:59:Enabled folding slot 01: READY gpu:0:GP104 [GeForce GTX 1080] 8873
2017-12-22:21:11:59:Enabled folding slot 00: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380
2017-12-22:21:11:59:Enabled folding slot 02: READY gpu:2:GP104 [GeForce GTX 1080] 8873
2017-12-22:21:11:59:Enabled folding slot 03: READY gpu:3:GP104 [GeForce GTX 1080] 8873
2017-12-22:21:11:59:WU01:FS00:Starting
2017-12-22:21:11:59:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1607 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
2017-12-22:21:11:59:WU01:FS00:Started FahCore on PID 1617
2017-12-22:21:11:59:WU01:FS00:Core PID:1621
2017-12-22:21:11:59:WU01:FS00:FahCore 0x21 started
2017-12-22:21:11:59:WU02:FS01:Starting
2017-12-22:21:11:59:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 02 -suffix 01 -version 704 -lifeline 1607 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2017-12-22:21:11:59:WU02:FS01:Started FahCore on PID 1622
2017-12-22:21:11:59:WU02:FS01:Core PID:1626
2017-12-22:21:11:59:WU02:FS01:FahCore 0x21 started
2017-12-22:21:12:00:WU04:FS02:Starting
2017-12-22:21:12:00:WU04:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 04 -suffix 01 -version 704 -lifeline 1607 -checkpoint 15 -gpu-vendor nvidia -opencl-device 2 -cuda-device 2 -gpu 2
2017-12-22:21:12:00:WU04:FS02:Started FahCore on PID 1629
2017-12-22:21:12:00:WU04:FS02:Core PID:1633
2017-12-22:21:12:00:WU04:FS02:FahCore 0x21 started
2017-12-22:21:12:00:WU00:FS03:Starting
2017-12-22:21:12:00:WU00:FS03:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1607 -checkpoint 15 -gpu-vendor nvidia -opencl-device 3 -cuda-device 3 -gpu 3
Image

1080 and 1080TI GPUs on Linux Mint
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: error:FAULTY project:9431

Post by bruce »

There is only one report of project:9431 run:74 clone:1 gen:721 being returned, and it wasn't' from someone folding as SteveWillis. Ordinary when a project fails, it is assigned to someone else. (This allows WUs which fail because of bad hardware to be completed with good hardware.)

As far as a GPU which thinks it something else, please capture the output of FAHClient --lspci and we can investigate what hardware you actually have. (It's possible that the client is identifying the GPUs in a different order than you think they are in.) In fact, it probably doesn't matter. The 1080 and the 1080 Ti are functionally equivalent as far as FAH is concerned ... they only differ in speed. Since you have set each of the GPUs index values, maybe that's where the problem started.
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: error:FAULTY project:9431

Post by SteveWillis »

Here is just from my last 8 logs

Code: Select all

*********************** Log Started 2017-12-19T05:46:24Z ***********************
log-20171220-222330.txt
2017-12-19:05:46:24:         Date: Jan 6 2017
******************************* Date: 2017-12-19 *******************************
2017-12-19:15:21:48:WU02:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
******************************* Date: 2017-12-19 *******************************
******************************* Date: 2017-12-19 *******************************
******************************* Date: 2017-12-20 *******************************
2017-12-20:11:10:02:WU01:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-20:11:10:07:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-20:11:10:07:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:1055 clone:2 gen:99 core:0x21 unit:0x00000080ab436c9d586fdd3c061ef4ae
2017-12-20:11:11:56:WU02:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-20:11:12:01:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-20:11:12:01:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:9431 run:1940 clone:2 gen:60 core:0x21 unit:0x0000004aab436c9d586fdd4314ee6472
******************************* Date: 2017-12-20 *******************************
2017-12-20:14:46:57:WU01:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
******************************* Date: 2017-12-20 *******************************
2017-12-20:21:37:49:WU01:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-20:21:37:54:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-20:21:37:54:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:103 clone:2 gen:349 core:0x21 unit:0x00000199ab436c9d586fdd34692b55c4


Current log -7
*********************** Log Started 2017-12-20T22:23:30Z ***********************
log-20171221-074007.txt
2017-12-20:22:23:30:         Date: Jan 6 2017
2017-12-21:00:20:59:WU01:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
******************************* Date: 2017-12-21 *******************************
2017-12-21:07:00:52:WU04:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-21:07:00:57:WARNING:WU04:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-21:07:00:57:WU04:FS00:Sending unit results: id:04 state:SEND error:FAULTY project:9431 run:1572 clone:2 gen:105 core:0x21 unit:0x0000007eab436c9d586fdd4078687ba0


Current log -6
*********************** Log Started 2017-12-21T07:40:07Z ***********************
log-20171221-220654.txt
2017-12-21:07:40:07:         Date: Jan 6 2017
******************************* Date: 2017-12-21 *******************************
******************************* Date: 2017-12-21 *******************************
2017-12-21:21:23:53:WU03:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-21:21:23:58:WARNING:WU03:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-21:21:23:58:WU03:FS00:Sending unit results: id:03 state:SEND error:FAULTY project:9431 run:19 clone:1 gen:770 core:0x21 unit:0x00000384ab436c9d586fdd331058bd97


Current log -5
*********************** Log Started 2017-12-21T22:06:54Z ***********************
log-20171222-010313.txt
2017-12-21:22:06:54:         Date: Jan 6 2017
2017-12-22:00:21:55:WU04:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-22:00:22:00:WARNING:WU04:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-22:00:22:00:WU04:FS00:Sending unit results: id:04 state:SEND error:FAULTY project:9431 run:155 clone:4 gen:93 core:0x21 unit:0x00000072ab436c9d586fdd350a385278
2017-12-22:00:34:37:WU02:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-22:00:34:41:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-22:00:34:41:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:9431 run:1650 clone:2 gen:203 core:0x21 unit:0x000000f3ab436c9d586fdd40ac510a71
2017-12-22:00:35:21:WU04:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-22:00:35:25:WARNING:WU04:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-22:00:35:25:WU04:FS00:Sending unit results: id:04 state:SEND error:FAULTY project:9431 run:1016 clone:4 gen:80 core:0x21 unit:0x0000005fab436c9d586fdd3b67e40230


Current log -4
*********************** Log Started 2017-12-22T01:03:13Z ***********************
log-20171222-205943.txt
2017-12-22:01:03:13:         Date: Jan 6 2017
2017-12-22:05:17:14:WU01:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
******************************* Date: 2017-12-22 *******************************
******************************* Date: 2017-12-22 *******************************
2017-12-22:18:18:17:WU02:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
******************************* Date: 2017-12-22 *******************************
2017-12-22:20:23:03:WU02:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-22:20:23:07:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-22:20:23:07:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:9431 run:1617 clone:0 gen:518 core:0x21 unit:0x00000260ab436c9d586fdd408979b4f4


Current log -3
*********************** Log Started 2017-12-22T20:59:43Z ***********************
log-20171222-211159.txt
2017-12-22:20:59:43:         Date: Jan 6 2017


Current log -2
*********************** Log Started 2017-12-22T21:11:59Z ***********************
log-20171223-102844.txt
2017-12-22:21:11:59:         Date: Jan 6 2017
******************************* Date: 2017-12-23 *******************************
******************************* Date: 2017-12-23 *******************************
2017-12-23:09:46:24:WU04:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-23:09:46:29:WARNING:WU04:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-23:09:46:29:WU04:FS00:Sending unit results: id:04 state:SEND error:FAULTY project:9431 run:74 clone:1 gen:721 core:0x21 unit:0x0000035eab436c9d586fdd344eef242b
2017-12-23:10:19:08:WU01:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?


Current log -1
*********************** Log Started 2017-12-23T10:28:44Z ***********************
log-20171224-011527.txt
2017-12-23:10:28:44:         Date: Jan 6 2017
******************************* Date: 2017-12-23 *******************************
******************************* Date: 2017-12-23 *******************************
2017-12-23:23:31:10:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
2017-12-24:00:30:35:WU01:FS00:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
2017-12-24:00:30:40:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2017-12-24:00:30:40:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:32 clone:1 gen:348 core:0x21 unit:0x000001a0ab436c9d586fdd33ce284380


*********************** Log Started 2017-12-24T01:15:27Z ***********************
log.txt
2017-12-24:01:15:27:         Date: Jan 6 2017

###########################################################################
Sat Dec 23 19:14:02 CST 2017  Download stuck
 21:43:45 up  2:29,  1 user,  load average: 4.42, 4.55, 4.35
Sat Dec 23 21:43:45 CST 2017
Image

1080 and 1080TI GPUs on Linux Mint
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: error:FAULTY project:9431

Post by SteveWillis »

I fold as
DarthMouse_ALL_1GD5nCZbh7gNo1SESPLT24xEd2Jsu4rTP9
Image

1080 and 1080TI GPUs on Linux Mint
Joe_H
Site Admin
Posts: 7856
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: error:FAULTY project:9431

Post by Joe_H »

All but the most recent couple WU's that have failed on your system has been successfully processed by someone else. My guess is that there is a problem of some sort with either the setup for that one GPU slot, or the GPU itself.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: error:FAULTY project:9431

Post by SteveWillis »

Strange that it's ONLY for that project. The GPU doesn't have problems with other projects
Image

1080 and 1080TI GPUs on Linux Mint
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: error:FAULTY project:9431

Post by bruce »

My guess is that if you reduce the clock rate on that one CPU, you'll be able to run those projects.
Second guess: Improve the airflow around that GPU.
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: error:FAULTY project:9431

Post by SteveWillis »

Thanks bruce for your suggestions.
I had already lowered the clock rate to -200 mhz which was the lowest it could go and the temperature never gets over 70C. Your suggestion did make me think to change the GPU from maximum performance to auto so maybe that will help.
It also occurs to me that if FAH is misreporting which GPU corresponds to the affected FS then possibly I'm underclocking the wrong GPU, so I'm underclocking all the GPUs on that machine now as a test. We'll see.
Image

1080 and 1080TI GPUs on Linux Mint
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: error:FAULTY project:9431

Post by bruce »

FAH's methodology of enumerating GPUs is often inconsistent. What works in Linux is different than what works in Windows. and I'm not sure they've ever resolve all the issues that arise. This was never a problem back when most systems had only one GPU.

The only dependable method I've found is to pause the GPUs one-at-a-time and feel which one cools off. Then you can adjust the index values to make it work the way you want.
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: error:FAULTY project:9431

Post by SteveWillis »

Underclocking all the GPUs as a test did stop the error. Thanks for the suggestion on getting them straightened out.
Image

1080 and 1080TI GPUs on Linux Mint
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: error:FAULTY project:9431

Post by SteveWillis »

I finished all the WUs, deleted all the FSs, one at a time created each FS, saved and restarted folding before going on to the next one. Now the FSs are in the proper order and the proper GPUs are associated with them.
Image

1080 and 1080TI GPUs on Linux Mint
toTOW
Site Moderator
Posts: 6296
Joined: Sun Dec 02, 2007 10:38 am
Location: Bordeaux, France
Contact:

Re: error:FAULTY project:9431

Post by toTOW »

Until your next reboot ... :roll:
Image

Folding@Home beta tester since 2002. Folding Forum moderator since July 2008.
Joe_H
Site Admin
Posts: 7856
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: error:FAULTY project:9431

Post by Joe_H »

Or driver update...
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Post Reply