WU 13414 hangs forever on RX5700XT

[zookeeny]

For some reason, all of the 13414 work units I've received have locked up my RX5700XT card - see the log below for an example where nothing happened for almost 13 hours. The only way I'm able to get things working again is to manually dump the WU.

Code: Select all

00:25:00:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13414 run:3534 clone:50 gen:1 core:0x22 unit:0x0000000512bc7d9a5ef50d675ea5a9cf
00:25:04:WU00:FS01:FahCore 0x22 started
00:25:04:WU00:FS01:0x22:*********************** Log Started 2020-06-29T00:25:04Z ***********************
00:25:04:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
00:25:04:WU00:FS01:0x22:       Core: Core22
00:25:04:WU00:FS01:0x22:       Type: 0x22
00:25:04:WU00:FS01:0x22:    Version: 0.0.10
00:25:04:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
00:25:04:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
00:25:04:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
00:25:04:WU00:FS01:0x22:       Date: Jun 16 2020
00:25:04:WU00:FS01:0x22:       Time: 15:55:31
00:25:04:WU00:FS01:0x22:   Revision: 147051aad40bcbec7d4b25105bbedfab425f1dc2
00:25:04:WU00:FS01:0x22:     Branch: core22-0.0.10
00:25:04:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
00:25:04:WU00:FS01:0x22:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
00:25:04:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
00:25:04:WU00:FS01:0x22:       Bits: 64
00:25:04:WU00:FS01:0x22:       Mode: Release
00:25:04:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
00:25:04:WU00:FS01:0x22:             <peastman@stanford.edu>
00:25:04:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 25434 -checkpoint 15
00:25:04:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
00:25:04:WU00:FS01:0x22:************************************ libFAH ************************************
00:25:04:WU00:FS01:0x22:       Date: Jun 2 2020
00:25:04:WU00:FS01:0x22:       Time: 00:07:31
00:25:04:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
00:25:04:WU00:FS01:0x22:     Branch: HEAD
00:25:04:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
00:25:04:WU00:FS01:0x22:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
00:25:04:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
00:25:04:WU00:FS01:0x22:       Bits: 64
00:25:04:WU00:FS01:0x22:       Mode: Release
00:25:04:WU00:FS01:0x22:************************************ CBang *************************************
00:25:04:WU00:FS01:0x22:       Date: May 31 2020
00:25:04:WU00:FS01:0x22:       Time: 20:16:34
00:25:04:WU00:FS01:0x22:   Revision: 75fcee0b8e713cb47f5191a3689d5f4f07244c7f
00:25:04:WU00:FS01:0x22:     Branch: HEAD
00:25:04:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
00:25:04:WU00:FS01:0x22:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
00:25:04:WU00:FS01:0x22:             -fPIC
00:25:04:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
00:25:04:WU00:FS01:0x22:       Bits: 64
00:25:04:WU00:FS01:0x22:       Mode: Release
00:25:04:WU00:FS01:0x22:************************************ System ************************************
00:25:04:WU00:FS01:0x22:        CPU: AMD Ryzen 9 3900X 12-Core Processor
00:25:04:WU00:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
00:25:04:WU00:FS01:0x22:       CPUs: 24
00:25:04:WU00:FS01:0x22:     Memory: 31.37GiB
00:25:04:WU00:FS01:0x22:Free Memory: 24.47GiB
00:25:04:WU00:FS01:0x22:    Threads: POSIX_THREADS
00:25:04:WU00:FS01:0x22: OS Version: 5.4
00:25:04:WU00:FS01:0x22:Has Battery: false
00:25:04:WU00:FS01:0x22: On Battery: false
00:25:04:WU00:FS01:0x22: UTC Offset: -4
00:25:04:WU00:FS01:0x22:        PID: 25438
00:25:04:WU00:FS01:0x22:        CWD: /var/lib/fahclient/work
00:25:04:WU00:FS01:0x22:********************************************************************************
00:25:04:WU00:FS01:0x22:Project: 13414 (Run 3534, Clone 50, Gen 1)
00:25:04:WU00:FS01:0x22:Unit: 0x0000000512bc7d9a5ef50d675ea5a9cf
00:25:04:WU00:FS01:0x22:Reading tar file core.xml
00:25:04:WU00:FS01:0x22:Reading tar file integrator.xml
00:25:04:WU00:FS01:0x22:Reading tar file state.xml
00:25:04:WU00:FS01:0x22:Reading tar file system.xml
00:25:04:WU00:FS01:0x22:Digital signatures verified
00:25:04:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
00:25:04:WU00:FS01:0x22:Version 0.0.10
00:25:05:WU00:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
00:25:05:WU00:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
00:25:05:WU00:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
00:25:05:WU00:FS01:0x22:  Global context and integrator variables write interval: 250 steps (0.025%) [4000 total]
13:12:12:WU00:FS01:0x22:Caught signal SIGINT(2) on PID 25438
13:12:12:WU00:FS01:0x22:Exiting, please wait. . .
13:13:14:WU00:Sending unit results: id:00 state:SEND error:DUMPED project:13414 run:3534 clone:50 gen:1 core:0x22 unit:0x0000000512bc7d9a5ef50d675ea5a9cf


[bruce]

Do you have a permissions problem with the CWD location?

[zookeeny]

I don't think so - everything under /var/lib/fahclient appears to be owned by user fahclient. Is there a particular file or directory I should verify?

The thing that leads me to believe it's a problem with the 13414 WUs is that everything else seems to work fine, and the few that don't work (bad WUs) exit quickly. Only the 13414 WUs appear to never make progress, and never exit.

[zookeeny]

Just had this happen again... this time locking up my GPU for almost 24 hours before I dumped it:

Code: Select all

15:59:45:WU00:FS01:Connecting to assign1.foldingathome.org:80
15:59:45:WU00:FS01:Assigned to work server 18.188.125.154
15:59:45:WU00:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] from 18.188.125.154
15:59:45:WU00:FS01:Connecting to 18.188.125.154:8080
15:59:45:WU00:FS01:Downloading 7.01MiB
15:59:46:WU00:FS01:Download complete
15:59:47:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13416 run:198 clone:118 gen:0 core:0x22 unit:0x0000000012bc7d9a5f009d1bbfdc71c3
16:00:44:WU00:FS01:Starting
16:00:44:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 00 -suffix 01 -version 706 -lifeline 3144 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
16:00:44:WU00:FS01:Started FahCore on PID 88973
16:00:44:WU00:FS01:Core PID:88977
16:00:44:WU00:FS01:FahCore 0x22 started
16:00:44:WU00:FS01:0x22:*********************** Log Started 2020-07-04T16:00:44Z ***********************
16:00:44:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
16:00:44:WU00:FS01:0x22:       Core: Core22
16:00:44:WU00:FS01:0x22:       Type: 0x22
16:00:44:WU00:FS01:0x22:    Version: 0.0.11
16:00:44:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:00:44:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
16:00:44:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
16:00:44:WU00:FS01:0x22:       Date: Jun 27 2020
16:00:44:WU00:FS01:0x22:       Time: 22:50:00
16:00:44:WU00:FS01:0x22:   Revision: cfc2940c5dd1aa80f60daa6e28d4a2a417f74edb
16:00:44:WU00:FS01:0x22:     Branch: core22-0.0.11
16:00:44:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
16:00:44:WU00:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
16:00:44:WU00:FS01:0x22:             -funroll-loops
16:00:44:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
16:00:44:WU00:FS01:0x22:       Bits: 64
16:00:44:WU00:FS01:0x22:       Mode: Release
16:00:44:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
16:00:44:WU00:FS01:0x22:             <peastman@stanford.edu>
16:00:44:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 88973 -checkpoint 15
16:00:44:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
16:00:44:WU00:FS01:0x22:************************************ libFAH ************************************
16:00:44:WU00:FS01:0x22:       Date: Jun 27 2020
16:00:44:WU00:FS01:0x22:       Time: 22:11:04
16:00:44:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
16:00:44:WU00:FS01:0x22:     Branch: HEAD
16:00:44:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
16:00:44:WU00:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
16:00:44:WU00:FS01:0x22:             -funroll-loops
16:00:44:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
16:00:44:WU00:FS01:0x22:       Bits: 64
16:00:44:WU00:FS01:0x22:       Mode: Release
16:00:44:WU00:FS01:0x22:************************************ CBang *************************************
16:00:44:WU00:FS01:0x22:       Date: Jun 27 2020
16:00:44:WU00:FS01:0x22:       Time: 22:10:11
16:00:44:WU00:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
16:00:44:WU00:FS01:0x22:     Branch: HEAD
16:00:44:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
16:00:44:WU00:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
16:00:44:WU00:FS01:0x22:             -funroll-loops -fPIC
16:00:44:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
16:00:44:WU00:FS01:0x22:       Bits: 64
16:00:44:WU00:FS01:0x22:       Mode: Release
16:00:44:WU00:FS01:0x22:************************************ System ************************************
16:00:44:WU00:FS01:0x22:        CPU: AMD Ryzen 9 3900X 12-Core Processor
16:00:44:WU00:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
16:00:44:WU00:FS01:0x22:       CPUs: 24
16:00:44:WU00:FS01:0x22:     Memory: 31.37GiB
16:00:44:WU00:FS01:0x22:Free Memory: 22.78GiB
16:00:44:WU00:FS01:0x22:    Threads: POSIX_THREADS
16:00:44:WU00:FS01:0x22: OS Version: 5.4
16:00:44:WU00:FS01:0x22:Has Battery: false
16:00:44:WU00:FS01:0x22: On Battery: false
16:00:44:WU00:FS01:0x22: UTC Offset: -4
16:00:44:WU00:FS01:0x22:        PID: 88977
16:00:44:WU00:FS01:0x22:        CWD: /var/lib/fahclient/work
16:00:44:WU00:FS01:0x22:********************************************************************************
16:00:44:WU00:FS01:0x22:Project: 13416 (Run 198, Clone 118, Gen 0)
16:00:44:WU00:FS01:0x22:Unit: 0x0000000012bc7d9a5f009d1bbfdc71c3
16:00:44:WU00:FS01:0x22:Reading tar file core.xml
16:00:44:WU00:FS01:0x22:Reading tar file integrator.xml
16:00:44:WU00:FS01:0x22:Reading tar file state.xml.bz2
16:00:44:WU00:FS01:0x22:Reading tar file system.xml.bz2
16:00:44:WU00:FS01:0x22:Digital signatures verified
16:00:44:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
16:00:44:WU00:FS01:0x22:Version 0.0.11
16:00:44:WU00:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
16:00:44:WU00:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
16:00:44:WU00:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
16:00:44:WU00:FS01:0x22:  Global context and integrator variables write interval: 2500 steps (0.25%) [400 total]
******************************* Date: 2020-07-04 *******************************
******************************* Date: 2020-07-05 *******************************
******************************* Date: 2020-07-05 *******************************
******************************* Date: 2020-07-05 *******************************
15:55:11:WU00:FS01:0x22:Caught signal SIGINT(2) on PID 88977
15:55:11:WU00:FS01:0x22:Exiting, please wait. . .


[zookeeny]

And another...

Code: Select all

20:28:25:WU03:FS01:Connecting to assign1.foldingathome.org:80
20:28:25:WU03:FS01:Assigned to work server 18.188.125.154
20:28:25:WU03:FS01:Requesting new work unit for slot 01: READY gpu:0:Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] from 18.188.125.154
20:28:25:WU03:FS01:Connecting to 18.188.125.154:8080
20:28:25:WU03:FS01:Downloading 7.03MiB
20:28:26:WU03:FS01:Download complete
20:28:26:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13416 run:922 clone:66 gen:0 core:0x22 unit:0x0000000112bc7d9a5f02af90704199b6
20:28:26:WU03:FS01:Starting
20:28:26:WU03:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 03 -suffix 01 -version 706 -lifeline 3293 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:28:26:WU03:FS01:Started FahCore on PID 74670
20:28:26:WU03:FS01:Core PID:74674
20:28:26:WU03:FS01:FahCore 0x22 started
20:28:27:WU03:FS01:0x22:*********************** Log Started 2020-07-07T20:28:26Z ***********************
20:28:27:WU03:FS01:0x22:*************************** Core22 Folding@home Core ***************************
20:28:27:WU03:FS01:0x22:       Core: Core22
20:28:27:WU03:FS01:0x22:       Type: 0x22
20:28:27:WU03:FS01:0x22:    Version: 0.0.11
20:28:27:WU03:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
20:28:27:WU03:FS01:0x22:  Copyright: 2020 foldingathome.org
20:28:27:WU03:FS01:0x22:   Homepage: https://foldingathome.org/
20:28:27:WU03:FS01:0x22:       Date: Jun 27 2020
20:28:27:WU03:FS01:0x22:       Time: 22:50:00
20:28:27:WU03:FS01:0x22:   Revision: cfc2940c5dd1aa80f60daa6e28d4a2a417f74edb
20:28:27:WU03:FS01:0x22:     Branch: core22-0.0.11
20:28:27:WU03:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
20:28:27:WU03:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
20:28:27:WU03:FS01:0x22:             -funroll-loops
20:28:27:WU03:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
20:28:27:WU03:FS01:0x22:       Bits: 64
20:28:27:WU03:FS01:0x22:       Mode: Release
20:28:27:WU03:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
20:28:27:WU03:FS01:0x22:             <peastman@stanford.edu>
20:28:27:WU03:FS01:0x22:       Args: -dir 03 -suffix 01 -version 706 -lifeline 74670 -checkpoint 15
20:28:27:WU03:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:28:27:WU03:FS01:0x22:************************************ libFAH ************************************
20:28:27:WU03:FS01:0x22:       Date: Jun 27 2020
20:28:27:WU03:FS01:0x22:       Time: 22:11:04
20:28:27:WU03:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
20:28:27:WU03:FS01:0x22:     Branch: HEAD
20:28:27:WU03:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
20:28:27:WU03:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
20:28:27:WU03:FS01:0x22:             -funroll-loops
20:28:27:WU03:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
20:28:27:WU03:FS01:0x22:       Bits: 64
20:28:27:WU03:FS01:0x22:       Mode: Release
20:28:27:WU03:FS01:0x22:************************************ CBang *************************************
20:28:27:WU03:FS01:0x22:       Date: Jun 27 2020
20:28:27:WU03:FS01:0x22:       Time: 22:10:11
20:28:27:WU03:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
20:28:27:WU03:FS01:0x22:     Branch: HEAD
20:28:27:WU03:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
20:28:27:WU03:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
20:28:27:WU03:FS01:0x22:             -funroll-loops -fPIC
20:28:27:WU03:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
20:28:27:WU03:FS01:0x22:       Bits: 64
20:28:27:WU03:FS01:0x22:       Mode: Release
20:28:27:WU03:FS01:0x22:************************************ System ************************************
20:28:27:WU03:FS01:0x22:        CPU: AMD Ryzen 9 3900X 12-Core Processor
20:28:27:WU03:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
20:28:27:WU03:FS01:0x22:       CPUs: 24
20:28:27:WU03:FS01:0x22:     Memory: 31.37GiB
20:28:27:WU03:FS01:0x22:Free Memory: 22.09GiB
20:28:27:WU03:FS01:0x22:    Threads: POSIX_THREADS
20:28:27:WU03:FS01:0x22: OS Version: 5.4
20:28:27:WU03:FS01:0x22:Has Battery: false
20:28:27:WU03:FS01:0x22: On Battery: false
20:28:27:WU03:FS01:0x22: UTC Offset: -4
20:28:27:WU03:FS01:0x22:        PID: 74674
20:28:27:WU03:FS01:0x22:        CWD: /var/lib/fahclient/work
20:28:27:WU03:FS01:0x22:********************************************************************************
20:28:27:WU03:FS01:0x22:Project: 13416 (Run 922, Clone 66, Gen 0)
20:28:27:WU03:FS01:0x22:Unit: 0x0000000112bc7d9a5f02af90704199b6
20:28:27:WU03:FS01:0x22:Reading tar file core.xml
20:28:27:WU03:FS01:0x22:Reading tar file integrator.xml
20:28:27:WU03:FS01:0x22:Reading tar file state.xml.bz2
20:28:27:WU03:FS01:0x22:Reading tar file system.xml.bz2
20:28:27:WU03:FS01:0x22:Digital signatures verified
20:28:27:WU03:FS01:0x22:Folding@home GPU Core22 Folding@home Core
20:28:27:WU03:FS01:0x22:Version 0.0.11
20:28:27:WU03:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
20:28:27:WU03:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
20:28:27:WU03:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
20:28:27:WU03:FS01:0x22:  Global context and integrator variables write interval: 2500 steps (0.25%) [400 total]
21:00:27:WU03:FS01:0x22:Caught signal SIGINT(2) on PID 74674
21:00:27:WU03:FS01:0x22:Exiting, please wait. . .
21:01:29:WU03:FS01:FahCore returned: INTERRUPTED (102 = 0x66)


[zookeeny]

One update I'd like to make, in case anyone is still following this - the WUs I included above all had two things in common:

• The log showed no percentage progress at all, not even the initial 0% entry
• FAHControl progress also stayed at 0%, with an ETA of 24:00 hours

Today I just got another 13416 WU [13416 (968, 118, 1)] that is also showing no progress in the log, but is showing progress in FAHControl (currently 25% complete). I don't know why one would show progress and other nothing, but there it is. If this one works it will be the first 13416 WU that wasn't either flagged as FAULTY or DUMPED by me after locking up my GPU. Fingers crossed!

[zookeeny]

Ok, this will be my last update. I allowed the 13416 WU mentioned above to run the full 4 1/2 hours, and now FAHControl indicates that it's 99.99% complete with 1.00 seconds remaining, and it's been stuck like that for the last 15 minutes. I'm done with these WUs.