Dual Radeon VII trouble

It seems that a lot of GPU problems revolve around specific versions of drivers. Though AMD has their own support structure, you can often learn from information reported by others who fold.

Moderators: Site Moderators, FAHC Science Team

Dual Radeon VII trouble

Postby ThWuensche » Fri May 29, 2020 5:32 pm

Hi there,

after running FAH for a few days on a single Radeon VII with AMD ROCM 3.3 I bought and installed a second Radeon VII. I have set up one more slot for the second GPU, two WUs are running. So far good.

But looking at device information from rocm-smi it seems as if both jobs were executed on one of the GPUs only. The power consumption and fan speed of one GPU went up compared to what I was used to before. The second GPU seems idle. Here is my config.xml:

<config>
<!-- Client Control -->
<fold-anon v='true'/>

<!-- Slot Control -->
<power v='MEDIUM'/>

<!-- User Information -->
<passkey v='.....'/>
<team v='.....'/>
<user v='......'/>

<!-- Folding Slots -->
<slot id='0' type='CPU'/>
<slot id='1' type='GPU'>
<gpu-index v='0'/>
<opencl-index v='0'/>
</slot>
<slot id='2' type='GPU'>
<gpu-index v='1'/>
<opencl-index v='1'/>
</slot>
</config>

And thats the output of rocm-smi:

========================ROCm System Management Interface========================
================================================================================
GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU%
0 80.0c 266.0W 1802Mhz 1001Mhz 60.0% auto 250.0W 9% 99%
1 59.0c 18.0W 809Mhz 351Mhz 23.92% auto 250.0W 0% 0%
================================================================================
==============================End of ROCm SMI Log ==============================

The nice thing is that one card can do even more than before :)

But still I would like to put the second GPU to operation. Any ideas?

Regards, Thomas
ThWuensche
 
Posts: 71
Joined: Fri May 29, 2020 5:10 pm

Re: Dual Radeon VII trouble

Postby JimboPalmer » Fri May 29, 2020 6:54 pm

Welcome to Folding@Home!

I would have preferred the entire configuration, but if you can't bring yourself to give all the facts, please do include this part

08:46:42: GPUs: 1
08:46:42: GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 GP107 [GeForce GTX 1050 Ti] 2138
08:46:42: CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:11.0
08:46:42:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:446.14

(This is an Nvidia card, your AMD cards won't have a CUDA driver)

I can promise, the more we know, the better our advice.

The 'little things' like what OS you have, what CPU you have, what client you have, etc. do change our guesses.

viewtopic.php?f=24&t=26036 How to submit a log
Tsar of all the Rushers
I tried to remain childlike, all I achieved was childish.
A friend to those who want no friends
JimboPalmer
 
Posts: 2031
Joined: Mon Feb 16, 2009 5:12 am
Location: Greenwood MS USA

Re: Dual Radeon VII trouble

Postby ThWuensche » Fri May 29, 2020 7:35 pm

Sorry, didn't want to hide any relevant information. I'm running on Debian Buster, CPU is a Ryzen 3700X, 96G of ECC memory.

I did not yet get FAHControl to run, the config was modified manually.

Hope the attached log (biggest part of completion messages removed) includes what is needed for better understanding of the problem.

Code: Select all
*********************** Log Started 2020-05-29T15:52:27Z ***********************
15:52:27:Trying to access database...
15:52:27:Successfully acquired database lock
15:52:27:Read GPUs.txt
15:52:27:Enabled folding slot 00: PAUSED cpu:13 (by user)
15:52:27:Enabled folding slot 01: PAUSED gpu:0:Vega 20 [Radeon VII] (by user)
15:52:27:Enabled folding slot 02: PAUSED gpu:1:Vega 20 [Radeon VII] (by user)
15:52:27:WARNING:WU04:Slot ID 5 no longer exists, migrating to FS00
15:52:27:ERROR:Exception: Unit not found
15:52:27:WARNING:WU05:Slot ID 3 no longer exists, migrating to FS00
15:52:27:ERROR:Exception: Unit not found
15:52:27:****************************** FAHClient ******************************
15:52:27:        Version: 7.6.13
15:52:27:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
15:52:27:      Copyright: 2020 foldingathome.org
15:52:27:       Homepage: https://foldingathome.org/
15:52:27:           Date: Apr 28 2020
15:52:27:           Time: 04:20:16
15:52:27:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
15:52:27:         Branch: master
15:52:27:       Compiler: GNU 8.3.0
15:52:27:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
15:52:27:                 -funroll-loops -fno-pie
15:52:27:       Platform: linux2 4.19.0-5-amd64
15:52:27:           Bits: 64
15:52:27:           Mode: Release
15:52:27:           Args: /etc/fahclient/config.xml
15:52:27:         Config: /etc/fahclient/config.xml
15:52:27:******************************** CBang ********************************
15:52:27:           Date: Apr 25 2020
15:52:27:           Time: 00:07:53
15:52:27:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
15:52:27:         Branch: master
15:52:27:       Compiler: GNU 8.3.0
15:52:27:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
15:52:27:                 -funroll-loops -fno-pie -fPIC
15:52:27:       Platform: linux2 4.19.0-5-amd64
15:52:27:           Bits: 64
15:52:27:           Mode: Release
15:52:27:******************************* System ********************************
15:52:27:            CPU: AMD Ryzen 7 3700X 8-Core Processor
15:52:27:         CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
15:52:27:           CPUs: 16
15:52:27:         Memory: 94.33GiB
15:52:27:    Free Memory: 88.03GiB
15:52:27:        Threads: POSIX_THREADS
15:52:27:     OS Version: 5.4
15:52:27:    Has Battery: false
15:52:27:     On Battery: false
15:52:27:     UTC Offset: 2
15:52:27:            PID: 11024
15:52:27:            CWD: /var/lib/fahclient
15:52:27:             OS: Linux 5.4.0-0.bpo.4-amd64 x86_64
15:52:27:        OS Arch: AMD64
15:52:27:           GPUs: 2
15:52:27:          GPU 0: Bus:6 Slot:0 Func:0 AMD:5 Vega 20 [Radeon VII]
15:52:27:          GPU 1: Bus:13 Slot:0 Func:0 AMD:5 Vega 20 [Radeon VII]
15:52:27:           CUDA: Not detected: Failed to open dynamic library 'libcuda.so':
15:52:27:                 libcuda.so: cannot open shared object file: No such file or
15:52:27:                 directory
15:52:27:OpenCL Device 0: Platform:0 Device:0 Bus:6 Slot:0 Compute:2.0 Driver:3098.0
15:52:27:OpenCL Device 1: Platform:0 Device:1 Bus:13 Slot:0 Compute:2.0 Driver:3098.0
15:52:27:******************************* libFAH ********************************
15:52:27:           Date: Apr 15 2020
15:52:27:           Time: 21:43:24
15:52:27:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
15:52:27:         Branch: master
15:52:27:       Compiler: GNU 8.3.0
15:52:27:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
15:52:27:                 -funroll-loops -fno-pie
15:52:27:       Platform: linux2 4.19.0-5-amd64
15:52:27:           Bits: 64
15:52:27:           Mode: Release
15:52:27:***********************************************************************
15:52:27:<config>
15:52:27:  <!-- Client Control -->
15:52:27:  <fold-anon v='true'/>
15:52:27:
15:52:27:  <!-- Slot Control -->
15:52:27:  <power v='MEDIUM'/>
15:52:27:
15:52:27:  <!-- User Information -->
15:52:27:  <passkey v='*****'/>
15:52:27:  <team v='265730'/>
15:52:27:  <user v='tw@ems-wuensche'/>
15:52:27:
15:52:27:  <!-- Folding Slots -->
15:52:27:  <slot id='0' type='CPU'>
15:52:27:    <paused v='true'/>
15:52:27:  </slot>
15:52:27:  <slot id='1' type='GPU'>
15:52:27:    <gpu-index v='0'/>
15:52:27:    <opencl-index v='0'/>
15:52:27:    <paused v='true'/>
15:52:27:  </slot>
15:52:27:  <slot id='2' type='GPU'>
15:52:27:    <gpu-index v='1'/>
15:52:27:    <opencl-index v='1'/>
15:52:27:    <paused v='true'/>
15:52:27:  </slot>
15:52:27:</config>
15:52:34:18:127.0.0.1:New Web session
15:52:39:FS00:Unpaused
15:52:39:FS01:Unpaused
15:52:39:FS02:Unpaused
15:52:39:WU02:FS02:Starting
15:52:39:WU02:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 02 -suffix 01 -version 706 -lifeline 11024 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 1 -gpu 1
15:52:39:WU02:FS02:Started FahCore on PID 11051
15:52:39:WU02:FS02:Core PID:11055
15:52:39:WU02:FS02:FahCore 0x22 started
15:52:39:WU00:FS01:Starting
15:52:39:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 00 -suffix 01 -version 706 -lifeline 11024 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
15:52:39:WU00:FS01:Started FahCore on PID 11058
15:52:39:WU00:FS01:Core PID:11062
15:52:39:WU00:FS01:FahCore 0x22 started
15:52:39:WU03:FS00:Starting
15:52:39:WARNING:WU03:FS00:Changed SMP threads from 11 to 13 this can cause some work units to fail
15:52:39:WU03:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 03 -suffix 01 -version 706 -lifeline 11024 -checkpoint 15 -np 13
15:52:39:WU03:FS00:Started FahCore on PID 11067
15:52:39:WU03:FS00:Core PID:11071
15:52:39:WU03:FS00:FahCore 0xa7 started
15:52:39:WU02:FS02:0x22:*********************** Log Started 2020-05-29T15:52:39Z ***********************
15:52:39:WU02:FS02:0x22:*************************** Core22 Folding@home Core ***************************
15:52:39:WU02:FS02:0x22:       Type: 0x22
15:52:39:WU02:FS02:0x22:       Core: Core22
15:52:39:WU02:FS02:0x22:    Website: https://foldingathome.org/
15:52:39:WU02:FS02:0x22:  Copyright: (c) 2009-2018 foldingathome.org
15:52:39:WU02:FS02:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
15:52:39:WU02:FS02:0x22:             <rafal.wiewiora@choderalab.org>
15:52:39:WU02:FS02:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 11051 -checkpoint 15
15:52:39:WU02:FS02:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 1 -gpu 1
15:52:39:WU02:FS02:0x22:     Config: <none>
15:52:39:WU02:FS02:0x22:************************************ Build *************************************
15:52:39:WU02:FS02:0x22:    Version: 0.0.5
15:52:39:WU02:FS02:0x22:       Date: Apr 22 2020
15:52:39:WU02:FS02:0x22:       Time: 03:57:11
15:52:39:WU02:FS02:0x22: Repository: Git
15:52:39:WU02:FS02:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
15:52:39:WU02:FS02:0x22:     Branch: HEAD
15:52:39:WU02:FS02:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
15:52:39:WU02:FS02:0x22:    Options: -std=c++11 -O3 -funroll-loops
15:52:39:WU02:FS02:0x22:   Platform: linux2 4.19.76-linuxkit
15:52:39:WU02:FS02:0x22:       Bits: 64
15:52:39:WU02:FS02:0x22:       Mode: Release
15:52:39:WU02:FS02:0x22:************************************ System ************************************
15:52:39:WU02:FS02:0x22:        CPU: AMD Ryzen 7 3700X 8-Core Processor
15:52:39:WU02:FS02:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
15:52:39:WU02:FS02:0x22:       CPUs: 16
15:52:39:WU02:FS02:0x22:     Memory: 94.33GiB
15:52:39:WU02:FS02:0x22:Free Memory: 87.93GiB
15:52:39:WU02:FS02:0x22:    Threads: POSIX_THREADS
15:52:39:WU02:FS02:0x22: OS Version: 5.4
15:52:39:WU02:FS02:0x22:Has Battery: false
15:52:39:WU02:FS02:0x22: On Battery: false
15:52:39:WU02:FS02:0x22: UTC Offset: 2
15:52:39:WU02:FS02:0x22:        PID: 11055
15:52:39:WU02:FS02:0x22:        CWD: /var/lib/fahclient/work
15:52:39:WU02:FS02:0x22:         OS: Linux 5.4.0-0.bpo.4-amd64 x86_64
15:52:39:WU02:FS02:0x22:    OS Arch: AMD64
15:52:39:WU02:FS02:0x22:********************************************************************************
15:52:39:WU02:FS02:0x22:Project: 11760 (Run 0, Clone 5773, Gen 5)
15:52:39:WU02:FS02:0x22:Unit: 0x0000001380fccb0a5e6f0a0a0301f948
15:52:39:WU02:FS02:0x22:Digital signatures verified
15:52:39:WU02:FS02:0x22:Folding@home GPU Core22 Folding@home Core
15:52:39:WU02:FS02:0x22:Version 0.0.5
15:52:39:WU00:FS01:0x22:*********************** Log Started 2020-05-29T15:52:39Z ***********************
15:52:39:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
15:52:39:WU00:FS01:0x22:       Type: 0x22
15:52:39:WU00:FS01:0x22:       Core: Core22
15:52:39:WU00:FS01:0x22:    Website: https://foldingathome.org/
15:52:39:WU00:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
15:52:39:WU00:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
15:52:39:WU00:FS01:0x22:             <rafal.wiewiora@choderalab.org>
15:52:39:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 11058 -checkpoint 15
15:52:39:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
15:52:39:WU00:FS01:0x22:     Config: <none>
15:52:39:WU00:FS01:0x22:************************************ Build *************************************
15:52:39:WU00:FS01:0x22:    Version: 0.0.5
15:52:39:WU00:FS01:0x22:       Date: Apr 22 2020
15:52:39:WU00:FS01:0x22:       Time: 03:57:11
15:52:39:WU00:FS01:0x22: Repository: Git
15:52:39:WU00:FS01:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
15:52:39:WU00:FS01:0x22:     Branch: HEAD
15:52:39:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
15:52:39:WU00:FS01:0x22:    Options: -std=c++11 -O3 -funroll-loops
15:52:39:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
15:52:39:WU00:FS01:0x22:       Bits: 64
15:52:39:WU00:FS01:0x22:       Mode: Release
15:52:39:WU00:FS01:0x22:************************************ System ************************************
15:52:39:WU00:FS01:0x22:        CPU: AMD Ryzen 7 3700X 8-Core Processor
15:52:39:WU00:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
15:52:39:WU00:FS01:0x22:       CPUs: 16
15:52:39:WU00:FS01:0x22:     Memory: 94.33GiB
15:52:39:WU00:FS01:0x22:Free Memory: 87.92GiB
15:52:39:WU00:FS01:0x22:    Threads: POSIX_THREADS
15:52:39:WU00:FS01:0x22: OS Version: 5.4
15:52:39:WU00:FS01:0x22:Has Battery: false
15:52:39:WU00:FS01:0x22: On Battery: false
15:52:39:WU00:FS01:0x22: UTC Offset: 2
15:52:39:WU00:FS01:0x22:        PID: 11062
15:52:39:WU00:FS01:0x22:        CWD: /var/lib/fahclient/work
15:52:39:WU00:FS01:0x22:         OS: Linux 5.4.0-0.bpo.4-amd64 x86_64
15:52:39:WU00:FS01:0x22:    OS Arch: AMD64
15:52:39:WU00:FS01:0x22:********************************************************************************
15:52:39:WU00:FS01:0x22:Project: 14253 (Run 102, Clone 3, Gen 42)
15:52:39:WU00:FS01:0x22:Unit: 0x00000045cedfaa925eab70388ea4326c
15:52:39:WU00:FS01:0x22:Digital signatures verified
15:52:39:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
15:52:39:WU00:FS01:0x22:Version 0.0.5
15:52:39:WU00:FS01:0x22:  Found a checkpoint file
15:52:40:WU03:FS00:0xa7:*********************** Log Started 2020-05-29T15:52:39Z ***********************
15:52:40:WU03:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
15:52:40:WU03:FS00:0xa7:       Type: 0xa7
15:52:40:WU03:FS00:0xa7:       Core: Gromacs
15:52:40:WU03:FS00:0xa7:       Args: -dir 03 -suffix 01 -version 706 -lifeline 11067 -checkpoint 15 -np
15:52:40:WU03:FS00:0xa7:             13
15:52:40:WU03:FS00:0xa7:************************************ CBang *************************************
15:52:40:WU03:FS00:0xa7:       Date: Nov 5 2019
15:52:40:WU03:FS00:0xa7:       Time: 06:06:57
15:52:40:WU03:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
15:52:40:WU03:FS00:0xa7:     Branch: master
15:52:40:WU03:FS00:0xa7:   Compiler: GNU 8.3.0
15:52:40:WU03:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
15:52:40:WU03:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
15:52:40:WU03:FS00:0xa7:       Bits: 64
15:52:40:WU03:FS00:0xa7:       Mode: Release
15:52:40:WU03:FS00:0xa7:************************************ System ************************************
15:52:40:WU03:FS00:0xa7:        CPU: AMD Ryzen 7 3700X 8-Core Processor
15:52:40:WU03:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
15:52:40:WU03:FS00:0xa7:       CPUs: 16
15:52:40:WU03:FS00:0xa7:     Memory: 94.33GiB
15:52:40:WU03:FS00:0xa7:Free Memory: 87.81GiB
15:52:40:WU03:FS00:0xa7:    Threads: POSIX_THREADS
15:52:40:WU03:FS00:0xa7: OS Version: 5.4
15:52:40:WU03:FS00:0xa7:Has Battery: false
15:52:40:WU03:FS00:0xa7: On Battery: false
15:52:40:WU03:FS00:0xa7: UTC Offset: 2
15:52:40:WU03:FS00:0xa7:        PID: 11071
15:52:40:WU03:FS00:0xa7:        CWD: /var/lib/fahclient/work
15:52:40:WU03:FS00:0xa7:******************************** Build - libFAH ********************************
15:52:40:WU03:FS00:0xa7:    Version: 0.0.18
15:52:40:WU03:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
15:52:40:WU03:FS00:0xa7:  Copyright: 2019 foldingathome.org
15:52:40:WU03:FS00:0xa7:   Homepage: https://foldingathome.org/
15:52:40:WU03:FS00:0xa7:       Date: Nov 5 2019
15:52:40:WU03:FS00:0xa7:       Time: 06:13:26
15:52:40:WU03:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
15:52:40:WU03:FS00:0xa7:     Branch: master
15:52:40:WU03:FS00:0xa7:   Compiler: GNU 8.3.0
15:52:40:WU03:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
15:52:40:WU03:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
15:52:40:WU03:FS00:0xa7:       Bits: 64
15:52:40:WU03:FS00:0xa7:       Mode: Release
15:52:40:WU03:FS00:0xa7:************************************ Build *************************************
15:52:40:WU03:FS00:0xa7:       SIMD: avx_256
15:52:40:WU03:FS00:0xa7:********************************************************************************
15:52:40:WU03:FS00:0xa7:Project: 16446 (Run 161, Clone 3, Gen 9)
15:52:40:WU03:FS00:0xa7:Unit: 0x0000000980fccb015eb9fa7423f0efa1
15:52:40:WU03:FS00:0xa7:Digital signatures verified
15:52:40:WU03:FS00:0xa7:Reducing thread count from 13 to 12 to avoid domain decomposition by a prime number > 3
15:52:40:WU03:FS00:0xa7:Calling: mdrun -s frame9.tpr -o frame9.trr -x frame9.xtc -cpi state.cpt -cpt 15 -nt 12
15:52:40:WU03:FS00:0xa7:Steps: first=2250000 total=250000
15:52:41:WU03:FS00:0xa7:Completed 148282 out of 250000 steps (59%)
15:53:15:WU00:FS01:0x22:Completed 180000 out of 500000 steps (36%)
15:53:15:WU00:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
15:53:26:WU03:FS00:0xa7:Completed 150000 out of 250000 steps (60%)
15:53:28:Removing old file 'configs/config-20200522-065454.xml'
15:53:28:Saving configuration to /etc/fahclient/config.xml
15:53:28:<config>
15:53:28:  <!-- Client Control -->
15:53:28:  <fold-anon v='true'/>
15:53:28:
15:53:28:  <!-- Slot Control -->
15:53:28:  <power v='MEDIUM'/>
15:53:28:
15:53:28:  <!-- User Information -->
15:53:28:  <passkey v='*****'/>
15:53:28:  <team v='265730'/>
15:53:28:  <user v='tw@ems-wuensche'/>
15:53:28:
15:53:28:  <!-- Folding Slots -->
15:53:28:  <slot id='0' type='CPU'/>
15:53:28:  <slot id='1' type='GPU'>
15:53:28:    <gpu-index v='0'/>
15:53:28:    <opencl-index v='0'/>
15:53:28:  </slot>
15:53:28:  <slot id='2' type='GPU'>
15:53:28:    <gpu-index v='1'/>
15:53:28:    <opencl-index v='1'/>
15:53:28:  </slot>
15:53:28:</config>
16:34:37:WU01:FS00:Connecting to assign1.foldingathome.org:80
16:34:37:WU01:FS00:Assigned to work server 128.252.203.2
16:34:37:WU01:FS00:Requesting new work unit for slot 00: RUNNING cpu:13 from 128.252.203.2
16:34:37:WU01:FS00:Connecting to 128.252.203.2:8080
16:34:40:WU01:FS00:Downloading 3.78MiB
16:34:43:WU01:FS00:Download complete
16:34:43:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14589 run:0 clone:1726 gen:84 core:0xa7 unit:0x0000006080fccb025e8be039d892e861
16:35:35:WU00:FS01:0x22:Completed 335000 out of 500000 steps (67%)
16:35:39:WU03:FS00:0xa7:Completed 250000 out of 250000 steps (100%)
16:35:40:WU03:FS00:0xa7:Saving result file ../logfile_01.txt
16:35:40:WU03:FS00:0xa7:Saving result file frame9.trr
16:35:40:WU03:FS00:0xa7:Saving result file frame9.xtc
16:35:40:WU03:FS00:0xa7:Saving result file md.log
16:35:40:WU03:FS00:0xa7:Saving result file science.log
16:35:40:WU03:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
16:35:41:WU03:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
16:35:41:WU03:FS00:Sending unit results: id:03 state:SEND error:NO_ERROR project:16446 run:161 clone:3 gen:9 core:0xa7 unit:0x0000000980fccb015eb9fa7423f0efa1
16:35:41:WU03:FS00:Uploading 138.62MiB to 128.252.203.1
16:35:41:WU03:FS00:Connecting to 128.252.203.1:8080
16:35:41:WU01:FS00:Starting
16:35:41:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 706 -lifeline 11024 -checkpoint 15 -np 13
16:35:41:WU01:FS00:Started FahCore on PID 12076
16:35:41:WU01:FS00:Core PID:12080
16:35:41:WU01:FS00:FahCore 0xa7 started
16:35:41:WU01:FS00:0xa7:*********************** Log Started 2020-05-29T16:35:41Z ***********************
16:35:41:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
16:35:41:WU01:FS00:0xa7:       Type: 0xa7
16:35:41:WU01:FS00:0xa7:       Core: Gromacs
16:35:41:WU01:FS00:0xa7:       Args: -dir 01 -suffix 01 -version 706 -lifeline 12076 -checkpoint 15 -np
16:35:41:WU01:FS00:0xa7:             13
16:35:41:WU01:FS00:0xa7:************************************ CBang *************************************
16:35:41:WU01:FS00:0xa7:       Date: Nov 5 2019
16:35:41:WU01:FS00:0xa7:       Time: 06:06:57
16:35:41:WU01:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
16:35:41:WU01:FS00:0xa7:     Branch: master
16:35:41:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
16:35:41:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
16:35:41:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
16:35:41:WU01:FS00:0xa7:       Bits: 64
16:35:41:WU01:FS00:0xa7:       Mode: Release
16:35:41:WU01:FS00:0xa7:************************************ System ************************************
16:35:41:WU01:FS00:0xa7:        CPU: AMD Ryzen 7 3700X 8-Core Processor
16:35:41:WU01:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
16:35:41:WU01:FS00:0xa7:       CPUs: 16
16:35:41:WU01:FS00:0xa7:     Memory: 94.33GiB
16:35:41:WU01:FS00:0xa7:Free Memory: 85.07GiB
16:35:41:WU01:FS00:0xa7:    Threads: POSIX_THREADS
16:35:41:WU01:FS00:0xa7: OS Version: 5.4
16:35:41:WU01:FS00:0xa7:Has Battery: false
16:35:41:WU01:FS00:0xa7: On Battery: false
16:35:41:WU01:FS00:0xa7: UTC Offset: 2
16:35:41:WU01:FS00:0xa7:        PID: 12080
16:35:41:WU01:FS00:0xa7:        CWD: /var/lib/fahclient/work
16:35:41:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
16:35:41:WU01:FS00:0xa7:    Version: 0.0.18
16:35:41:WU01:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:35:41:WU01:FS00:0xa7:  Copyright: 2019 foldingathome.org
16:35:41:WU01:FS00:0xa7:   Homepage: https://foldingathome.org/
16:35:41:WU01:FS00:0xa7:       Date: Nov 5 2019
16:35:41:WU01:FS00:0xa7:       Time: 06:13:26
16:35:41:WU01:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
16:35:41:WU01:FS00:0xa7:     Branch: master
16:35:41:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
16:35:41:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
16:35:41:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
16:35:41:WU01:FS00:0xa7:       Bits: 64
16:35:41:WU01:FS00:0xa7:       Mode: Release
16:35:41:WU01:FS00:0xa7:************************************ Build *************************************
16:35:41:WU01:FS00:0xa7:       SIMD: avx_256
16:35:41:WU01:FS00:0xa7:********************************************************************************
16:35:41:WU01:FS00:0xa7:Project: 14589 (Run 0, Clone 1726, Gen 84)
16:35:41:WU01:FS00:0xa7:Unit: 0x0000006080fccb025e8be039d892e861
16:35:41:WU01:FS00:0xa7:Reading tar file core.xml
16:35:41:WU01:FS00:0xa7:Reading tar file frame84.tpr
16:35:41:WU01:FS00:0xa7:Digital signatures verified
16:35:41:WU01:FS00:0xa7:Reducing thread count from 13 to 12 to avoid domain decomposition by a prime number > 3
16:35:41:WU01:FS00:0xa7:Calling: mdrun -s frame84.tpr -o frame84.trr -x frame84.xtc -cpt 15 -nt 12
16:35:41:WU01:FS00:0xa7:Steps: first=21000000 total=250000
16:35:42:WU01:FS00:0xa7:Completed 1 out of 250000 steps (0%)
16:35:47:WU03:FS00:Upload 1.76%
16:35:53:WU03:FS00:Upload 3.43%
16:35:59:WU03:FS00:Upload 5.05%
16:36:05:WU03:FS00:Upload 6.72%
16:36:11:WU03:FS00:Upload 8.39%
16:36:17:WU03:FS00:Upload 10.01%
16:36:23:WU03:FS00:Upload 11.50%
16:36:29:WU03:FS00:Upload 13.03%
16:36:35:WU03:FS00:Upload 14.74%
16:36:41:WU03:FS00:Upload 16.46%
16:36:47:WU03:FS00:Upload 18.12%
16:36:53:WU03:FS00:Upload 19.57%
16:36:59:WU03:FS00:Upload 21.24%
16:37:05:WU03:FS00:Upload 22.90%
16:37:11:WU03:FS00:Upload 24.17%
16:37:17:WU03:FS00:Upload 25.38%
16:37:23:WU03:FS00:Upload 26.83%
16:37:29:WU03:FS00:Upload 28.45%
16:37:35:WU03:FS00:Upload 30.07%
16:37:41:WU03:FS00:Upload 31.79%
16:37:47:WU03:FS00:Upload 33.41%
16:37:53:WU03:FS00:Upload 34.81%
16:37:59:WU03:FS00:Upload 36.16%
16:38:05:WU03:FS00:Upload 37.96%
16:38:11:WU03:FS00:Upload 39.45%
16:38:17:WU03:FS00:Upload 41.12%
16:38:23:WU03:FS00:Upload 42.70%
16:38:29:WU03:FS00:Upload 44.14%
16:38:35:WU03:FS00:Upload 45.67%
16:38:41:WU03:FS00:Upload 46.98%
16:38:47:WU03:FS00:Upload 48.60%
16:38:53:WU03:FS00:Upload 50.27%
16:38:59:WU03:FS00:Upload 52.03%
16:39:05:WU03:FS00:Upload 53.74%
16:39:11:WU03:FS00:Upload 55.32%
16:39:17:WU03:FS00:Upload 56.94%
16:39:23:WU03:FS00:Upload 58.70%
16:39:29:WU03:FS00:Upload 60.19%
16:39:35:WU03:FS00:Upload 61.86%
16:39:41:WU03:FS00:Upload 63.39%
16:39:47:WU03:FS00:Upload 65.15%
16:39:53:WU03:FS00:Upload 66.82%
16:39:59:WU03:FS00:Upload 68.35%
16:40:05:WU03:FS00:Upload 70.16%
16:40:11:WU03:FS00:Upload 71.82%
16:40:17:WU03:FS00:Upload 72.95%
16:40:23:WU03:FS00:Upload 74.66%
16:40:29:WU03:FS00:Upload 76.29%
16:40:35:WU03:FS00:Upload 78.00%
16:40:41:WU03:FS00:Upload 79.62%
16:40:47:WU03:FS00:Upload 80.98%
16:40:53:WU03:FS00:Upload 82.33%
16:40:59:WU03:FS00:Upload 83.64%
16:41:05:WU03:FS00:Upload 85.17%
16:41:11:WU03:FS00:Upload 86.84%
16:41:17:WU03:FS00:Upload 88.37%
16:41:23:WU03:FS00:Upload 89.77%
16:41:29:WU03:FS00:Upload 91.21%
16:41:35:WU03:FS00:Upload 92.52%
16:41:41:WU03:FS00:Upload 94.01%
16:41:47:WU03:FS00:Upload 95.49%
16:41:53:WU03:FS00:Upload 97.03%
16:41:59:WU03:FS00:Upload 98.20%
16:42:05:WU03:FS00:Upload 99.64%
16:42:07:WU03:FS00:Upload complete
16:42:07:WU03:FS00:Server responded WORK_ACK (400)
16:42:07:WU03:FS00:Final credit estimate, 10375.00 points
16:42:07:WU03:FS00:Cleaning up
17:19:22:WU03:FS01:Connecting to assign1.foldingathome.org:80
17:19:23:WARNING:WU03:FS01:Failed to get assignment from 'assign1.foldingathome.org:80': No WUs available for this configuration
17:19:23:WU03:FS01:Connecting to assign2.foldingathome.org:80
17:19:23:WARNING:WU03:FS01:Failed to get assignment from 'assign2.foldingathome.org:80': No WUs available for this configuration
17:19:23:WU03:FS01:Connecting to assign3.foldingathome.org:80
17:19:24:WU03:FS01:Assigned to work server 128.252.203.10
17:19:24:WU03:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:Vega 20 [Radeon VII] from 128.252.203.10
17:19:24:WU03:FS01:Connecting to 128.252.203.10:8080
17:20:37:WU03:FS01:Downloading 51.49MiB
17:20:43:WU03:FS01:Download 9.35%
17:20:49:WU03:FS01:Download 21.36%
17:20:49:WU00:FS01:0x22:Saving result file ../logfile_01.txt
17:20:49:WU00:FS01:0x22:Saving result file checkpointState.xml
17:20:55:WU03:FS01:Download 31.68%
17:20:55:WU00:FS01:0x22:Saving result file checkpt.crc
17:20:55:WU00:FS01:0x22:Saving result file positions.xtc
17:20:55:WU00:FS01:0x22:Saving result file science.log
17:20:55:WU00:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
17:20:55:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
17:20:55:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14253 run:102 clone:3 gen:42 core:0x22 unit:0x00000045cedfaa925eab70388ea4326c
17:20:55:WU00:FS01:Uploading 39.83MiB to 206.223.170.146
17:20:55:WU00:FS01:Connecting to 206.223.170.146:8080
17:21:01:WU03:FS01:Download 39.81%
17:21:01:WU00:FS01:Upload 6.12%
17:21:01:WU01:FS00:0xa7:Completed 152500 out of 250000 steps (61%)
17:21:07:WU03:FS01:Download 46.13%
17:21:07:WU00:FS01:Upload 12.71%
17:21:13:WU03:FS01:Download 51.47%
17:21:13:WU00:FS01:Upload 19.30%
17:21:19:WU00:FS01:Upload 25.89%
17:21:19:WU03:FS01:Download 57.17%
17:21:25:WU03:FS01:Download 65.43%
17:21:25:WU00:FS01:Upload 32.80%
17:21:31:WU03:FS01:Download 73.56%
17:21:31:WU00:FS01:Upload 39.39%
17:21:37:WU03:FS01:Download 81.33%
17:21:37:WU00:FS01:Upload 46.45%
17:21:43:WU03:FS01:Download 87.88%
17:21:43:WU00:FS01:Upload 49.59%
17:21:45:WU01:FS00:0xa7:Completed 155000 out of 250000 steps (62%)
17:21:49:WU03:FS01:Download 93.59%
17:21:49:WU00:FS01:Upload 55.87%
17:21:55:WU03:FS01:Download 99.66%
17:21:55:WU00:FS01:Upload 61.67%
17:21:55:WU03:FS01:Download complete
17:21:55:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:11762 run:0 clone:2982 gen:56 core:0x22 unit:0x0000007b80fccb0a5e6d80f4a9acd54a
17:21:55:WU03:FS01:Starting
17:21:55:WU03:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 03 -suffix 01 -version 706 -lifeline 11024 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
17:21:55:WU03:FS01:Started FahCore on PID 13573
17:21:55:WU03:FS01:Core PID:13577
17:21:55:WU03:FS01:FahCore 0x22 started
17:21:56:WU03:FS01:0x22:*********************** Log Started 2020-05-29T17:21:55Z ***********************
17:21:56:WU03:FS01:0x22:*************************** Core22 Folding@home Core ***************************
17:21:56:WU03:FS01:0x22:       Type: 0x22
17:21:56:WU03:FS01:0x22:       Core: Core22
17:21:56:WU03:FS01:0x22:    Website: https://foldingathome.org/
17:21:56:WU03:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
17:21:56:WU03:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
17:21:56:WU03:FS01:0x22:             <rafal.wiewiora@choderalab.org>
17:21:56:WU03:FS01:0x22:       Args: -dir 03 -suffix 01 -version 706 -lifeline 13573 -checkpoint 15
17:21:56:WU03:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
17:21:56:WU03:FS01:0x22:     Config: <none>
17:21:56:WU03:FS01:0x22:************************************ Build *************************************
17:21:56:WU03:FS01:0x22:    Version: 0.0.5
17:21:56:WU03:FS01:0x22:       Date: Apr 22 2020
17:21:56:WU03:FS01:0x22:       Time: 03:57:11
17:21:56:WU03:FS01:0x22: Repository: Git
17:21:56:WU03:FS01:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
17:21:56:WU03:FS01:0x22:     Branch: HEAD
17:21:56:WU03:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
17:21:56:WU03:FS01:0x22:    Options: -std=c++11 -O3 -funroll-loops
17:21:56:WU03:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
17:21:56:WU03:FS01:0x22:       Bits: 64
17:21:56:WU03:FS01:0x22:       Mode: Release
17:21:56:WU03:FS01:0x22:************************************ System ************************************
17:21:56:WU03:FS01:0x22:        CPU: AMD Ryzen 7 3700X 8-Core Processor
17:21:56:WU03:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
17:21:56:WU03:FS01:0x22:       CPUs: 16
17:21:56:WU03:FS01:0x22:     Memory: 94.33GiB
17:21:56:WU03:FS01:0x22:Free Memory: 87.22GiB
17:21:56:WU03:FS01:0x22:    Threads: POSIX_THREADS
17:21:56:WU03:FS01:0x22: OS Version: 5.4
17:21:56:WU03:FS01:0x22:Has Battery: false
17:21:56:WU03:FS01:0x22: On Battery: false
17:21:56:WU03:FS01:0x22: UTC Offset: 2
17:21:56:WU03:FS01:0x22:        PID: 13577
17:21:56:WU03:FS01:0x22:        CWD: /var/lib/fahclient/work
17:21:56:WU03:FS01:0x22:         OS: Linux 5.4.0-0.bpo.4-amd64 x86_64
17:21:56:WU03:FS01:0x22:    OS Arch: AMD64
17:21:56:WU03:FS01:0x22:********************************************************************************
17:21:56:WU03:FS01:0x22:Project: 11762 (Run 0, Clone 2982, Gen 56)
17:21:56:WU03:FS01:0x22:Unit: 0x0000007b80fccb0a5e6d80f4a9acd54a
17:21:56:WU03:FS01:0x22:Reading tar file core.xml
17:21:56:WU03:FS01:0x22:Reading tar file integrator.xml
17:21:56:WU03:FS01:0x22:Reading tar file state.xml
17:21:56:WU03:FS01:0x22:Reading tar file system.xml
17:21:56:WU03:FS01:0x22:Digital signatures verified
17:21:56:WU03:FS01:0x22:Folding@home GPU Core22 Folding@home Core
17:21:56:WU03:FS01:0x22:Version 0.0.5
17:22:01:WU00:FS01:Upload 68.26%
17:22:07:WU00:FS01:Upload 74.23%
17:22:11:WU03:FS01:0x22:Completed 0 out of 1000000 steps (0%)
17:22:11:WU03:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
17:22:13:WU00:FS01:Upload 81.13%
17:22:19:WU00:FS01:Upload 87.72%
17:22:25:WU00:FS01:Upload 93.21%
17:22:29:WU01:FS00:0xa7:Completed 157500 out of 250000 steps (63%)
17:22:31:WU00:FS01:Upload 99.80%
17:22:33:WU00:FS01:Upload complete
17:22:33:WU00:FS01:Server responded WORK_ACK (400)
17:22:33:WU00:FS01:Final credit estimate, 148543.00 points
17:22:33:WU00:FS01:Cleaning up
17:49:01:WU00:FS00:Connecting to assign1.foldingathome.org:80
17:49:02:WU00:FS00:Assigned to work server 40.114.52.201
17:49:02:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:13 from 40.114.52.201
17:49:02:WU00:FS00:Connecting to 40.114.52.201:8080
17:49:02:WU00:FS00:Downloading 2.85MiB
17:49:05:WU00:FS00:Download complete
17:49:05:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14582 run:0 clone:1524 gen:200 core:0xa7 unit:0x000000e8287234c95e7a3a49f7688b38
17:49:46:WU01:FS00:0xa7:Saving result file ../logfile_01.txt
17:49:46:WU01:FS00:0xa7:Saving result file frame84.trr
17:49:46:WU01:FS00:0xa7:Saving result file frame84.xtc
17:49:46:WU01:FS00:0xa7:Saving result file md.log
17:49:46:WU01:FS00:0xa7:Saving result file science.log
17:49:46:WU01:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
17:49:46:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
17:49:46:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14589 run:0 clone:1726 gen:84 core:0xa7 unit:0x0000006080fccb025e8be039d892e861
17:49:46:WU01:FS00:Uploading 3.20MiB to 128.252.203.2
17:49:46:WU01:FS00:Connecting to 128.252.203.2:8080
17:49:46:WU00:FS00:Starting
17:49:46:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 00 -suffix 01 -version 706 -lifeline 11024 -checkpoint 15 -np 13
17:49:46:WU00:FS00:Started FahCore on PID 14040
17:49:46:WU00:FS00:Core PID:14044
17:49:46:WU00:FS00:FahCore 0xa7 started
17:49:47:WU00:FS00:0xa7:*********************** Log Started 2020-05-29T17:49:46Z ***********************
17:49:47:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
17:49:47:WU00:FS00:0xa7:       Type: 0xa7
17:49:47:WU00:FS00:0xa7:       Core: Gromacs
17:49:47:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 14040 -checkpoint 15 -np
17:49:47:WU00:FS00:0xa7:             13
17:49:47:WU00:FS00:0xa7:************************************ CBang *************************************
17:49:47:WU00:FS00:0xa7:       Date: Nov 5 2019
17:49:47:WU00:FS00:0xa7:       Time: 06:06:57
17:49:47:WU00:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
17:49:47:WU00:FS00:0xa7:     Branch: master
17:49:47:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
17:49:47:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
17:49:47:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
17:49:47:WU00:FS00:0xa7:       Bits: 64
17:49:47:WU00:FS00:0xa7:       Mode: Release
17:49:47:WU00:FS00:0xa7:************************************ System ************************************
17:49:47:WU00:FS00:0xa7:        CPU: AMD Ryzen 7 3700X 8-Core Processor
17:49:47:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
17:49:47:WU00:FS00:0xa7:       CPUs: 16
17:49:47:WU00:FS00:0xa7:     Memory: 94.33GiB
17:49:47:WU00:FS00:0xa7:Free Memory: 86.83GiB
17:49:47:WU00:FS00:0xa7:    Threads: POSIX_THREADS
17:49:47:WU00:FS00:0xa7: OS Version: 5.4
17:49:47:WU00:FS00:0xa7:Has Battery: false
17:49:47:WU00:FS00:0xa7: On Battery: false
17:49:47:WU00:FS00:0xa7: UTC Offset: 2
17:49:47:WU00:FS00:0xa7:        PID: 14044
17:49:47:WU00:FS00:0xa7:        CWD: /var/lib/fahclient/work
17:49:47:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
17:49:47:WU00:FS00:0xa7:    Version: 0.0.18
17:49:47:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:49:47:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
17:49:47:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
17:49:47:WU00:FS00:0xa7:       Date: Nov 5 2019
17:49:47:WU00:FS00:0xa7:       Time: 06:13:26
17:49:47:WU00:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
17:49:47:WU00:FS00:0xa7:     Branch: master
17:49:47:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
17:49:47:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
17:49:47:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
17:49:47:WU00:FS00:0xa7:       Bits: 64
17:49:47:WU00:FS00:0xa7:       Mode: Release
17:49:47:WU00:FS00:0xa7:************************************ Build *************************************
17:49:47:WU00:FS00:0xa7:       SIMD: avx_256
17:49:47:WU00:FS00:0xa7:********************************************************************************
17:49:47:WU00:FS00:0xa7:Project: 14582 (Run 0, Clone 1524, Gen 200)
17:49:47:WU00:FS00:0xa7:Unit: 0x000000e8287234c95e7a3a49f7688b38
17:49:47:WU00:FS00:0xa7:Reading tar file core.xml
17:49:47:WU00:FS00:0xa7:Reading tar file frame200.tpr
17:49:47:WU00:FS00:0xa7:Digital signatures verified
17:49:47:WU00:FS00:0xa7:Reducing thread count from 13 to 12 to avoid domain decomposition by a prime number > 3
17:49:47:WU00:FS00:0xa7:Calling: mdrun -s frame200.tpr -o frame200.trr -x frame200.xtc -cpt 15 -nt 12
17:49:47:WU00:FS00:0xa7:Steps: first=50000000 total=250000
17:49:47:WU00:FS00:0xa7:Completed 1 out of 250000 steps (0%)
17:49:52:WU01:FS00:Upload complete
17:49:52:WU01:FS00:Server responded WORK_ACK (400)
17:49:52:WU01:FS00:Final credit estimate, 8971.00 points
17:49:52:WU01:FS00:Cleaning up

ThWuensche
 
Posts: 71
Joined: Fri May 29, 2020 5:10 pm

Re: Dual Radeon VII trouble

Postby ThWuensche » Fri May 29, 2020 8:01 pm

One more observation: Though the progress bar for GPU:1 was moving, there are no completion messages in the log. Also the bar is now standing at 99,99%, seemingly nothing uploaded, no new WUs loaded.
ThWuensche
 
Posts: 71
Joined: Fri May 29, 2020 5:10 pm

Re: Dual Radeon VII trouble

Postby JimboPalmer » Fri May 29, 2020 8:09 pm

Not the issue of discussion, but
15:52:40:WU03:FS00:0xa7:Reducing thread count from 13 to 12 to avoid domain decomposition by a prime number > 3

You may wish to set CPUs to 12.

To my eye, your GPU setups look OK. Your Driver is higher than I am used to, I usually see in the low 3000s

15:52:27:OpenCL Device 0: Platform:0 Device:0 Bus:6 Slot:0 Compute:2.0 Driver:3098.0

I usually see 3004 or 3005, are you sure these are the drivers from the AMD site? (I am in over my head, I have not run an AMD card more recent than a HD 2600 Pro)
JimboPalmer
 
Posts: 2031
Joined: Mon Feb 16, 2009 5:12 am
Location: Greenwood MS USA

Re: Dual Radeon VII trouble

Postby foldy » Fri May 29, 2020 8:24 pm

2nd GPU folding slot is stuck on starting the new work unit.

15:52:39:WU02:FS02:0x22:Folding@home GPU Core22 Folding@home Core
15:52:39:WU02:FS02:0x22:Version 0.0.5

This is not a configuration issue but the stuck work unit of FS02 needs to get removed.

I would recommend to remove the 2nd GPU slots again. Close and Restart FahClient. Add 2nd GPU slot again. Close and Restart FahClient.
foldy
 
Posts: 2006
Joined: Sat Dec 01, 2012 4:43 pm

Re: Dual Radeon VII trouble

Postby Joe_H » Fri May 29, 2020 8:36 pm

ThWuensche wrote:One more observation: Though the progress bar for GPU:1 was moving, there are no completion messages in the log. Also the bar is now standing at 99,99%, seemingly nothing uploaded, no new WUs loaded.


Adding new hardware can sometimes take uninstalling the F@h client along with the data files and reinstalling after the hardware and its drivers have been installed. Or you might get things squared away with just a reboot of your Linux system

The bar going to 99.9% and staying there often indicates a driver crash and reset. From the section of log posted, I can see a WU started, but no progress past this point:
Code: Select all
17:22:11:WU03:FS01:0x22:Completed 0 out of 1000000 steps (0%)


I also did not see any log messages for the second GPU folding slot defined in the last configuration shown in the log.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Joe_H
Site Admin
 
Posts: 6578
Joined: Tue Apr 21, 2009 5:41 pm
Location: W. MA

Re: Dual Radeon VII trouble

Postby ThWuensche » Fri May 29, 2020 9:04 pm

The driver 3098.0 comes from the ROCM open source computing package from AMD, combined with the kernel part from Linux standard kernel. I have this package installed, as lately I was using the card for neural network training with tensorflow. I'm choosing AMD open source components here, since I don't like the NVidia closed source policy with their cards and the Radeon VII is the only affordable GPU with 16G memory for the NN training task.

As for the progress in the logs: After stopping, removing the second GPU, restarting, stopping, adding second GPU (as suggested) I now also have no progress in the log for GPU:0. But in the web interface the progress bars for both GPUs are moving.

I will wait and see what happens if they get to the end (which is still a few hours).
ThWuensche
 
Posts: 71
Joined: Fri May 29, 2020 5:10 pm

Re: Dual Radeon VII trouble

Postby _r2w_ben » Sat May 30, 2020 12:59 am

The work unit for slot 2 looks like it's hung. I would recommend pausing the slot, deleting /var/lib/fahclient/work/02/ and then unpausing the slot. It should grab a fresh work unit and hopefully progress.
_r2w_ben
 
Posts: 281
Joined: Wed Apr 23, 2008 4:11 pm

Re: Dual Radeon VII trouble

Postby bruce » Sat May 30, 2020 2:42 am

Note that at this time, you had slots 3 and 5 which had WUs enqueued on them the slots had been deleted. When that happens, the WUs are moved to the first slot that matches the hardware that had been deleted, so two more WUs were moved to slot 0. They're now enqueued on slot 0 and will be processed one-at-a-time.
Code: Select all
15:52:27:Enabled folding slot 00: PAUSED cpu:13 (by user)
15:52:27:Enabled folding slot 01: PAUSED gpu:0:Vega 20 [Radeon VII] (by user)
15:52:27:Enabled folding slot 02: PAUSED gpu:1:Vega 20 [Radeon VII] (by user)
15:52:27:WARNING:WU04:Slot ID 5 no longer exists, migrating to FS00
15:52:27:ERROR:Exception: Unit not found
15:52:27:WARNING:WU05:Slot ID 3 no longer exists, migrating to FS00


You're making a lot of unnecessary edits to your Config. That's actually causing problems that you wouldn't have if you let FAHClient configure things for you.
Code: Select all
[93m15:52:39:WARNING:WU03:FS00:Changed SMP threads from 11 to 13 this can cause some work units to fail


Note that FAH fixed it for you by changing it to 12 CPUs.
Code: Select all
15:52:40:WU03:FS00:0xa7:Reducing thread count from 13 to 12 to avoid domain decomposition by a prime number >


You had a slot that was configured for 11 threads and you changed it to 13. Neither 11 nor 13 are valid configurations since they're prime numbers.
bruce
 
Posts: 19970
Joined: Thu Nov 29, 2007 11:13 pm
Location: So. Cal.

Re: Dual Radeon VII trouble

Postby foldy » Sat May 30, 2020 10:55 am

Auto configuration by FAH after reinstall works most of the time.
foldy
 
Posts: 2006
Joined: Sat Dec 01, 2012 4:43 pm

Re: Dual Radeon VII trouble

Postby ThWuensche » Sat May 30, 2020 6:33 pm

You're making a lot of unnecessary edits to your Config. That's actually causing problems that you wouldn't have if you let FAHClient configure things for you.
Code: Select all
[93m15:52:39:WARNING:WU03:FS00:Changed SMP threads from 11 to 13 this can cause some work units to fail


Note that FAH fixed it for you by changing it to 12 CPUs.
Code: Select all
15:52:40:WU03:FS00:0xa7:Reducing thread count from 13 to 12 to avoid domain decomposition by a prime number >


You had a slot that was configured for 11 threads and you changed it to 13. Neither 11 nor 13 are valid configurations since they're prime numbers.



I have no configuration of thread count in my config.xml. The change in thread count was done by FAH client without any prior specification by me.
ThWuensche
 
Posts: 71
Joined: Fri May 29, 2020 5:10 pm

Re: Dual Radeon VII trouble

Postby ThWuensche » Sat May 30, 2020 6:37 pm

The dual GPU trouble seems to be related to the driver, not FAH. I can find dumps in /var/log/messages, will try to sort that out with AMD ROCm guys.

Thank you for you comments, which took away doubts about my configuration.
ThWuensche
 
Posts: 71
Joined: Fri May 29, 2020 5:10 pm

Re: Dual Radeon VII trouble

Postby bruce » Sat May 30, 2020 6:44 pm

I know nothing about AMD driver installation, but for NVidia drivers I think I remember that you have to reinstall the drivers AFTER both devices are connected. It's worth a try. if you haven't already done that.
bruce
 
Posts: 19970
Joined: Thu Nov 29, 2007 11:13 pm
Location: So. Cal.

Re: Dual Radeon VII trouble

Postby ThWuensche » Sat May 30, 2020 9:04 pm

OK, I think I found a solution, it is linux/hardware related. Setting

iommu=soft

as kernel parameter in grub default did the trick. Now two Radeon VII are going to support the project. Hope this outcome helps also others to provide GPU computing power to the project.

In short again my software configuration:

debian buster with kernel 5.4/5.5 from backports
amdgpu driver from linux kernel
rocm 3.3 for opencl

Thanks a lot for the support!
ThWuensche
 
Posts: 71
Joined: Fri May 29, 2020 5:10 pm

Next

Return to Problems with AMD/ATI drivers

Who is online

Users browsing this forum: Google [Bot] and 3 guests

cron