Project: 16438 Decomposition Fail

[RunningTurtle]

As requested, here is my log showing Project: 16438 (Run 0, Clone 253, Gen 148) trying to run on a 15 core client.

Code: Select all

21:57:12:WU00:FS00:Connecting to 65.254.110.245:8080
21:57:12:WU00:FS00:Assigned to work server 128.252.203.1
21:57:12:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:15 from 128.252.203.1
21:57:12:WU00:FS00:Connecting to 128.252.203.1:8080
21:57:12:WU00:FS00:Downloading 1.33MiB
21:57:12:WU00:FS00:Download complete
21:57:12:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:16438 run:0 clone:253 gen:148 core:0xa7 unit:0x000000a480fccb015ea74d902f940057
21:58:18:WU00:FS00:Starting
21:58:18:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 00 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -np 15
21:58:18:WU00:FS00:Started FahCore on PID 4313
21:58:18:WU00:FS00:Core PID:4317
21:58:18:WU00:FS00:FahCore 0xa7 started
21:58:18:WU00:FS00:0xa7:*********************** Log Started 2020-06-01T21:58:18Z ***********************
21:58:18:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
21:58:18:WU00:FS00:0xa7:       Type: 0xa7
21:58:18:WU00:FS00:0xa7:       Core: Gromacs
21:58:18:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 705 -lifeline 4313 -checkpoint 15 -np
21:58:18:WU00:FS00:0xa7:             15
21:58:18:WU00:FS00:0xa7:************************************ CBang *************************************
21:58:18:WU00:FS00:0xa7:       Date: Nov 5 2019
21:58:18:WU00:FS00:0xa7:       Time: 06:06:57
21:58:18:WU00:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
21:58:18:WU00:FS00:0xa7:     Branch: master
21:58:18:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
21:58:18:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
21:58:18:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
21:58:18:WU00:FS00:0xa7:       Bits: 64
21:58:18:WU00:FS00:0xa7:       Mode: Release
21:58:18:WU00:FS00:0xa7:************************************ System ************************************
21:58:18:WU00:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
21:58:18:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
21:58:18:WU00:FS00:0xa7:       CPUs: 16
21:58:18:WU00:FS00:0xa7:     Memory: 125.81GiB
21:58:18:WU00:FS00:0xa7:Free Memory: 122.61GiB
21:58:18:WU00:FS00:0xa7:    Threads: POSIX_THREADS
21:58:18:WU00:FS00:0xa7: OS Version: 4.19
21:58:18:WU00:FS00:0xa7:Has Battery: false
21:58:18:WU00:FS00:0xa7: On Battery: false
21:58:18:WU00:FS00:0xa7: UTC Offset: -4
21:58:18:WU00:FS00:0xa7:        PID: 4317
21:58:18:WU00:FS00:0xa7:        CWD: /var/lib/fahclient/work
21:58:18:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
21:58:18:WU00:FS00:0xa7:    Version: 0.0.18
21:58:18:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
21:58:18:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
21:58:18:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
21:58:18:WU00:FS00:0xa7:       Date: Nov 5 2019
21:58:18:WU00:FS00:0xa7:       Time: 06:13:26
21:58:18:WU00:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
21:58:18:WU00:FS00:0xa7:     Branch: master
21:58:18:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
21:58:18:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
21:58:18:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
21:58:18:WU00:FS00:0xa7:       Bits: 64
21:58:18:WU00:FS00:0xa7:       Mode: Release
21:58:18:WU00:FS00:0xa7:************************************ Build *************************************
21:58:18:WU00:FS00:0xa7:       SIMD: avx_256
21:58:18:WU00:FS00:0xa7:********************************************************************************
21:58:18:WU00:FS00:0xa7:Project: 16438 (Run 0, Clone 253, Gen 148)
21:58:18:WU00:FS00:0xa7:Unit: 0x000000a480fccb015ea74d902f940057
21:58:18:WU00:FS00:0xa7:Reading tar file core.xml
21:58:18:WU00:FS00:0xa7:Reading tar file frame148.tpr
21:58:18:WU00:FS00:0xa7:Digital signatures verified
21:58:18:WU00:FS00:0xa7:Calling: mdrun -s frame148.tpr -o frame148.trr -x frame148.xtc -cpt 15 -nt 15
21:58:18:WU00:FS00:0xa7:Steps: first=74000000 total=500000
21:58:18:WU00:FS00:0xa7:ERROR:
21:58:18:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
21:58:18:WU00:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20191026-456f0d636-unknown
21:58:18:WU00:FS00:0xa7:ERROR:Source code file: /host/debian-stable-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/mdlib/domdec.c, line: 6902
21:58:18:WU00:FS00:0xa7:ERROR:
21:58:18:WU00:FS00:0xa7:ERROR:Fatal error:
21:58:18:WU00:FS00:0xa7:ERROR:There is no domain decomposition for 15 ranks that is compatible with the given box and a minimum cell size of 1.39789 nm
21:58:18:WU00:FS00:0xa7:ERROR:Change the number of ranks or mdrun option -rcon or -dds or your LINCS settings
21:58:18:WU00:FS00:0xa7:ERROR:Look in the log file for details on the domain decomposition
21:58:18:WU00:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
21:58:18:WU00:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
21:58:18:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
21:58:23:WU00:FS00:0xa7:WARNING:Unexpected exit() call
21:58:23:WU00:FS00:0xa7:WARNING:Unexpected exit from science code
21:58:23:WU00:FS00:0xa7:Saving result file ../logfile_01.txt
21:58:23:WU00:FS00:0xa7:Saving result file md.log
21:58:23:WU00:FS00:0xa7:Saving result file science.log
21:58:23:WU00:FS00:FahCore returned: INTERRUPTED (102 = 0x66)

[PantherX]

Thanks for your report, the researcher has been notified of this

[foldy]

As a workaround you could limit CPU slot threads to 14

[bruce]

For some unknown reason, you were assigned that WU. FAHClient is supposed to reduce the number of CPUs to 15 > 14 > 13 > 12 where it would run successfully. GROMACS will produce decomposition errors at certain numbers of threads and the client is supposed to correct for them.

Please post the first couple of pages of FAH's log.

Did you just add a GPU?

[RunningTurtle]

Did you just add a GPU?

No. I built these machines late March just for FAH. The FAH configuration has been running as-is (15 CPU only cores, 1 GPU+1 CPU core) since then. The only software change since then 21.May, when I did an apt-get upgrade. The nvidia driver changed from 418.74 to 418.113, along with various library upgrades for bug/security fixes.

After the upgrade the machine continued for 9 days, successfully processing CPU and GPU workunits, until it was assigned the 16438 project. For a little more than 3 days, the CPU continued to try to run the 16438 workunit, until I limited the CPU to 12 cores.

Please post the first couple of pages of FAH's log.

First 900 lines attached (removed workunit progress entries 1-99%)"

Code: Select all

cat /var/lib/fahclient/log.txt | egrep -v 'Completed.*out\ of.*\steps \([1-9]|[1-9][0-9]\%\)' | head -n 900
*********************** Log Started 2020-05-21T16:19:33Z ***********************
16:19:33:************************* Folding@home Client *************************
16:19:33:        Website: https://foldingathome.org/
16:19:33:      Copyright: (c) 2009-2018 foldingathome.org
16:19:33:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:19:33:           Args: --child --lifeline 887 /etc/fahclient/config.xml --run-as
16:19:33:                 fahclient --pid-file=/var/run/fahclient.pid --daemon
16:19:33:         Config: /etc/fahclient/config.xml
16:19:33:******************************** Build ********************************
16:19:33:        Version: 7.5.1
16:19:33:           Date: May 11 2018
16:19:33:           Time: 19:59:04
16:19:33:     Repository: Git
16:19:33:       Revision: 4705bf53c635f88b8fe85af7675557e15d491ff0
16:19:33:         Branch: master
16:19:33:       Compiler: GNU 6.3.0 20170516
16:19:33:        Options: -std=gnu++98 -O3 -funroll-loops
16:19:33:       Platform: linux2 4.14.0-3-amd64
16:19:33:           Bits: 64
16:19:33:           Mode: Release
16:19:33:******************************* System ********************************
16:19:33:            CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
16:19:33:         CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
16:19:33:           CPUs: 16
16:19:33:         Memory: 125.81GiB
16:19:33:    Free Memory: 125.34GiB
16:19:33:        Threads: POSIX_THREADS
16:19:33:     OS Version: 4.19
16:19:33:    Has Battery: false
16:19:33:     On Battery: false
16:19:33:     UTC Offset: -4
16:19:33:            PID: 889
16:19:33:            CWD: /var/lib/fahclient
16:19:33:             OS: Linux 4.19.0-8-amd64 x86_64
16:19:33:        OS Arch: AMD64
16:19:33:           GPUs: 1
16:19:33:          GPU 0: Bus:4 Slot:0 Func:0 NVIDIA:5 GM206GL [Quadro M2000]
16:19:33:  CUDA Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:5.2 Driver:10.1
16:19:33:OpenCL Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:1.2 Driver:418.113
16:19:33:***********************************************************************
16:19:33:<config>
16:19:33:  <!-- HTTP Server -->
16:19:33:  <allow v=' 127.0.0.1 172.28.0.24'/>
16:19:33:
16:19:33:  <!-- Network -->
16:19:33:  <proxy v=':8080'/>
16:19:33:
16:19:33:  <!-- Remote Command Server -->
16:19:33:  <password v='**********************'/>
16:19:33:
16:19:33:  <!-- Slot Control -->
16:19:33:  <power v='full'/>
16:19:33:
16:19:33:  <!-- User Information -->
16:19:33:  <passkey v='********************************'/>
16:19:33:  <user v='RunningTurtle'/>
16:19:33:
16:19:33:  <!-- Folding Slots -->
16:19:33:  <slot id='0' type='CPU'/>
16:19:33:  <slot id='1' type='GPU'/>
16:19:33:</config>
16:19:33:Switching to user fahclient
16:19:33:Trying to access database...
16:19:33:Successfully acquired database lock
16:19:33:Enabled folding slot 00: READY cpu:15
16:19:33:Enabled folding slot 01: READY gpu:0:GM206GL [Quadro M2000]
16:19:33:ERROR:WU00:FS00:Exception: Could not get IP address for assign1.foldingathome.org: Temporary failure in name resolution
16:19:33:ERROR:WU00:FS00:Exception: Could not get IP address for assign2.foldingathome.org: Temporary failure in name resolution
16:19:33:WARNING:WU00:FS00:Exception: Failed to find any IP addresses for assignment servers
16:19:33:ERROR:WU00:FS00:Exception: Could not get an assignment
16:19:33:ERROR:WU01:FS01:Exception: Could not get IP address for assign1.foldingathome.org: Temporary failure in name resolution
16:19:33:ERROR:WU01:FS01:Exception: Could not get IP address for assign2.foldingathome.org: Temporary failure in name resolution
16:19:33:WARNING:WU01:FS01:Exception: Failed to find any IP addresses for assignment servers
16:19:33:ERROR:WU01:FS01:Exception: Could not get an assignment
16:19:34:ERROR:WU00:FS00:Exception: Could not get IP address for assign1.foldingathome.org: Temporary failure in name resolution
16:19:34:ERROR:WU00:FS00:Exception: Could not get IP address for assign2.foldingathome.org: Temporary failure in name resolution
16:19:34:WARNING:WU00:FS00:Exception: Failed to find any IP addresses for assignment servers
16:19:34:ERROR:WU00:FS00:Exception: Could not get an assignment
16:19:34:ERROR:WU01:FS01:Exception: Could not get IP address for assign1.foldingathome.org: Temporary failure in name resolution
16:19:34:ERROR:WU01:FS01:Exception: Could not get IP address for assign2.foldingathome.org: Temporary failure in name resolution
16:19:34:WARNING:WU01:FS01:Exception: Failed to find any IP addresses for assignment servers
16:19:34:ERROR:WU01:FS01:Exception: Could not get an assignment
16:20:34:WU00:FS00:Connecting to 65.254.110.245:8080
16:20:34:WU00:FS00:Assigned to work server 130.237.11.145
16:20:34:WU00:FS00:Requesting new work unit for slot 00: READY cpu:15 from 130.237.11.145
16:20:34:WU00:FS00:Connecting to 130.237.11.145:8080
16:20:34:WU01:FS01:Connecting to 65.254.110.245:8080
16:20:34:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
16:20:34:WU01:FS01:Connecting to 18.218.241.186:80
16:20:34:WU00:FS00:Downloading 4.31MiB
16:20:35:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
16:20:35:ERROR:WU01:FS01:Exception: Could not get an assignment
16:20:36:WU00:FS00:Download complete
16:20:36:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:16805 run:8 clone:817 gen:2 core:0xa7 unit:0x0000000282ed0b915eb4230050ae05c2
16:20:36:WU00:FS00:Starting
16:20:36:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 00 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -np 15
16:20:36:WU00:FS00:Started FahCore on PID 1298
16:20:36:WU00:FS00:Core PID:1302
16:20:36:WU00:FS00:FahCore 0xa7 started
16:20:36:WU00:FS00:0xa7:*********************** Log Started 2020-05-21T16:20:36Z ***********************
16:20:36:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
16:20:36:WU00:FS00:0xa7:       Type: 0xa7
16:20:36:WU00:FS00:0xa7:       Core: Gromacs
16:20:36:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 705 -lifeline 1298 -checkpoint 15 -np
16:20:36:WU00:FS00:0xa7:             15
16:20:36:WU00:FS00:0xa7:************************************ CBang *************************************
16:20:36:WU00:FS00:0xa7:       Date: Nov 5 2019
16:20:36:WU00:FS00:0xa7:       Time: 06:06:57
16:20:36:WU00:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
16:20:36:WU00:FS00:0xa7:     Branch: master
16:20:36:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
16:20:36:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
16:20:36:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
16:20:36:WU00:FS00:0xa7:       Bits: 64
16:20:36:WU00:FS00:0xa7:       Mode: Release
16:20:36:WU00:FS00:0xa7:************************************ System ************************************
16:20:36:WU00:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
16:20:36:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
16:20:36:WU00:FS00:0xa7:       CPUs: 16
16:20:36:WU00:FS00:0xa7:     Memory: 125.81GiB
16:20:36:WU00:FS00:0xa7:Free Memory: 125.29GiB
16:20:36:WU00:FS00:0xa7:    Threads: POSIX_THREADS
16:20:36:WU00:FS00:0xa7: OS Version: 4.19
16:20:36:WU00:FS00:0xa7:Has Battery: false
16:20:36:WU00:FS00:0xa7: On Battery: false
16:20:36:WU00:FS00:0xa7: UTC Offset: -4
16:20:36:WU00:FS00:0xa7:        PID: 1302
16:20:36:WU00:FS00:0xa7:        CWD: /var/lib/fahclient/work
16:20:36:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
16:20:36:WU00:FS00:0xa7:    Version: 0.0.18
16:20:36:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:20:36:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
16:20:36:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
16:20:36:WU00:FS00:0xa7:       Date: Nov 5 2019
16:20:36:WU00:FS00:0xa7:       Time: 06:13:26
16:20:36:WU00:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
16:20:36:WU00:FS00:0xa7:     Branch: master
16:20:36:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
16:20:36:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
16:20:36:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
16:20:36:WU00:FS00:0xa7:       Bits: 64
16:20:36:WU00:FS00:0xa7:       Mode: Release
16:20:36:WU00:FS00:0xa7:************************************ Build *************************************
16:20:36:WU00:FS00:0xa7:       SIMD: avx_256
16:20:36:WU00:FS00:0xa7:********************************************************************************
16:20:36:WU00:FS00:0xa7:Project: 16805 (Run 8, Clone 817, Gen 2)
16:20:36:WU00:FS00:0xa7:Unit: 0x0000000282ed0b915eb4230050ae05c2
16:20:36:WU00:FS00:0xa7:Reading tar file core.xml
16:20:36:WU00:FS00:0xa7:Reading tar file frame2.tpr
16:20:36:WU00:FS00:0xa7:Digital signatures verified
16:20:36:WU00:FS00:0xa7:Calling: mdrun -s frame2.tpr -o frame2.trr -cpt 15 -nt 15
16:20:36:WU00:FS00:0xa7:Steps: first=1000000 total=500000
16:20:41:WU00:FS00:0xa7:Completed 1 out of 500000 steps (0%)
16:22:11:WU01:FS01:Connecting to 65.254.110.245:8080
16:22:11:WU01:FS01:Assigned to work server 128.252.203.1
16:22:11:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GM206GL [Quadro M2000] from 128.252.203.1
16:22:11:WU01:FS01:Connecting to 128.252.203.1:8080
16:22:13:WU01:FS01:Downloading 6.28MiB
16:22:14:WU01:FS01:Download complete
16:22:15:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:16444 run:0 clone:1788 gen:18 core:0x22 unit:0x0000002580fccb015eaa0014ab9ca778
16:22:15:WU01:FS01:Starting
16:22:15:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 01 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
16:22:15:WU01:FS01:Started FahCore on PID 2517
16:22:15:WU01:FS01:Core PID:2521
16:22:15:WU01:FS01:FahCore 0x22 started
16:22:15:WU01:FS01:0x22:*********************** Log Started 2020-05-21T16:22:15Z ***********************
16:22:15:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
16:22:15:WU01:FS01:0x22:       Type: 0x22
16:22:15:WU01:FS01:0x22:       Core: Core22
16:22:15:WU01:FS01:0x22:    Website: https://foldingathome.org/
16:22:15:WU01:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
16:22:15:WU01:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
16:22:15:WU01:FS01:0x22:             <rafal.wiewiora@choderalab.org>
16:22:15:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 705 -lifeline 2517 -checkpoint 15
16:22:15:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
16:22:15:WU01:FS01:0x22:             0 -gpu 0
16:22:15:WU01:FS01:0x22:     Config: <none>
16:22:15:WU01:FS01:0x22:************************************ Build *************************************
16:22:15:WU01:FS01:0x22:    Version: 0.0.5
16:22:15:WU01:FS01:0x22:       Date: Apr 22 2020
16:22:15:WU01:FS01:0x22:       Time: 03:57:11
16:22:15:WU01:FS01:0x22: Repository: Git
16:22:15:WU01:FS01:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
16:22:15:WU01:FS01:0x22:     Branch: HEAD
16:22:15:WU01:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
16:22:15:WU01:FS01:0x22:    Options: -std=c++11 -O3 -funroll-loops
16:22:15:WU01:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
16:22:15:WU01:FS01:0x22:       Bits: 64
16:22:15:WU01:FS01:0x22:       Mode: Release
16:22:15:WU01:FS01:0x22:************************************ System ************************************
16:22:15:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
16:22:15:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
16:22:15:WU01:FS01:0x22:       CPUs: 16
16:22:15:WU01:FS01:0x22:     Memory: 125.81GiB
16:22:15:WU01:FS01:0x22:Free Memory: 125.00GiB
16:22:15:WU01:FS01:0x22:    Threads: POSIX_THREADS
16:22:15:WU01:FS01:0x22: OS Version: 4.19
16:22:15:WU01:FS01:0x22:Has Battery: false
16:22:15:WU01:FS01:0x22: On Battery: false
16:22:15:WU01:FS01:0x22: UTC Offset: -4
16:22:15:WU01:FS01:0x22:        PID: 2521
16:22:15:WU01:FS01:0x22:        CWD: /var/lib/fahclient/work
16:22:15:WU01:FS01:0x22:         OS: Linux 4.19.0-8-amd64 x86_64
16:22:15:WU01:FS01:0x22:    OS Arch: AMD64
16:22:15:WU01:FS01:0x22:********************************************************************************
16:22:15:WU01:FS01:0x22:Project: 16444 (Run 0, Clone 1788, Gen 18)
16:22:15:WU01:FS01:0x22:Unit: 0x0000002580fccb015eaa0014ab9ca778
16:22:15:WU01:FS01:0x22:Reading tar file core.xml
16:22:15:WU01:FS01:0x22:Reading tar file integrator.xml
16:22:15:WU01:FS01:0x22:Reading tar file state.xml
16:22:15:WU01:FS01:0x22:Reading tar file system.xml
16:22:16:WU01:FS01:0x22:Digital signatures verified
16:22:16:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
16:22:16:WU01:FS01:0x22:Version 0.0.5
16:22:35:WU01:FS01:0x22:Completed 0 out of 5000000 steps (0%)
16:22:35:WU01:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
16:24:10:WU00:FS00:0xa7:Completed 5000 out of 500000 steps (1%)
16:27:38:WU00:FS00:0xa7:Completed 10000 out of 500000 steps (2%)
16:31:03:WU00:FS00:0xa7:Completed 15000 out of 500000 steps (3%)
16:32:40:WU01:FS01:0x22:Completed 50000 out of 5000000 steps (1%)
16:34:27:WU00:FS00:0xa7:Completed 20000 out of 500000 steps (4%)
16:37:53:WU00:FS00:0xa7:Completed 25000 out of 500000 steps (5%)
16:41:15:WU00:FS00:0xa7:Completed 30000 out of 500000 steps (6%)
16:42:54:WU01:FS01:0x22:Completed 100000 out of 5000000 steps (2%)
16:44:36:WU00:FS00:0xa7:Completed 35000 out of 500000 steps (7%)
16:47:57:WU00:FS00:0xa7:Completed 40000 out of 500000 steps (8%)
16:51:19:WU00:FS00:0xa7:Completed 45000 out of 500000 steps (9%)
16:53:13:WU01:FS01:0x22:Completed 150000 out of 5000000 steps (3%)
17:03:27:WU01:FS01:0x22:Completed 200000 out of 5000000 steps (4%)
17:13:45:WU01:FS01:0x22:Completed 250000 out of 5000000 steps (5%)
17:23:59:WU01:FS01:0x22:Completed 300000 out of 5000000 steps (6%)
17:34:17:WU01:FS01:0x22:Completed 350000 out of 5000000 steps (7%)
17:44:32:WU01:FS01:0x22:Completed 400000 out of 5000000 steps (8%)
17:54:49:WU01:FS01:0x22:Completed 450000 out of 5000000 steps (9%)
21:52:37:WU02:FS00:Connecting to 65.254.110.245:8080
21:52:37:WU02:FS00:Assigned to work server 128.252.203.9
21:52:37:WU02:FS00:Requesting new work unit for slot 00: RUNNING cpu:15 from 128.252.203.9
21:52:37:WU02:FS00:Connecting to 128.252.203.9:8080
21:55:35:WU02:FS00:Downloading 4.59MiB
21:55:35:WU02:FS00:Download complete
21:55:36:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13832 run:0 clone:106 gen:59 core:0xa7 unit:0x0000005980fccb095e6d1f5bc2523434
21:55:56:WU00:FS00:0xa7:Completed 500000 out of 500000 steps (100%)
21:56:00:WU00:FS00:0xa7:Saving result file ../logfile_01.txt
21:56:00:WU00:FS00:0xa7:Saving result file ener.edr
21:56:00:WU00:FS00:0xa7:Saving result file frame2.trr
21:56:00:WU00:FS00:0xa7:Saving result file md.log
21:56:00:WU00:FS00:0xa7:Saving result file science.log
21:56:00:WU00:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
21:56:01:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
21:56:01:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16805 run:8 clone:817 gen:2 core:0xa7 unit:0x0000000282ed0b915eb4230050ae05c2
21:56:01:WU00:FS00:Uploading 13.85MiB to 130.237.11.145
21:56:01:WU00:FS00:Connecting to 130.237.11.145:8080
21:56:01:WU02:FS00:Starting
21:56:01:WU02:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 02 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -np 15
21:56:01:WU02:FS00:Started FahCore on PID 15174
21:56:01:WU02:FS00:Core PID:15178
21:56:01:WU02:FS00:FahCore 0xa7 started
21:56:02:WU02:FS00:0xa7:*********************** Log Started 2020-05-21T21:56:01Z ***********************
21:56:02:WU02:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
21:56:02:WU02:FS00:0xa7:       Type: 0xa7
21:56:02:WU02:FS00:0xa7:       Core: Gromacs
21:56:02:WU02:FS00:0xa7:       Args: -dir 02 -suffix 01 -version 705 -lifeline 15174 -checkpoint 15 -np
21:56:02:WU02:FS00:0xa7:             15
21:56:02:WU02:FS00:0xa7:************************************ CBang *************************************
21:56:02:WU02:FS00:0xa7:       Date: Nov 5 2019
21:56:02:WU02:FS00:0xa7:       Time: 06:06:57
21:56:02:WU02:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
21:56:02:WU02:FS00:0xa7:     Branch: master
21:56:02:WU02:FS00:0xa7:   Compiler: GNU 8.3.0
21:56:02:WU02:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
21:56:02:WU02:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
21:56:02:WU02:FS00:0xa7:       Bits: 64
21:56:02:WU02:FS00:0xa7:       Mode: Release
21:56:02:WU02:FS00:0xa7:************************************ System ************************************
21:56:02:WU02:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
21:56:02:WU02:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
21:56:02:WU02:FS00:0xa7:       CPUs: 16
21:56:02:WU02:FS00:0xa7:     Memory: 125.81GiB
21:56:02:WU02:FS00:0xa7:Free Memory: 124.61GiB
21:56:02:WU02:FS00:0xa7:    Threads: POSIX_THREADS
21:56:02:WU02:FS00:0xa7: OS Version: 4.19
21:56:02:WU02:FS00:0xa7:Has Battery: false
21:56:02:WU02:FS00:0xa7: On Battery: false
21:56:02:WU02:FS00:0xa7: UTC Offset: -4
21:56:02:WU02:FS00:0xa7:        PID: 15178
21:56:02:WU02:FS00:0xa7:        CWD: /var/lib/fahclient/work
21:56:02:WU02:FS00:0xa7:******************************** Build - libFAH ********************************
21:56:02:WU02:FS00:0xa7:    Version: 0.0.18
21:56:02:WU02:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
21:56:02:WU02:FS00:0xa7:  Copyright: 2019 foldingathome.org
21:56:02:WU02:FS00:0xa7:   Homepage: https://foldingathome.org/
21:56:02:WU02:FS00:0xa7:       Date: Nov 5 2019
21:56:02:WU02:FS00:0xa7:       Time: 06:13:26
21:56:02:WU02:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
21:56:02:WU02:FS00:0xa7:     Branch: master
21:56:02:WU02:FS00:0xa7:   Compiler: GNU 8.3.0
21:56:02:WU02:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
21:56:02:WU02:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
21:56:02:WU02:FS00:0xa7:       Bits: 64
21:56:02:WU02:FS00:0xa7:       Mode: Release
21:56:02:WU02:FS00:0xa7:************************************ Build *************************************
21:56:02:WU02:FS00:0xa7:       SIMD: avx_256
21:56:02:WU02:FS00:0xa7:********************************************************************************
21:56:02:WU02:FS00:0xa7:Project: 13832 (Run 0, Clone 106, Gen 59)
21:56:02:WU02:FS00:0xa7:Unit: 0x0000005980fccb095e6d1f5bc2523434
21:56:02:WU02:FS00:0xa7:Reading tar file core.xml
21:56:02:WU02:FS00:0xa7:Reading tar file frame59.tpr
21:56:02:WU02:FS00:0xa7:Digital signatures verified
21:56:02:WU02:FS00:0xa7:Calling: mdrun -s frame59.tpr -o frame59.trr -x frame59.xtc -cpt 15 -nt 15
21:56:02:WU02:FS00:0xa7:Steps: first=14750000 total=250000
21:56:06:WU02:FS00:0xa7:Completed 1 out of 250000 steps (0%)
21:56:07:WU00:FS00:Upload 30.69%
21:56:13:WU00:FS00:Upload 49.20%
21:56:19:WU00:FS00:Upload 71.32%
21:56:25:WU00:FS00:Upload 90.28%
21:56:31:WU00:FS00:Upload complete
21:56:31:WU00:FS00:Server responded WORK_ACK (400)
21:56:31:WU00:FS00:Final credit estimate, 19676.00 points
21:56:31:WU00:FS00:Cleaning up
21:57:27:WU02:FS00:0xa7:Completed 2500 out of 250000 steps (1%)
21:58:46:WU02:FS00:0xa7:Completed 5000 out of 250000 steps (2%)
22:00:04:WU02:FS00:0xa7:Completed 7500 out of 250000 steps (3%)
22:01:22:WU02:FS00:0xa7:Completed 10000 out of 250000 steps (4%)
22:02:39:WU02:FS00:0xa7:Completed 12500 out of 250000 steps (5%)
22:03:57:WU02:FS00:0xa7:Completed 15000 out of 250000 steps (6%)
22:05:15:WU02:FS00:0xa7:Completed 17500 out of 250000 steps (7%)
22:06:33:WU02:FS00:0xa7:Completed 20000 out of 250000 steps (8%)
22:07:51:WU02:FS00:0xa7:Completed 22500 out of 250000 steps (9%)
******************************* Date: 2020-05-21 *******************************
00:04:47:WU00:FS00:Connecting to 65.254.110.245:8080
00:04:47:WU00:FS00:Assigned to work server 128.252.203.1
00:04:47:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:15 from 128.252.203.1
00:04:47:WU00:FS00:Connecting to 128.252.203.1:8080
00:04:48:WU00:FS00:Downloading 2.50MiB
00:04:48:WU00:FS00:Download complete
00:04:48:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:16437 run:0 clone:1804 gen:42 core:0xa7 unit:0x0000003180fccb015ea74dc6048d6148
00:06:03:WU02:FS00:0xa7:Completed 250000 out of 250000 steps (100%)
00:06:07:WU02:FS00:0xa7:Saving result file ../logfile_01.txt
00:06:07:WU02:FS00:0xa7:Saving result file frame59.trr
00:06:07:WU02:FS00:0xa7:Saving result file frame59.xtc
00:06:07:WU02:FS00:0xa7:Saving result file md.log
00:06:07:WU02:FS00:0xa7:Saving result file science.log
00:06:07:WU02:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
00:06:08:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
00:06:08:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:13832 run:0 clone:106 gen:59 core:0xa7 unit:0x0000005980fccb095e6d1f5bc2523434
00:06:08:WU02:FS00:Uploading 9.60MiB to 128.252.203.9
00:06:08:WU02:FS00:Connecting to 128.252.203.9:8080
00:06:08:WU00:FS00:Starting
00:06:08:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 00 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -np 15
00:06:08:WU00:FS00:Started FahCore on PID 15366
00:06:08:WU00:FS00:Core PID:15370
00:06:08:WU00:FS00:FahCore 0xa7 started
00:06:08:WU00:FS00:0xa7:*********************** Log Started 2020-05-22T00:06:08Z ***********************
00:06:08:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
00:06:08:WU00:FS00:0xa7:       Type: 0xa7
00:06:08:WU00:FS00:0xa7:       Core: Gromacs
00:06:08:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 705 -lifeline 15366 -checkpoint 15 -np
00:06:08:WU00:FS00:0xa7:             15
00:06:08:WU00:FS00:0xa7:************************************ CBang *************************************
00:06:08:WU00:FS00:0xa7:       Date: Nov 5 2019
00:06:08:WU00:FS00:0xa7:       Time: 06:06:57
00:06:08:WU00:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
00:06:08:WU00:FS00:0xa7:     Branch: master
00:06:08:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
00:06:08:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
00:06:08:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
00:06:08:WU00:FS00:0xa7:       Bits: 64
00:06:08:WU00:FS00:0xa7:       Mode: Release
00:06:08:WU00:FS00:0xa7:************************************ System ************************************
00:06:08:WU00:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
00:06:08:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
00:06:08:WU00:FS00:0xa7:       CPUs: 16
00:06:08:WU00:FS00:0xa7:     Memory: 125.81GiB
00:06:08:WU00:FS00:0xa7:Free Memory: 124.54GiB
00:06:08:WU00:FS00:0xa7:    Threads: POSIX_THREADS
00:06:08:WU00:FS00:0xa7: OS Version: 4.19
00:06:08:WU00:FS00:0xa7:Has Battery: false
00:06:08:WU00:FS00:0xa7: On Battery: false
00:06:08:WU00:FS00:0xa7: UTC Offset: -4
00:06:08:WU00:FS00:0xa7:        PID: 15370
00:06:08:WU00:FS00:0xa7:        CWD: /var/lib/fahclient/work
00:06:08:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
00:06:08:WU00:FS00:0xa7:    Version: 0.0.18
00:06:08:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
00:06:08:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
00:06:08:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
00:06:08:WU00:FS00:0xa7:       Date: Nov 5 2019
00:06:08:WU00:FS00:0xa7:       Time: 06:13:26
00:06:08:WU00:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
00:06:08:WU00:FS00:0xa7:     Branch: master
00:06:08:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
00:06:08:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
00:06:08:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
00:06:08:WU00:FS00:0xa7:       Bits: 64
00:06:08:WU00:FS00:0xa7:       Mode: Release
00:06:08:WU00:FS00:0xa7:************************************ Build *************************************
00:06:08:WU00:FS00:0xa7:       SIMD: avx_256
00:06:08:WU00:FS00:0xa7:********************************************************************************
00:06:08:WU00:FS00:0xa7:Project: 16437 (Run 0, Clone 1804, Gen 42)
00:06:08:WU00:FS00:0xa7:Unit: 0x0000003180fccb015ea74dc6048d6148
00:06:08:WU00:FS00:0xa7:Reading tar file core.xml
00:06:08:WU00:FS00:0xa7:Reading tar file frame42.tpr
00:06:08:WU00:FS00:0xa7:Digital signatures verified
00:06:08:WU00:FS00:0xa7:Calling: mdrun -s frame42.tpr -o frame42.trr -x frame42.xtc -cpt 15 -nt 15
00:06:08:WU00:FS00:0xa7:Steps: first=10500000 total=250000
00:06:09:WU00:FS00:0xa7:Completed 1 out of 250000 steps (0%)
00:07:04:WU00:FS00:0xa7:Completed 2500 out of 250000 steps (1%)
00:07:49:WU02:FS00:Upload 12.37%
00:07:55:WU02:FS00:Upload 82.69%
00:08:00:WU00:FS00:0xa7:Completed 5000 out of 250000 steps (2%)
00:08:55:WU00:FS00:0xa7:Completed 7500 out of 250000 steps (3%)
00:09:40:WU02:FS00:Upload complete
00:09:40:WU02:FS00:Server responded WORK_ACK (400)
00:09:40:WU02:FS00:Final credit estimate, 6090.00 points
00:09:40:WU02:FS00:Cleaning up
00:09:49:WU00:FS00:0xa7:Completed 10000 out of 250000 steps (4%)
00:10:44:WU00:FS00:0xa7:Completed 12500 out of 250000 steps (5%)
00:11:40:WU00:FS00:0xa7:Completed 15000 out of 250000 steps (6%)
00:12:34:WU00:FS00:0xa7:Completed 17500 out of 250000 steps (7%)
00:13:29:WU00:FS00:0xa7:Completed 20000 out of 250000 steps (8%)
00:14:24:WU00:FS00:0xa7:Completed 22500 out of 250000 steps (9%)
01:37:06:WU02:FS00:Connecting to 65.254.110.245:8080
01:37:06:WU02:FS00:Assigned to work server 40.114.52.201
01:37:06:WU02:FS00:Requesting new work unit for slot 00: RUNNING cpu:15 from 40.114.52.201
01:37:06:WU02:FS00:Connecting to 40.114.52.201:8080
01:37:06:WU02:FS00:Downloading 2.14MiB
01:37:06:WU02:FS00:Download complete
01:37:06:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13850 run:0 clone:38518 gen:127 core:0xa7 unit:0x00000095287234c95e78894f6fb219bb
01:38:00:WU00:FS00:0xa7:Completed 250000 out of 250000 steps (100%)
01:38:02:WU00:FS00:0xa7:Saving result file ../logfile_01.txt
01:38:02:WU00:FS00:0xa7:Saving result file frame42.trr
01:38:03:WU00:FS00:0xa7:Saving result file frame42.xtc
01:38:03:WU00:FS00:0xa7:Saving result file md.log
01:38:03:WU00:FS00:0xa7:Saving result file science.log
01:38:03:WU00:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
01:38:03:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
01:38:03:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16437 run:0 clone:1804 gen:42 core:0xa7 unit:0x0000003180fccb015ea74dc6048d6148
01:38:03:WU00:FS00:Uploading 2.12MiB to 128.252.203.1
01:38:03:WU00:FS00:Connecting to 128.252.203.1:8080
01:38:03:WU02:FS00:Starting
01:38:03:WU02:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 02 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -np 15
01:38:03:WU02:FS00:Started FahCore on PID 15483
01:38:03:WU02:FS00:Core PID:15487
01:38:03:WU02:FS00:FahCore 0xa7 started
01:38:04:WU02:FS00:0xa7:*********************** Log Started 2020-05-22T01:38:03Z ***********************
01:38:04:WU02:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
01:38:04:WU02:FS00:0xa7:       Type: 0xa7
01:38:04:WU02:FS00:0xa7:       Core: Gromacs
01:38:04:WU02:FS00:0xa7:       Args: -dir 02 -suffix 01 -version 705 -lifeline 15483 -checkpoint 15 -np
01:38:04:WU02:FS00:0xa7:             15
01:38:04:WU02:FS00:0xa7:************************************ CBang *************************************
01:38:04:WU02:FS00:0xa7:       Date: Nov 5 2019
01:38:04:WU02:FS00:0xa7:       Time: 06:06:57
01:38:04:WU02:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
01:38:04:WU02:FS00:0xa7:     Branch: master
01:38:04:WU02:FS00:0xa7:   Compiler: GNU 8.3.0
01:38:04:WU02:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
01:38:04:WU02:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
01:38:04:WU02:FS00:0xa7:       Bits: 64
01:38:04:WU02:FS00:0xa7:       Mode: Release
01:38:04:WU02:FS00:0xa7:************************************ System ************************************
01:38:04:WU02:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
01:38:04:WU02:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
01:38:04:WU02:FS00:0xa7:       CPUs: 16
01:38:04:WU02:FS00:0xa7:     Memory: 125.81GiB
01:38:04:WU02:FS00:0xa7:Free Memory: 124.56GiB
01:38:04:WU02:FS00:0xa7:    Threads: POSIX_THREADS
01:38:04:WU02:FS00:0xa7: OS Version: 4.19
01:38:04:WU02:FS00:0xa7:Has Battery: false
01:38:04:WU02:FS00:0xa7: On Battery: false
01:38:04:WU02:FS00:0xa7: UTC Offset: -4
01:38:04:WU02:FS00:0xa7:        PID: 15487
01:38:04:WU02:FS00:0xa7:        CWD: /var/lib/fahclient/work
01:38:04:WU02:FS00:0xa7:******************************** Build - libFAH ********************************
01:38:04:WU02:FS00:0xa7:    Version: 0.0.18
01:38:04:WU02:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
01:38:04:WU02:FS00:0xa7:  Copyright: 2019 foldingathome.org
01:38:04:WU02:FS00:0xa7:   Homepage: https://foldingathome.org/
01:38:04:WU02:FS00:0xa7:       Date: Nov 5 2019
01:38:04:WU02:FS00:0xa7:       Time: 06:13:26
01:38:04:WU02:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
01:38:04:WU02:FS00:0xa7:     Branch: master
01:38:04:WU02:FS00:0xa7:   Compiler: GNU 8.3.0
01:38:04:WU02:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
01:38:04:WU02:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
01:38:04:WU02:FS00:0xa7:       Bits: 64
01:38:04:WU02:FS00:0xa7:       Mode: Release
01:38:04:WU02:FS00:0xa7:************************************ Build *************************************
01:38:04:WU02:FS00:0xa7:       SIMD: avx_256
01:38:04:WU02:FS00:0xa7:********************************************************************************
01:38:04:WU02:FS00:0xa7:Project: 13850 (Run 0, Clone 38518, Gen 127)
01:38:04:WU02:FS00:0xa7:Unit: 0x00000095287234c95e78894f6fb219bb
01:38:04:WU02:FS00:0xa7:Reading tar file core.xml
01:38:04:WU02:FS00:0xa7:Reading tar file frame127.tpr
01:38:04:WU02:FS00:0xa7:Digital signatures verified
01:38:04:WU02:FS00:0xa7:Calling: mdrun -s frame127.tpr -o frame127.trr -x frame127.xtc -e frame127.edr -cpt 15 -nt 15
01:38:04:WU02:FS00:0xa7:Steps: first=63500000 total=500000
01:38:05:WU00:FS00:Upload complete
01:38:05:WU00:FS00:Server responded WORK_ACK (400)
01:38:05:WU00:FS00:Final credit estimate, 4482.00 points
01:38:06:WU00:FS00:Cleaning up
01:38:06:WU02:FS00:0xa7:Completed 1 out of 500000 steps (0%)
01:39:32:WU02:FS00:0xa7:Completed 5000 out of 500000 steps (1%)
01:40:59:WU02:FS00:0xa7:Completed 10000 out of 500000 steps (2%)
01:42:25:WU02:FS00:0xa7:Completed 15000 out of 500000 steps (3%)
01:43:50:WU02:FS00:0xa7:Completed 20000 out of 500000 steps (4%)
01:45:17:WU02:FS00:0xa7:Completed 25000 out of 500000 steps (5%)
01:46:43:WU02:FS00:0xa7:Completed 30000 out of 500000 steps (6%)
01:48:09:WU02:FS00:0xa7:Completed 35000 out of 500000 steps (7%)
01:49:35:WU02:FS00:0xa7:Completed 40000 out of 500000 steps (8%)
01:51:01:WU02:FS00:0xa7:Completed 45000 out of 500000 steps (9%)
03:59:43:WU00:FS00:Connecting to 65.254.110.245:8080
03:59:43:WU00:FS00:Assigned to work server 128.252.203.1
03:59:43:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:15 from 128.252.203.1
03:59:43:WU00:FS00:Connecting to 128.252.203.1:8080
03:59:43:WU00:FS00:Downloading 2.84MiB
03:59:44:WU00:FS00:Download complete
03:59:44:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:16415 run:906 clone:0 gen:36 core:0xa7 unit:0x0000002980fccb015ea74f945a761064
04:01:08:WU02:FS00:0xa7:Completed 500000 out of 500000 steps (100%)
04:01:10:WU02:FS00:0xa7:Saving result file ../logfile_01.txt
04:01:10:WU02:FS00:0xa7:Saving result file frame127.edr
04:01:10:WU02:FS00:0xa7:Saving result file frame127.trr
04:01:10:WU02:FS00:0xa7:Saving result file frame127.xtc
04:01:10:WU02:FS00:0xa7:Saving result file md.log
04:01:10:WU02:FS00:0xa7:Saving result file science.log
04:01:10:WU02:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
04:01:11:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
04:01:11:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:13850 run:0 clone:38518 gen:127 core:0xa7 unit:0x00000095287234c95e78894f6fb219bb
04:01:11:WU02:FS00:Uploading 2.49MiB to 40.114.52.201
04:01:11:WU02:FS00:Connecting to 40.114.52.201:8080
04:01:11:WU00:FS00:Starting
04:01:11:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 00 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -np 15
04:01:11:WU00:FS00:Started FahCore on PID 15663
04:01:11:WU00:FS00:Core PID:15667
04:01:11:WU00:FS00:FahCore 0xa7 started
04:01:11:WU00:FS00:0xa7:*********************** Log Started 2020-05-22T04:01:11Z ***********************
04:01:11:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
04:01:11:WU00:FS00:0xa7:       Type: 0xa7
04:01:11:WU00:FS00:0xa7:       Core: Gromacs
04:01:11:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 705 -lifeline 15663 -checkpoint 15 -np
04:01:11:WU00:FS00:0xa7:             15
04:01:11:WU00:FS00:0xa7:************************************ CBang *************************************
04:01:11:WU00:FS00:0xa7:       Date: Nov 5 2019
04:01:11:WU00:FS00:0xa7:       Time: 06:06:57
04:01:11:WU00:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
04:01:11:WU00:FS00:0xa7:     Branch: master
04:01:11:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
04:01:11:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
04:01:11:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
04:01:11:WU00:FS00:0xa7:       Bits: 64
04:01:11:WU00:FS00:0xa7:       Mode: Release
04:01:11:WU00:FS00:0xa7:************************************ System ************************************
04:01:11:WU00:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
04:01:11:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
04:01:11:WU00:FS00:0xa7:       CPUs: 16
04:01:11:WU00:FS00:0xa7:     Memory: 125.81GiB
04:01:11:WU00:FS00:0xa7:Free Memory: 124.51GiB
04:01:11:WU00:FS00:0xa7:    Threads: POSIX_THREADS
04:01:11:WU00:FS00:0xa7: OS Version: 4.19
04:01:11:WU00:FS00:0xa7:Has Battery: false
04:01:11:WU00:FS00:0xa7: On Battery: false
04:01:11:WU00:FS00:0xa7: UTC Offset: -4
04:01:11:WU00:FS00:0xa7:        PID: 15667
04:01:11:WU00:FS00:0xa7:        CWD: /var/lib/fahclient/work
04:01:11:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
04:01:11:WU00:FS00:0xa7:    Version: 0.0.18
04:01:11:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
04:01:11:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
04:01:11:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
04:01:11:WU00:FS00:0xa7:       Date: Nov 5 2019
04:01:11:WU00:FS00:0xa7:       Time: 06:13:26
04:01:11:WU00:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
04:01:11:WU00:FS00:0xa7:     Branch: master
04:01:11:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
04:01:11:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
04:01:11:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
04:01:11:WU00:FS00:0xa7:       Bits: 64
04:01:11:WU00:FS00:0xa7:       Mode: Release
04:01:11:WU00:FS00:0xa7:************************************ Build *************************************
04:01:11:WU00:FS00:0xa7:       SIMD: avx_256
04:01:11:WU00:FS00:0xa7:********************************************************************************
04:01:11:WU00:FS00:0xa7:Project: 16415 (Run 906, Clone 0, Gen 36)
04:01:11:WU00:FS00:0xa7:Unit: 0x0000002980fccb015ea74f945a761064
04:01:11:WU00:FS00:0xa7:Reading tar file core.xml
04:01:11:WU00:FS00:0xa7:Reading tar file frame36.tpr
04:01:11:WU00:FS00:0xa7:Digital signatures verified
04:01:11:WU00:FS00:0xa7:Calling: mdrun -s frame36.tpr -o frame36.trr -x frame36.xtc -cpt 15 -nt 15
04:01:11:WU00:FS00:0xa7:Steps: first=9000000 total=250000
04:01:12:WU02:FS00:Upload complete
04:01:12:WU02:FS00:Server responded WORK_ACK (400)
04:01:12:WU02:FS00:Final credit estimate, 7243.00 points
04:01:12:WU02:FS00:Cleaning up
04:01:15:WU00:FS00:0xa7:Completed 1 out of 250000 steps (0%)
04:02:18:WU00:FS00:0xa7:Completed 2500 out of 250000 steps (1%)
04:03:21:WU00:FS00:0xa7:Completed 5000 out of 250000 steps (2%)
04:04:25:WU00:FS00:0xa7:Completed 7500 out of 250000 steps (3%)
04:05:29:WU00:FS00:0xa7:Completed 10000 out of 250000 steps (4%)
04:06:32:WU00:FS00:0xa7:Completed 12500 out of 250000 steps (5%)
04:07:36:WU00:FS00:0xa7:Completed 15000 out of 250000 steps (6%)
04:08:39:WU00:FS00:0xa7:Completed 17500 out of 250000 steps (7%)
04:09:42:WU00:FS00:0xa7:Completed 20000 out of 250000 steps (8%)
04:10:44:WU00:FS00:0xa7:Completed 22500 out of 250000 steps (9%)
******************************* Date: 2020-05-22 *******************************
05:45:44:WU02:FS00:Connecting to 65.254.110.245:8080
05:45:44:WU02:FS00:Assigned to work server 128.252.203.1
05:45:44:WU02:FS00:Requesting new work unit for slot 00: RUNNING cpu:15 from 128.252.203.1
05:45:44:WU02:FS00:Connecting to 128.252.203.1:8080
05:45:45:WU02:FS00:Downloading 5.44MiB
05:45:45:WU02:FS00:Download complete
05:45:45:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:16440 run:0 clone:1303 gen:53 core:0xa7 unit:0x0000003780fccb015ea8eba73f51aa4b
05:46:48:WU00:FS00:0xa7:Completed 250000 out of 250000 steps (100%)
05:46:51:WU00:FS00:0xa7:Saving result file ../logfile_01.txt
05:46:51:WU00:FS00:0xa7:Saving result file frame36.trr
05:46:51:WU00:FS00:0xa7:Saving result file frame36.xtc
05:46:51:WU00:FS00:0xa7:Saving result file md.log
05:46:51:WU00:FS00:0xa7:Saving result file science.log
05:46:51:WU00:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
05:46:51:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
05:46:52:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16415 run:906 clone:0 gen:36 core:0xa7 unit:0x0000002980fccb015ea74f945a761064
05:46:52:WU00:FS00:Uploading 2.41MiB to 128.252.203.1
05:46:52:WU00:FS00:Connecting to 128.252.203.1:8080
05:46:52:WU02:FS00:Starting
05:46:52:WU02:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 02 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -np 15
05:46:52:WU02:FS00:Started FahCore on PID 15797
05:46:52:WU02:FS00:Core PID:15801
05:46:52:WU02:FS00:FahCore 0xa7 started
05:46:52:WU02:FS00:0xa7:*********************** Log Started 2020-05-22T05:46:52Z ***********************
05:46:52:WU02:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
05:46:52:WU02:FS00:0xa7:       Type: 0xa7
05:46:52:WU02:FS00:0xa7:       Core: Gromacs
05:46:52:WU02:FS00:0xa7:       Args: -dir 02 -suffix 01 -version 705 -lifeline 15797 -checkpoint 15 -np
05:46:52:WU02:FS00:0xa7:             15
05:46:52:WU02:FS00:0xa7:************************************ CBang *************************************
05:46:52:WU02:FS00:0xa7:       Date: Nov 5 2019
05:46:52:WU02:FS00:0xa7:       Time: 06:06:57
05:46:52:WU02:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
05:46:52:WU02:FS00:0xa7:     Branch: master
05:46:52:WU02:FS00:0xa7:   Compiler: GNU 8.3.0
05:46:52:WU02:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
05:46:52:WU02:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
05:46:52:WU02:FS00:0xa7:       Bits: 64
05:46:52:WU02:FS00:0xa7:       Mode: Release
05:46:52:WU02:FS00:0xa7:************************************ System ************************************
05:46:52:WU02:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
05:46:52:WU02:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
05:46:52:WU02:FS00:0xa7:       CPUs: 16
05:46:52:WU02:FS00:0xa7:     Memory: 125.81GiB
05:46:52:WU02:FS00:0xa7:Free Memory: 124.49GiB
05:46:52:WU02:FS00:0xa7:    Threads: POSIX_THREADS
05:46:52:WU02:FS00:0xa7: OS Version: 4.19
05:46:52:WU02:FS00:0xa7:Has Battery: false
05:46:52:WU02:FS00:0xa7: On Battery: false
05:46:52:WU02:FS00:0xa7: UTC Offset: -4
05:46:52:WU02:FS00:0xa7:        PID: 15801
05:46:52:WU02:FS00:0xa7:        CWD: /var/lib/fahclient/work
05:46:52:WU02:FS00:0xa7:******************************** Build - libFAH ********************************
05:46:52:WU02:FS00:0xa7:    Version: 0.0.18
05:46:52:WU02:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
05:46:52:WU02:FS00:0xa7:  Copyright: 2019 foldingathome.org
05:46:52:WU02:FS00:0xa7:   Homepage: https://foldingathome.org/
05:46:52:WU02:FS00:0xa7:       Date: Nov 5 2019
05:46:52:WU02:FS00:0xa7:       Time: 06:13:26
05:46:52:WU02:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
05:46:52:WU02:FS00:0xa7:     Branch: master
05:46:52:WU02:FS00:0xa7:   Compiler: GNU 8.3.0
05:46:52:WU02:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
05:46:52:WU02:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
05:46:52:WU02:FS00:0xa7:       Bits: 64
05:46:52:WU02:FS00:0xa7:       Mode: Release
05:46:52:WU02:FS00:0xa7:************************************ Build *************************************
05:46:52:WU02:FS00:0xa7:       SIMD: avx_256
05:46:52:WU02:FS00:0xa7:********************************************************************************
05:46:52:WU02:FS00:0xa7:Project: 16440 (Run 0, Clone 1303, Gen 53)
05:46:52:WU02:FS00:0xa7:Unit: 0x0000003780fccb015ea8eba73f51aa4b
05:46:52:WU02:FS00:0xa7:Reading tar file core.xml
05:46:52:WU02:FS00:0xa7:Reading tar file frame53.tpr
05:46:52:WU02:FS00:0xa7:Digital signatures verified
05:46:52:WU02:FS00:0xa7:Calling: mdrun -s frame53.tpr -o frame53.trr -x frame53.xtc -cpt 15 -nt 15
05:46:52:WU02:FS00:0xa7:Steps: first=13250000 total=250000
05:46:54:WU00:FS00:Upload complete
05:46:54:WU00:FS00:Server responded WORK_ACK (400)
05:46:54:WU00:FS00:Final credit estimate, 5190.00 points
05:46:54:WU00:FS00:Cleaning up
05:46:54:WU02:FS00:0xa7:Completed 1 out of 250000 steps (0%)
05:48:35:WU02:FS00:0xa7:Completed 2500 out of 250000 steps (1%)
05:50:16:WU02:FS00:0xa7:Completed 5000 out of 250000 steps (2%)
05:51:58:WU02:FS00:0xa7:Completed 7500 out of 250000 steps (3%)
05:53:39:WU02:FS00:0xa7:Completed 10000 out of 250000 steps (4%)
05:55:20:WU02:FS00:0xa7:Completed 12500 out of 250000 steps (5%)
05:57:00:WU02:FS00:0xa7:Completed 15000 out of 250000 steps (6%)
05:58:41:WU02:FS00:0xa7:Completed 17500 out of 250000 steps (7%)
06:00:22:WU02:FS00:0xa7:Completed 20000 out of 250000 steps (8%)
06:02:04:WU02:FS00:0xa7:Completed 22500 out of 250000 steps (9%)
08:33:05:WU00:FS00:Connecting to 65.254.110.245:8080
08:33:05:WU00:FS00:Assigned to work server 40.121.152.108
08:33:05:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:15 from 40.121.152.108
08:33:05:WU00:FS00:Connecting to 40.121.152.108:8080
08:33:05:WU00:FS00:Downloading 2.83MiB
08:33:05:WU00:FS00:Download complete
08:33:05:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14717 run:1002 clone:1 gen:13 core:0xa7 unit:0x000000112879986c5ea75687cac613ed
08:34:44:WU02:FS00:0xa7:Completed 250000 out of 250000 steps (100%)
08:34:49:WU02:FS00:0xa7:Saving result file ../logfile_01.txt
08:34:49:WU02:FS00:0xa7:Saving result file frame53.trr
08:34:49:WU02:FS00:0xa7:Saving result file frame53.xtc
08:34:49:WU02:FS00:0xa7:Saving result file md.log
08:34:49:WU02:FS00:0xa7:Saving result file science.log
08:34:49:WU02:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
08:34:50:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
08:34:50:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:16440 run:0 clone:1303 gen:53 core:0xa7 unit:0x0000003780fccb015ea8eba73f51aa4b
08:34:50:WU02:FS00:Uploading 4.53MiB to 128.252.203.1
08:34:50:WU02:FS00:Connecting to 128.252.203.1:8080
08:34:50:WU00:FS00:Starting
08:34:50:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 00 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -np 15
08:34:50:WU00:FS00:Started FahCore on PID 15981
08:34:50:WU00:FS00:Core PID:15985
08:34:50:WU00:FS00:FahCore 0xa7 started
08:34:50:WU00:FS00:0xa7:*********************** Log Started 2020-05-22T08:34:50Z ***********************
08:34:50:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
08:34:50:WU00:FS00:0xa7:       Type: 0xa7
08:34:50:WU00:FS00:0xa7:       Core: Gromacs
08:34:50:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 705 -lifeline 15981 -checkpoint 15 -np
08:34:50:WU00:FS00:0xa7:             15
08:34:50:WU00:FS00:0xa7:************************************ CBang *************************************
08:34:50:WU00:FS00:0xa7:       Date: Nov 5 2019
08:34:50:WU00:FS00:0xa7:       Time: 06:06:57
08:34:50:WU00:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
08:34:50:WU00:FS00:0xa7:     Branch: master
08:34:50:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
08:34:50:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
08:34:50:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
08:34:50:WU00:FS00:0xa7:       Bits: 64
08:34:50:WU00:FS00:0xa7:       Mode: Release
08:34:50:WU00:FS00:0xa7:************************************ System ************************************
08:34:50:WU00:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
08:34:50:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
08:34:50:WU00:FS00:0xa7:       CPUs: 16
08:34:50:WU00:FS00:0xa7:     Memory: 125.81GiB
08:34:50:WU00:FS00:0xa7:Free Memory: 124.42GiB
08:34:50:WU00:FS00:0xa7:    Threads: POSIX_THREADS
08:34:50:WU00:FS00:0xa7: OS Version: 4.19
08:34:50:WU00:FS00:0xa7:Has Battery: false
08:34:50:WU00:FS00:0xa7: On Battery: false
08:34:50:WU00:FS00:0xa7: UTC Offset: -4
08:34:50:WU00:FS00:0xa7:        PID: 15985
08:34:50:WU00:FS00:0xa7:        CWD: /var/lib/fahclient/work
08:34:50:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
08:34:50:WU00:FS00:0xa7:    Version: 0.0.18
08:34:50:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
08:34:50:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
08:34:50:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
08:34:50:WU00:FS00:0xa7:       Date: Nov 5 2019
08:34:50:WU00:FS00:0xa7:       Time: 06:13:26
08:34:50:WU00:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
08:34:50:WU00:FS00:0xa7:     Branch: master
08:34:50:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
08:34:50:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
08:34:50:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
08:34:50:WU00:FS00:0xa7:       Bits: 64
08:34:50:WU00:FS00:0xa7:       Mode: Release
08:34:50:WU00:FS00:0xa7:************************************ Build *************************************
08:34:50:WU00:FS00:0xa7:       SIMD: avx_256
08:34:50:WU00:FS00:0xa7:********************************************************************************
08:34:50:WU00:FS00:0xa7:Project: 14717 (Run 1002, Clone 1, Gen 13)
08:34:50:WU00:FS00:0xa7:Unit: 0x000000112879986c5ea75687cac613ed
08:34:50:WU00:FS00:0xa7:Reading tar file core.xml
08:34:50:WU00:FS00:0xa7:Reading tar file frame13.tpr
08:34:50:WU00:FS00:0xa7:Digital signatures verified
08:34:50:WU00:FS00:0xa7:Calling: mdrun -s frame13.tpr -o frame13.trr -cpt 15 -nt 15
08:34:50:WU00:FS00:0xa7:Steps: first=0 total=250000
08:34:53:WU00:FS00:0xa7:Completed 1 out of 250000 steps (0%)
08:34:53:WU02:FS00:Upload complete
08:34:53:WU02:FS00:Server responded WORK_ACK (400)
08:34:53:WU02:FS00:Final credit estimate, 8189.00 points
08:34:53:WU02:FS00:Cleaning up
08:35:58:WU00:FS00:0xa7:Completed 2500 out of 250000 steps (1%)
08:37:02:WU00:FS00:0xa7:Completed 5000 out of 250000 steps (2%)
08:38:05:WU00:FS00:0xa7:Completed 7500 out of 250000 steps (3%)
08:39:10:WU00:FS00:0xa7:Completed 10000 out of 250000 steps (4%)
08:40:16:WU00:FS00:0xa7:Completed 12500 out of 250000 steps (5%)
08:41:20:WU00:FS00:0xa7:Completed 15000 out of 250000 steps (6%)
08:42:25:WU00:FS00:0xa7:Completed 17500 out of 250000 steps (7%)
08:43:30:WU00:FS00:0xa7:Completed 20000 out of 250000 steps (8%)
08:44:35:WU00:FS00:0xa7:Completed 22500 out of 250000 steps (9%)
09:18:23:WU02:FS01:Connecting to 65.254.110.245:8080
09:18:23:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
09:18:23:WU02:FS01:Connecting to 18.218.241.186:80
09:18:23:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
09:18:23:ERROR:WU02:FS01:Exception: Could not get an assignment
09:18:23:WU02:FS01:Connecting to 65.254.110.245:8080
09:18:23:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
09:18:23:WU02:FS01:Connecting to 18.218.241.186:80
09:18:24:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
09:18:24:ERROR:WU02:FS01:Exception: Could not get an assignment
09:19:23:WU02:FS01:Connecting to 65.254.110.245:8080
09:19:23:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
09:19:23:WU02:FS01:Connecting to 18.218.241.186:80
09:19:24:WU02:FS01:Assigned to work server 128.252.203.2
09:19:24:WU02:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GM206GL [Quadro M2000] from 128.252.203.2
09:19:24:WU02:FS01:Connecting to 128.252.203.2:8080
09:19:34:ERROR:WU02:FS01:Exception: Server did not assign work unit
09:21:01:WU02:FS01:Connecting to 65.254.110.245:8080
09:21:01:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
09:21:01:WU02:FS01:Connecting to 18.218.241.186:80
09:21:01:WU02:FS01:Assigned to work server 128.252.203.2
09:21:01:WU02:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GM206GL [Quadro M2000] from 128.252.203.2
09:21:01:WU02:FS01:Connecting to 128.252.203.2:8080
09:21:39:WU02:FS01:Downloading 28.40MiB
09:21:41:WU02:FS01:Download complete
09:21:41:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:14562 run:0 clone:762 gen:49 core:0x22 unit:0x0000004d80fccb025e9615bd7b434dc9
09:28:37:WU01:FS01:0x22:Completed 5000000 out of 5000000 steps (100%)
09:28:41:WU01:FS01:0x22:Saving result file ../logfile_01.txt
09:28:41:WU01:FS01:0x22:Saving result file checkpointState.xml
09:28:43:WU01:FS01:0x22:Saving result file checkpt.crc
09:28:43:WU01:FS01:0x22:Saving result file positions.xtc
09:28:48:WU01:FS01:0x22:Saving result file science.log
09:28:48:WU01:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
09:28:48:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
09:28:48:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:16444 run:0 clone:1788 gen:18 core:0x22 unit:0x0000002580fccb015eaa0014ab9ca778
09:28:48:WU01:FS01:Uploading 63.41MiB to 128.252.203.1
09:28:48:WU01:FS01:Connecting to 128.252.203.1:8080
09:28:49:WU02:FS01:Starting
09:28:49:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 02 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
09:28:49:WU02:FS01:Started FahCore on PID 16056
09:28:49:WU02:FS01:Core PID:16060
09:28:49:WU02:FS01:FahCore 0x22 started
09:28:49:WU02:FS01:0x22:*********************** Log Started 2020-05-22T09:28:49Z ***********************
09:28:49:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
09:28:49:WU02:FS01:0x22:       Type: 0x22
09:28:49:WU02:FS01:0x22:       Core: Core22
09:28:49:WU02:FS01:0x22:    Website: https://foldingathome.org/
09:28:49:WU02:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
09:28:49:WU02:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
09:28:49:WU02:FS01:0x22:             <rafal.wiewiora@choderalab.org>
09:28:49:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 705 -lifeline 16056 -checkpoint 15
09:28:49:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
09:28:49:WU02:FS01:0x22:             0 -gpu 0
09:28:49:WU02:FS01:0x22:     Config: <none>
09:28:49:WU02:FS01:0x22:************************************ Build *************************************
09:28:49:WU02:FS01:0x22:    Version: 0.0.5
09:28:49:WU02:FS01:0x22:       Date: Apr 22 2020
09:28:49:WU02:FS01:0x22:       Time: 03:57:11
09:28:49:WU02:FS01:0x22: Repository: Git
09:28:49:WU02:FS01:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
09:28:49:WU02:FS01:0x22:     Branch: HEAD
09:28:49:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
09:28:49:WU02:FS01:0x22:    Options: -std=c++11 -O3 -funroll-loops
09:28:49:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
09:28:49:WU02:FS01:0x22:       Bits: 64
09:28:49:WU02:FS01:0x22:       Mode: Release
09:28:49:WU02:FS01:0x22:************************************ System ************************************
09:28:49:WU02:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
09:28:49:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
09:28:49:WU02:FS01:0x22:       CPUs: 16
09:28:49:WU02:FS01:0x22:     Memory: 125.81GiB
09:28:49:WU02:FS01:0x22:Free Memory: 124.71GiB
09:28:49:WU02:FS01:0x22:    Threads: POSIX_THREADS
09:28:49:WU02:FS01:0x22: OS Version: 4.19
09:28:49:WU02:FS01:0x22:Has Battery: false
09:28:49:WU02:FS01:0x22: On Battery: false
09:28:49:WU02:FS01:0x22: UTC Offset: -4
09:28:49:WU02:FS01:0x22:        PID: 16060
09:28:49:WU02:FS01:0x22:        CWD: /var/lib/fahclient/work
09:28:49:WU02:FS01:0x22:         OS: Linux 4.19.0-8-amd64 x86_64
09:28:49:WU02:FS01:0x22:    OS Arch: AMD64
09:28:49:WU02:FS01:0x22:********************************************************************************
09:28:49:WU02:FS01:0x22:Project: 14562 (Run 0, Clone 762, Gen 49)
09:28:49:WU02:FS01:0x22:Unit: 0x0000004d80fccb025e9615bd7b434dc9
09:28:49:WU02:FS01:0x22:Reading tar file core.xml
09:28:49:WU02:FS01:0x22:Reading tar file integrator.xml
09:28:49:WU02:FS01:0x22:Reading tar file state.xml
09:28:54:WU01:FS01:Upload 11.83%
09:28:57:WU02:FS01:0x22:Reading tar file system.xml
09:29:00:WU01:FS01:Upload 24.74%
09:29:02:WU02:FS01:0x22:Digital signatures verified
09:29:02:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
09:29:02:WU02:FS01:0x22:Version 0.0.5
09:29:06:WU01:FS01:Upload 36.77%
09:29:12:WU01:FS01:Upload 50.27%
09:29:18:WU01:FS01:Upload 63.18%
09:29:24:WU01:FS01:Upload 76.19%
09:29:30:WU01:FS01:Upload 89.70%
09:29:35:WU01:FS01:Upload complete
09:29:35:WU01:FS01:Server responded WORK_ACK (400)
09:29:35:WU01:FS01:Final credit estimate, 82057.00 points
09:29:35:WU01:FS01:Cleaning up
09:30:07:WU02:FS01:0x22:Completed 0 out of 500000 steps (0%)
09:30:07:WU02:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
09:37:58:WU02:FS01:0x22:Completed 5000 out of 500000 steps (1%)
09:45:46:WU02:FS01:0x22:Completed 10000 out of 500000 steps (2%)
09:53:33:WU02:FS01:0x22:Completed 15000 out of 500000 steps (3%)
10:01:21:WU02:FS01:0x22:Completed 20000 out of 500000 steps (4%)
10:09:31:WU02:FS01:0x22:Completed 25000 out of 500000 steps (5%)
10:17:19:WU02:FS01:0x22:Completed 30000 out of 500000 steps (6%)
******************************* Date: 2020-05-22 *******************************
10:22:15:WU01:FS00:Connecting to 65.254.110.245:8080
10:22:15:WU01:FS00:Assigned to work server 40.114.52.201
10:22:15:WU01:FS00:Requesting new work unit for slot 00: RUNNING cpu:15 from 40.114.52.201
10:22:15:WU01:FS00:Connecting to 40.114.52.201:8080
10:22:15:WU01:FS00:Downloading 2.14MiB
10:22:15:WU01:FS00:Download complete

[RunningTurtle]

An update. As of this morning, this FAH client is still getting newly assigned WorkUnits from Project 16438, which will not start on a 15 core machine:

Code: Select all

16:33:32:WU01:FS00:0xa7:*********************** Log Started 2020-06-09T16:33:32Z ***********************
16:33:32:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
16:33:32:WU01:FS00:0xa7:       Type: 0xa7
16:33:32:WU01:FS00:0xa7:       Core: Gromacs
16:33:32:WU01:FS00:0xa7:       Args: -dir 01 -suffix 01 -version 705 -lifeline 1901 -checkpoint 15 -np
16:33:32:WU01:FS00:0xa7:             15
16:33:32:WU01:FS00:0xa7:************************************ CBang *************************************
16:33:32:WU01:FS00:0xa7:       Date: Nov 5 2019
16:33:32:WU01:FS00:0xa7:       Time: 06:06:57
16:33:32:WU01:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
16:33:32:WU01:FS00:0xa7:     Branch: master
16:33:32:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
16:33:32:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
16:33:32:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
16:33:32:WU01:FS00:0xa7:       Bits: 64
16:33:32:WU01:FS00:0xa7:       Mode: Release
16:33:32:WU01:FS00:0xa7:************************************ System ************************************
16:33:32:WU01:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
16:33:32:WU01:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
16:33:32:WU01:FS00:0xa7:       CPUs: 16
16:33:32:WU01:FS00:0xa7:     Memory: 125.81GiB
16:33:32:WU01:FS00:0xa7:Free Memory: 122.14GiB
16:33:32:WU01:FS00:0xa7:    Threads: POSIX_THREADS
16:33:32:WU01:FS00:0xa7: OS Version: 4.19
16:33:32:WU01:FS00:0xa7:Has Battery: false
16:33:32:WU01:FS00:0xa7: On Battery: false
16:33:32:WU01:FS00:0xa7: UTC Offset: -4
16:33:32:WU01:FS00:0xa7:        PID: 1905
16:33:32:WU01:FS00:0xa7:        CWD: /var/lib/fahclient/work
16:33:32:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
16:33:32:WU01:FS00:0xa7:    Version: 0.0.18
16:33:32:WU01:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:33:32:WU01:FS00:0xa7:  Copyright: 2019 foldingathome.org
16:33:32:WU01:FS00:0xa7:   Homepage: https://foldingathome.org/
16:33:32:WU01:FS00:0xa7:       Date: Nov 5 2019
16:33:32:WU01:FS00:0xa7:       Time: 06:13:26
16:33:32:WU01:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
16:33:32:WU01:FS00:0xa7:     Branch: master
16:33:32:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
16:33:32:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
16:33:32:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
16:33:32:WU01:FS00:0xa7:       Bits: 64
16:33:32:WU01:FS00:0xa7:       Mode: Release
16:33:32:WU01:FS00:0xa7:************************************ Build *************************************
16:33:32:WU01:FS00:0xa7:       SIMD: avx_256
16:33:32:WU01:FS00:0xa7:********************************************************************************
16:33:32:WU01:FS00:0xa7:Project: 16438 (Run 0, Clone 1034, Gen 125)
16:33:32:WU01:FS00:0xa7:Unit: 0x0000009780fccb015ea74d8038f9ec86
16:33:32:WU01:FS00:0xa7:Reading tar file core.xml
16:33:32:WU01:FS00:0xa7:Reading tar file frame125.tpr
16:33:32:WU01:FS00:0xa7:Digital signatures verified
16:33:32:WU01:FS00:0xa7:Calling: mdrun -s frame125.tpr -o frame125.trr -x frame125.xtc -cpt 15 -nt 15
16:33:32:WU01:FS00:0xa7:Steps: first=62500000 total=500000
16:33:32:WU01:FS00:0xa7:ERROR:
16:33:32:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
16:33:32:WU01:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20191026-456f0d636-unknown
16:33:32:WU01:FS00:0xa7:ERROR:Source code file: /host/debian-stable-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/mdlib/domdec.c, line: 6902
16:33:32:WU01:FS00:0xa7:ERROR:
16:33:32:WU01:FS00:0xa7:ERROR:Fatal error:
16:33:32:WU01:FS00:0xa7:ERROR:There is no domain decomposition for 15 ranks that is compatible with the given box and a minimum cell size of 1.39789 nm
16:33:32:WU01:FS00:0xa7:ERROR:Change the number of ranks or mdrun option -rcon or -dds or your LINCS settings
16:33:32:WU01:FS00:0xa7:ERROR:Look in the log file for details on the domain decomposition
16:33:32:WU01:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
16:33:32:WU01:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
16:33:32:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
16:33:37:WU01:FS00:0xa7:WARNING:Unexpected exit() call
16:33:37:WU01:FS00:0xa7:WARNING:Unexpected exit from science code
16:33:37:WU01:FS00:0xa7:Saving result file ../logfile_01.txt
16:33:37:WU01:FS00:0xa7:Saving result file md.log
16:33:37:WU01:FS00:0xa7:Saving result file science.log
16:33:37:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)

[Joe_H]

Can you post the section of the log before what you have posted that shows the download request, and then getting this Project 16438 WU? The project has already been reported for this issue and that information is needed.

[RunningTurtle]

Here is the log from this morning, including the download of Project: 16438 (Run 0, Clone 1034, Gen 125), along with the first five failed attempts at running the work unit.

Code: Select all

07:34:06:WU01:FS00:Connecting to 65.254.110.245:8080
07:34:06:WU01:FS00:Assigned to work server 128.252.203.1
07:34:06:WU01:FS00:Requesting new work unit for slot 00: RUNNING cpu:15 from 128.252.203.1
07:34:06:WU01:FS00:Connecting to 128.252.203.1:8080
07:34:06:WU01:FS00:Downloading 1.33MiB
07:34:06:WU01:FS00:Download complete
07:34:06:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:16438 run:0 clone:1034 gen:125 core:0xa7 unit:0x0000009780fccb015ea74d8038f9ec86
07:35:11:WU01:FS00:Starting
07:35:11:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -np 15
07:35:11:WU01:FS00:Started FahCore on PID 21732
07:35:11:WU01:FS00:Core PID:21736
07:35:11:WU01:FS00:FahCore 0xa7 started
07:35:11:WU01:FS00:0xa7:*********************** Log Started 2020-06-09T07:35:11Z ***********************
07:35:11:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
07:35:11:WU01:FS00:0xa7:       Type: 0xa7
07:35:11:WU01:FS00:0xa7:       Core: Gromacs
07:35:11:WU01:FS00:0xa7:       Args: -dir 01 -suffix 01 -version 705 -lifeline 21732 -checkpoint 15 -np
07:35:11:WU01:FS00:0xa7:             15
07:35:11:WU01:FS00:0xa7:************************************ CBang *************************************
07:35:11:WU01:FS00:0xa7:       Date: Nov 5 2019
07:35:11:WU01:FS00:0xa7:       Time: 06:06:57
07:35:11:WU01:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
07:35:11:WU01:FS00:0xa7:     Branch: master
07:35:11:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
07:35:11:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
07:35:11:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
07:35:11:WU01:FS00:0xa7:       Bits: 64
07:35:11:WU01:FS00:0xa7:       Mode: Release
07:35:11:WU01:FS00:0xa7:************************************ System ************************************
07:35:11:WU01:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
07:35:11:WU01:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
07:35:11:WU01:FS00:0xa7:       CPUs: 16
07:35:11:WU01:FS00:0xa7:     Memory: 125.81GiB
07:35:11:WU01:FS00:0xa7:Free Memory: 122.29GiB
07:35:11:WU01:FS00:0xa7:    Threads: POSIX_THREADS
07:35:11:WU01:FS00:0xa7: OS Version: 4.19
07:35:11:WU01:FS00:0xa7:Has Battery: false
07:35:11:WU01:FS00:0xa7: On Battery: false
07:35:11:WU01:FS00:0xa7: UTC Offset: -4
07:35:11:WU01:FS00:0xa7:        PID: 21736
07:35:11:WU01:FS00:0xa7:        CWD: /var/lib/fahclient/work
07:35:11:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
07:35:11:WU01:FS00:0xa7:    Version: 0.0.18
07:35:11:WU01:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:35:11:WU01:FS00:0xa7:  Copyright: 2019 foldingathome.org
07:35:11:WU01:FS00:0xa7:   Homepage: https://foldingathome.org/
07:35:11:WU01:FS00:0xa7:       Date: Nov 5 2019
07:35:11:WU01:FS00:0xa7:       Time: 06:13:26
07:35:11:WU01:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
07:35:11:WU01:FS00:0xa7:     Branch: master
07:35:11:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
07:35:11:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
07:35:11:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
07:35:11:WU01:FS00:0xa7:       Bits: 64
07:35:11:WU01:FS00:0xa7:       Mode: Release
07:35:11:WU01:FS00:0xa7:************************************ Build *************************************
07:35:11:WU01:FS00:0xa7:       SIMD: avx_256
07:35:11:WU01:FS00:0xa7:********************************************************************************
07:35:11:WU01:FS00:0xa7:Project: 16438 (Run 0, Clone 1034, Gen 125)
07:35:11:WU01:FS00:0xa7:Unit: 0x0000009780fccb015ea74d8038f9ec86
07:35:11:WU01:FS00:0xa7:Reading tar file core.xml
07:35:11:WU01:FS00:0xa7:Reading tar file frame125.tpr
07:35:11:WU01:FS00:0xa7:Digital signatures verified
07:35:11:WU01:FS00:0xa7:Calling: mdrun -s frame125.tpr -o frame125.trr -x frame125.xtc -cpt 15 -nt 15
07:35:11:WU01:FS00:0xa7:Steps: first=62500000 total=500000
07:35:11:WU01:FS00:0xa7:ERROR:
07:35:11:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
07:35:11:WU01:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20191026-456f0d636-unknown
07:35:11:WU01:FS00:0xa7:ERROR:Source code file: /host/debian-stable-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/mdlib/domdec.c, line: 6902
07:35:11:WU01:FS00:0xa7:ERROR:
07:35:11:WU01:FS00:0xa7:ERROR:Fatal error:
07:35:11:WU01:FS00:0xa7:ERROR:There is no domain decomposition for 15 ranks that is compatible with the given box and a minimum cell size of 1.39789 nm
07:35:11:WU01:FS00:0xa7:ERROR:Change the number of ranks or mdrun option -rcon or -dds or your LINCS settings
07:35:11:WU01:FS00:0xa7:ERROR:Look in the log file for details on the domain decomposition
07:35:11:WU01:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
07:35:11:WU01:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
07:35:11:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
07:35:16:WU01:FS00:0xa7:WARNING:Unexpected exit() call
07:35:16:WU01:FS00:0xa7:WARNING:Unexpected exit from science code
07:35:16:WU01:FS00:0xa7:Saving result file ../logfile_01.txt
07:35:16:WU01:FS00:0xa7:Saving result file md.log
07:35:16:WU01:FS00:0xa7:Saving result file science.log
07:35:16:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
07:35:16:WU01:FS00:Starting
07:35:16:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -np 15
07:35:17:WU01:FS00:Started FahCore on PID 21754
07:35:17:WU01:FS00:Core PID:21758
07:35:17:WU01:FS00:FahCore 0xa7 started
07:35:17:WU01:FS00:0xa7:*********************** Log Started 2020-06-09T07:35:17Z ***********************
07:35:17:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
07:35:17:WU01:FS00:0xa7:       Type: 0xa7
07:35:17:WU01:FS00:0xa7:       Core: Gromacs
07:35:17:WU01:FS00:0xa7:       Args: -dir 01 -suffix 01 -version 705 -lifeline 21754 -checkpoint 15 -np
07:35:17:WU01:FS00:0xa7:             15
07:35:17:WU01:FS00:0xa7:************************************ CBang *************************************
07:35:17:WU01:FS00:0xa7:       Date: Nov 5 2019
07:35:17:WU01:FS00:0xa7:       Time: 06:06:57
07:35:17:WU01:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
07:35:17:WU01:FS00:0xa7:     Branch: master
07:35:17:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
07:35:17:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
07:35:17:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
07:35:17:WU01:FS00:0xa7:       Bits: 64
07:35:17:WU01:FS00:0xa7:       Mode: Release
07:35:17:WU01:FS00:0xa7:************************************ System ************************************
07:35:17:WU01:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
07:35:17:WU01:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
07:35:17:WU01:FS00:0xa7:       CPUs: 16
07:35:17:WU01:FS00:0xa7:     Memory: 125.81GiB
07:35:17:WU01:FS00:0xa7:Free Memory: 122.32GiB
07:35:17:WU01:FS00:0xa7:    Threads: POSIX_THREADS
07:35:17:WU01:FS00:0xa7: OS Version: 4.19
07:35:17:WU01:FS00:0xa7:Has Battery: false
07:35:17:WU01:FS00:0xa7: On Battery: false
07:35:17:WU01:FS00:0xa7: UTC Offset: -4
07:35:17:WU01:FS00:0xa7:        PID: 21758
07:35:17:WU01:FS00:0xa7:        CWD: /var/lib/fahclient/work
07:35:17:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
07:35:17:WU01:FS00:0xa7:    Version: 0.0.18
07:35:17:WU01:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:35:17:WU01:FS00:0xa7:  Copyright: 2019 foldingathome.org
07:35:17:WU01:FS00:0xa7:   Homepage: https://foldingathome.org/
07:35:17:WU01:FS00:0xa7:       Date: Nov 5 2019
07:35:17:WU01:FS00:0xa7:       Time: 06:13:26
07:35:17:WU01:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
07:35:17:WU01:FS00:0xa7:     Branch: master
07:35:17:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
07:35:17:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
07:35:17:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
07:35:17:WU01:FS00:0xa7:       Bits: 64
07:35:17:WU01:FS00:0xa7:       Mode: Release
07:35:17:WU01:FS00:0xa7:************************************ Build *************************************
07:35:17:WU01:FS00:0xa7:       SIMD: avx_256
07:35:17:WU01:FS00:0xa7:********************************************************************************
07:35:17:WU01:FS00:0xa7:Project: 16438 (Run 0, Clone 1034, Gen 125)
07:35:17:WU01:FS00:0xa7:Unit: 0x0000009780fccb015ea74d8038f9ec86
07:35:17:WU01:FS00:0xa7:Reading tar file core.xml
07:35:17:WU01:FS00:0xa7:Reading tar file frame125.tpr
07:35:17:WU01:FS00:0xa7:Digital signatures verified
07:35:17:WU01:FS00:0xa7:Calling: mdrun -s frame125.tpr -o frame125.trr -x frame125.xtc -cpt 15 -nt 15
07:35:17:WU01:FS00:0xa7:Steps: first=62500000 total=500000
07:35:17:WU01:FS00:0xa7:ERROR:
07:35:17:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
07:35:17:WU01:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20191026-456f0d636-unknown
07:35:17:WU01:FS00:0xa7:ERROR:Source code file: /host/debian-stable-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/mdlib/domdec.c, line: 6902
07:35:17:WU01:FS00:0xa7:ERROR:
07:35:17:WU01:FS00:0xa7:ERROR:Fatal error:
07:35:17:WU01:FS00:0xa7:ERROR:There is no domain decomposition for 15 ranks that is compatible with the given box and a minimum cell size of 1.39789 nm
07:35:17:WU01:FS00:0xa7:ERROR:Change the number of ranks or mdrun option -rcon or -dds or your LINCS settings
07:35:17:WU01:FS00:0xa7:ERROR:Look in the log file for details on the domain decomposition
07:35:17:WU01:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
07:35:17:WU01:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
07:35:17:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
07:35:22:WU01:FS00:0xa7:WARNING:Unexpected exit() call
07:35:22:WU01:FS00:0xa7:WARNING:Unexpected exit from science code
07:35:22:WU01:FS00:0xa7:Saving result file ../logfile_01.txt
07:35:22:WU01:FS00:0xa7:Saving result file md.log
07:35:22:WU01:FS00:0xa7:Saving result file science.log
07:35:22:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
07:36:17:WU01:FS00:Starting
07:36:17:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -np 15
07:36:17:WU01:FS00:Started FahCore on PID 21776
07:36:17:WU01:FS00:Core PID:21780
07:36:17:WU01:FS00:FahCore 0xa7 started
07:36:17:WU01:FS00:0xa7:*********************** Log Started 2020-06-09T07:36:17Z ***********************
07:36:17:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
07:36:17:WU01:FS00:0xa7:       Type: 0xa7
07:36:17:WU01:FS00:0xa7:       Core: Gromacs
07:36:17:WU01:FS00:0xa7:       Args: -dir 01 -suffix 01 -version 705 -lifeline 21776 -checkpoint 15 -np
07:36:17:WU01:FS00:0xa7:             15
07:36:17:WU01:FS00:0xa7:************************************ CBang *************************************
07:36:17:WU01:FS00:0xa7:       Date: Nov 5 2019
07:36:17:WU01:FS00:0xa7:       Time: 06:06:57
07:36:17:WU01:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
07:36:17:WU01:FS00:0xa7:     Branch: master
07:36:17:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
07:36:17:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
07:36:17:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
07:36:17:WU01:FS00:0xa7:       Bits: 64
07:36:17:WU01:FS00:0xa7:       Mode: Release
07:36:17:WU01:FS00:0xa7:************************************ System ************************************
07:36:17:WU01:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
07:36:17:WU01:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
07:36:17:WU01:FS00:0xa7:       CPUs: 16
07:36:17:WU01:FS00:0xa7:     Memory: 125.81GiB
07:36:17:WU01:FS00:0xa7:Free Memory: 122.32GiB
07:36:17:WU01:FS00:0xa7:    Threads: POSIX_THREADS
07:36:17:WU01:FS00:0xa7: OS Version: 4.19
07:36:17:WU01:FS00:0xa7:Has Battery: false
07:36:17:WU01:FS00:0xa7: On Battery: false
07:36:17:WU01:FS00:0xa7: UTC Offset: -4
07:36:17:WU01:FS00:0xa7:        PID: 21780
07:36:17:WU01:FS00:0xa7:        CWD: /var/lib/fahclient/work
07:36:17:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
07:36:17:WU01:FS00:0xa7:    Version: 0.0.18
07:36:17:WU01:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:36:17:WU01:FS00:0xa7:  Copyright: 2019 foldingathome.org
07:36:17:WU01:FS00:0xa7:   Homepage: https://foldingathome.org/
07:36:17:WU01:FS00:0xa7:       Date: Nov 5 2019
07:36:17:WU01:FS00:0xa7:       Time: 06:13:26
07:36:17:WU01:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
07:36:17:WU01:FS00:0xa7:     Branch: master
07:36:17:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
07:36:17:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
07:36:17:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
07:36:17:WU01:FS00:0xa7:       Bits: 64
07:36:17:WU01:FS00:0xa7:       Mode: Release
07:36:17:WU01:FS00:0xa7:************************************ Build *************************************
07:36:17:WU01:FS00:0xa7:       SIMD: avx_256
07:36:17:WU01:FS00:0xa7:********************************************************************************
07:36:17:WU01:FS00:0xa7:Project: 16438 (Run 0, Clone 1034, Gen 125)
07:36:17:WU01:FS00:0xa7:Unit: 0x0000009780fccb015ea74d8038f9ec86
07:36:17:WU01:FS00:0xa7:Reading tar file core.xml
07:36:17:WU01:FS00:0xa7:Reading tar file frame125.tpr
07:36:17:WU01:FS00:0xa7:Digital signatures verified
07:36:17:WU01:FS00:0xa7:Calling: mdrun -s frame125.tpr -o frame125.trr -x frame125.xtc -cpt 15 -nt 15
07:36:17:WU01:FS00:0xa7:Steps: first=62500000 total=500000
07:36:17:WU01:FS00:0xa7:ERROR:
07:36:17:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
07:36:17:WU01:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20191026-456f0d636-unknown
07:36:17:WU01:FS00:0xa7:ERROR:Source code file: /host/debian-stable-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/mdlib/domdec.c, line: 6902
07:36:17:WU01:FS00:0xa7:ERROR:
07:36:17:WU01:FS00:0xa7:ERROR:Fatal error:
07:36:17:WU01:FS00:0xa7:ERROR:There is no domain decomposition for 15 ranks that is compatible with the given box and a minimum cell size of 1.39789 nm
07:36:17:WU01:FS00:0xa7:ERROR:Change the number of ranks or mdrun option -rcon or -dds or your LINCS settings
07:36:17:WU01:FS00:0xa7:ERROR:Look in the log file for details on the domain decomposition
07:36:17:WU01:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
07:36:17:WU01:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
07:36:17:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
07:36:22:WU01:FS00:0xa7:WARNING:Unexpected exit() call
07:36:22:WU01:FS00:0xa7:WARNING:Unexpected exit from science code
07:36:22:WU01:FS00:0xa7:Saving result file ../logfile_01.txt
07:36:22:WU01:FS00:0xa7:Saving result file md.log
07:36:22:WU01:FS00:0xa7:Saving result file science.log
07:36:22:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
07:37:17:WU01:FS00:Starting
07:37:17:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -np 15
07:37:17:WU01:FS00:Started FahCore on PID 21798
07:37:17:WU01:FS00:Core PID:21802
07:37:17:WU01:FS00:FahCore 0xa7 started
07:37:17:WU01:FS00:0xa7:*********************** Log Started 2020-06-09T07:37:17Z ***********************
07:37:17:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
07:37:17:WU01:FS00:0xa7:       Type: 0xa7
07:37:17:WU01:FS00:0xa7:       Core: Gromacs
07:37:17:WU01:FS00:0xa7:       Args: -dir 01 -suffix 01 -version 705 -lifeline 21798 -checkpoint 15 -np
07:37:17:WU01:FS00:0xa7:             15
07:37:17:WU01:FS00:0xa7:************************************ CBang *************************************
07:37:17:WU01:FS00:0xa7:       Date: Nov 5 2019
07:37:17:WU01:FS00:0xa7:       Time: 06:06:57
07:37:17:WU01:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
07:37:17:WU01:FS00:0xa7:     Branch: master
07:37:17:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
07:37:17:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
07:37:17:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
07:37:17:WU01:FS00:0xa7:       Bits: 64
07:37:17:WU01:FS00:0xa7:       Mode: Release
07:37:17:WU01:FS00:0xa7:************************************ System ************************************
07:37:17:WU01:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
07:37:17:WU01:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
07:37:17:WU01:FS00:0xa7:       CPUs: 16
07:37:17:WU01:FS00:0xa7:     Memory: 125.81GiB
07:37:17:WU01:FS00:0xa7:Free Memory: 122.33GiB
07:37:17:WU01:FS00:0xa7:    Threads: POSIX_THREADS
07:37:17:WU01:FS00:0xa7: OS Version: 4.19
07:37:17:WU01:FS00:0xa7:Has Battery: false
07:37:17:WU01:FS00:0xa7: On Battery: false
07:37:17:WU01:FS00:0xa7: UTC Offset: -4
07:37:17:WU01:FS00:0xa7:        PID: 21802
07:37:17:WU01:FS00:0xa7:        CWD: /var/lib/fahclient/work
07:37:17:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
07:37:17:WU01:FS00:0xa7:    Version: 0.0.18
07:37:17:WU01:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:37:17:WU01:FS00:0xa7:  Copyright: 2019 foldingathome.org
07:37:17:WU01:FS00:0xa7:   Homepage: https://foldingathome.org/
07:37:17:WU01:FS00:0xa7:       Date: Nov 5 2019
07:37:17:WU01:FS00:0xa7:       Time: 06:13:26
07:37:17:WU01:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
07:37:17:WU01:FS00:0xa7:     Branch: master
07:37:17:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
07:37:17:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
07:37:17:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
07:37:17:WU01:FS00:0xa7:       Bits: 64
07:37:17:WU01:FS00:0xa7:       Mode: Release
07:37:17:WU01:FS00:0xa7:************************************ Build *************************************
07:37:17:WU01:FS00:0xa7:       SIMD: avx_256
07:37:17:WU01:FS00:0xa7:********************************************************************************
07:37:17:WU01:FS00:0xa7:Project: 16438 (Run 0, Clone 1034, Gen 125)
07:37:17:WU01:FS00:0xa7:Unit: 0x0000009780fccb015ea74d8038f9ec86
07:37:17:WU01:FS00:0xa7:Reading tar file core.xml
07:37:17:WU01:FS00:0xa7:Reading tar file frame125.tpr
07:37:17:WU01:FS00:0xa7:Digital signatures verified
07:37:17:WU01:FS00:0xa7:Calling: mdrun -s frame125.tpr -o frame125.trr -x frame125.xtc -cpt 15 -nt 15
07:37:17:WU01:FS00:0xa7:Steps: first=62500000 total=500000
07:37:17:WU01:FS00:0xa7:ERROR:
07:37:17:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
07:37:17:WU01:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20191026-456f0d636-unknown
07:37:17:WU01:FS00:0xa7:ERROR:Source code file: /host/debian-stable-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/mdlib/domdec.c, line: 6902
07:37:17:WU01:FS00:0xa7:ERROR:
07:37:17:WU01:FS00:0xa7:ERROR:Fatal error:
07:37:17:WU01:FS00:0xa7:ERROR:There is no domain decomposition for 15 ranks that is compatible with the given box and a minimum cell size of 1.39789 nm
07:37:17:WU01:FS00:0xa7:ERROR:Change the number of ranks or mdrun option -rcon or -dds or your LINCS settings
07:37:17:WU01:FS00:0xa7:ERROR:Look in the log file for details on the domain decomposition
07:37:17:WU01:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
07:37:17:WU01:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
07:37:17:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
07:37:22:WU01:FS00:0xa7:WARNING:Unexpected exit() call
07:37:22:WU01:FS00:0xa7:WARNING:Unexpected exit from science code
07:37:22:WU01:FS00:0xa7:Saving result file ../logfile_01.txt
07:37:22:WU01:FS00:0xa7:Saving result file md.log
07:37:22:WU01:FS00:0xa7:Saving result file science.log
07:37:22:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
07:38:17:WU01:FS00:Starting
07:38:17:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 705 -lifeline 889 -checkpoint 15 -np 15
07:38:17:WU01:FS00:Started FahCore on PID 21820
07:38:17:WU01:FS00:Core PID:21824
07:38:17:WU01:FS00:FahCore 0xa7 started
07:38:17:WU01:FS00:0xa7:*********************** Log Started 2020-06-09T07:38:17Z ***********************
07:38:17:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
07:38:17:WU01:FS00:0xa7:       Type: 0xa7
07:38:17:WU01:FS00:0xa7:       Core: Gromacs
07:38:17:WU01:FS00:0xa7:       Args: -dir 01 -suffix 01 -version 705 -lifeline 21820 -checkpoint 15 -np
07:38:17:WU01:FS00:0xa7:             15
07:38:17:WU01:FS00:0xa7:************************************ CBang *************************************
07:38:17:WU01:FS00:0xa7:       Date: Nov 5 2019
07:38:17:WU01:FS00:0xa7:       Time: 06:06:57
07:38:17:WU01:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
07:38:17:WU01:FS00:0xa7:     Branch: master
07:38:17:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
07:38:17:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
07:38:17:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
07:38:17:WU01:FS00:0xa7:       Bits: 64
07:38:17:WU01:FS00:0xa7:       Mode: Release
07:38:17:WU01:FS00:0xa7:************************************ System ************************************
07:38:17:WU01:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
07:38:17:WU01:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 86 Stepping 3
07:38:17:WU01:FS00:0xa7:       CPUs: 16
07:38:17:WU01:FS00:0xa7:     Memory: 125.81GiB
07:38:17:WU01:FS00:0xa7:Free Memory: 122.32GiB
07:38:17:WU01:FS00:0xa7:    Threads: POSIX_THREADS
07:38:17:WU01:FS00:0xa7: OS Version: 4.19
07:38:17:WU01:FS00:0xa7:Has Battery: false
07:38:17:WU01:FS00:0xa7: On Battery: false
07:38:17:WU01:FS00:0xa7: UTC Offset: -4
07:38:17:WU01:FS00:0xa7:        PID: 21824
07:38:17:WU01:FS00:0xa7:        CWD: /var/lib/fahclient/work
07:38:17:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
07:38:17:WU01:FS00:0xa7:    Version: 0.0.18
07:38:17:WU01:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:38:17:WU01:FS00:0xa7:  Copyright: 2019 foldingathome.org
07:38:17:WU01:FS00:0xa7:   Homepage: https://foldingathome.org/
07:38:17:WU01:FS00:0xa7:       Date: Nov 5 2019
07:38:17:WU01:FS00:0xa7:       Time: 06:13:26
07:38:17:WU01:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
07:38:17:WU01:FS00:0xa7:     Branch: master
07:38:17:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
07:38:17:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
07:38:17:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
07:38:17:WU01:FS00:0xa7:       Bits: 64
07:38:17:WU01:FS00:0xa7:       Mode: Release
07:38:17:WU01:FS00:0xa7:************************************ Build *************************************
07:38:17:WU01:FS00:0xa7:       SIMD: avx_256
07:38:17:WU01:FS00:0xa7:********************************************************************************
07:38:17:WU01:FS00:0xa7:Project: 16438 (Run 0, Clone 1034, Gen 125)
07:38:17:WU01:FS00:0xa7:Unit: 0x0000009780fccb015ea74d8038f9ec86
07:38:17:WU01:FS00:0xa7:Reading tar file core.xml
07:38:17:WU01:FS00:0xa7:Reading tar file frame125.tpr
07:38:17:WU01:FS00:0xa7:Digital signatures verified
07:38:17:WU01:FS00:0xa7:Calling: mdrun -s frame125.tpr -o frame125.trr -x frame125.xtc -cpt 15 -nt 15
07:38:17:WU01:FS00:0xa7:Steps: first=62500000 total=500000
07:38:17:WU01:FS00:0xa7:ERROR:
07:38:17:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
07:38:17:WU01:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20191026-456f0d636-unknown
07:38:17:WU01:FS00:0xa7:ERROR:Source code file: /host/debian-stable-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/mdlib/domdec.c, line: 6902
07:38:17:WU01:FS00:0xa7:ERROR:
07:38:17:WU01:FS00:0xa7:ERROR:Fatal error:
07:38:17:WU01:FS00:0xa7:ERROR:There is no domain decomposition for 15 ranks that is compatible with the given box and a minimum cell size of 1.39789 nm
07:38:17:WU01:FS00:0xa7:ERROR:Change the number of ranks or mdrun option -rcon or -dds or your LINCS settings
07:38:17:WU01:FS00:0xa7:ERROR:Look in the log file for details on the domain decomposition
07:38:17:WU01:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
07:38:17:WU01:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
07:38:17:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
07:38:22:WU01:FS00:0xa7:WARNING:Unexpected exit() call
07:38:22:WU01:FS00:0xa7:WARNING:Unexpected exit from science code
07:38:22:WU01:FS00:0xa7:Saving result file ../logfile_01.txt
07:38:22:WU01:FS00:0xa7:Saving result file md.log
07:38:22:WU01:FS00:0xa7:Saving result file science.log
07:38:22:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)

[bruce]

See https://github.com/FoldingAtHome/fah-issues/issues/1521