Failed???

If you're new to FAH and need help getting started or you have very basic questions, start here.

Moderators: Site Moderators, FAHC Science Team

Post Reply
Harpo
Posts: 29
Joined: Thu Mar 26, 2020 9:10 pm

Failed???

Post by Harpo »

One of my slots showed a "Failed" status this morning, log below. I restarted the card, it's running fine now and temp logs show no problems during the "failure". Any idea what went wrong here?

Code: Select all

00:47:14:WU01:FS00:FahCore 0x22 started
00:47:14:WU01:FS00:0x22:*********************** Log Started 2020-05-15T00:47:14Z ***********************
00:47:14:WU01:FS00:0x22:*************************** Core22 Folding@home Core ***************************
00:47:14:WU01:FS00:0x22:       Type: 0x22
00:47:14:WU01:FS00:0x22:       Core: Core22
00:47:14:WU01:FS00:0x22:    Website: https://foldingathome.org/
00:47:14:WU01:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
00:47:14:WU01:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
00:47:14:WU01:FS00:0x22:             <rafal.wiewiora@choderalab.org>
00:47:14:WU01:FS00:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 8492 -checkpoint 15
00:47:14:WU01:FS00:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
00:47:14:WU01:FS00:0x22:             0 -gpu 0
00:47:14:WU01:FS00:0x22:     Config: <none>
00:47:14:WU01:FS00:0x22:************************************ Build *************************************
00:47:14:WU01:FS00:0x22:    Version: 0.0.5
00:47:14:WU01:FS00:0x22:       Date: Apr 22 2020
00:47:14:WU01:FS00:0x22:       Time: 04:42:59
00:47:14:WU01:FS00:0x22: Repository: Git
00:47:14:WU01:FS00:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
00:47:14:WU01:FS00:0x22:     Branch: HEAD
00:47:14:WU01:FS00:0x22:   Compiler: Visual C++ 2008
00:47:14:WU01:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
00:47:14:WU01:FS00:0x22:   Platform: win32 10
00:47:14:WU01:FS00:0x22:       Bits: 64
00:47:14:WU01:FS00:0x22:       Mode: Release
00:47:14:WU01:FS00:0x22:************************************ System ************************************
00:47:14:WU01:FS00:0x22:        CPU: Intel(R) Core(TM) i5-9600KF CPU @ 3.70GHz
00:47:14:WU01:FS00:0x22:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 13
00:47:14:WU01:FS00:0x22:       CPUs: 6
00:47:14:WU01:FS00:0x22:     Memory: 7.94GiB
00:47:14:WU01:FS00:0x22:Free Memory: 5.01GiB
00:47:14:WU01:FS00:0x22:    Threads: WINDOWS_THREADS
00:47:14:WU01:FS00:0x22: OS Version: 6.2
00:47:14:WU01:FS00:0x22:Has Battery: false
00:47:14:WU01:FS00:0x22: On Battery: false
00:47:14:WU01:FS00:0x22: UTC Offset: -5
00:47:14:WU01:FS00:0x22:        PID: 10200
00:47:14:WU01:FS00:0x22:        CWD: C:\Users\FOLD5\AppData\Roaming\FAHClient\work
00:47:14:WU01:FS00:0x22:         OS: Windows 10 Pro
00:47:14:WU01:FS00:0x22:    OS Arch: AMD64
00:47:14:WU01:FS00:0x22:********************************************************************************
00:47:14:WU01:FS00:0x22:Project: 13404 (Run 695, Clone 11, Gen 0)
00:47:14:WU01:FS00:0x22:Unit: 0x0000000212bc7d9a5ebc6feac4cfa480
00:47:14:WU01:FS00:0x22:Reading tar file core.xml
00:47:14:WU01:FS00:0x22:Reading tar file integrator.xml
00:47:14:WU01:FS00:0x22:Reading tar file state.xml
00:47:14:WU01:FS00:0x22:Reading tar file system.xml
00:47:15:WU01:FS00:0x22:Digital signatures verified
00:47:15:WU01:FS00:0x22:Folding@home GPU Core22 Folding@home Core
00:47:15:WU01:FS00:0x22:Version 0.0.5
00:47:23:WU01:FS00:0x22:Completed 0 out of 1000000 steps (0%)
00:47:23:WU01:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
00:49:37:WU01:FS00:0x22:Completed 10000 out of 1000000 steps (1%)
00:51:50:WU01:FS00:0x22:Completed 20000 out of 1000000 steps (2%)
00:54:03:WU01:FS00:0x22:Completed 30000 out of 1000000 steps (3%)
00:56:17:WU01:FS00:0x22:Completed 40000 out of 1000000 steps (4%)
00:58:30:WU01:FS00:0x22:Completed 50000 out of 1000000 steps (5%)
01:00:44:WU01:FS00:0x22:Completed 60000 out of 1000000 steps (6%)
01:02:57:WU01:FS00:0x22:Completed 70000 out of 1000000 steps (7%)
01:04:23:WU01:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:04:23:WU01:FS00:0x22:Following exception occured: Particle coordinate is nan
01:04:34:WU01:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:04:34:WU01:FS00:0x22:Following exception occured: Particle coordinate is nan
01:04:46:WU01:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:04:46:WU01:FS00:0x22:Following exception occured: Particle coordinate is nan
01:04:46:WU01:FS00:0x22:ERROR:114: Max Retries Reached
01:04:46:WU01:FS00:0x22:Saving result file ..\logfile_01.txt
01:04:46:WU01:FS00:0x22:Saving result file badstate-0.xml
01:04:46:WU01:FS00:0x22:Saving result file badstate-1.xml
01:04:46:WU01:FS00:0x22:Saving result file badstate-2.xml
01:04:46:WU01:FS00:0x22:Saving result file checkpt.crc
01:04:46:WU01:FS00:0x22:Saving result file globals.csv
01:04:46:WU01:FS00:0x22:Saving result file science.log
01:04:46:WU01:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
01:04:46:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
01:04:46:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13404 run:695 clone:11 gen:0 core:0x22 unit:0x0000000212bc7d9a5ebc6feac4cfa480
01:04:46:WU01:FS00:Uploading 104.41KiB to 18.188.125.154
01:04:46:WU01:FS00:Connecting to 18.188.125.154:8080
01:04:47:WU01:FS00:Upload complete
01:04:47:WU01:FS00:Server responded WORK_ACK (400)
01:04:47:WU01:FS00:Cleaning up
01:04:47:WU00:FS00:Connecting to 65.254.110.245:80
01:04:47:WU00:FS00:Assigned to work server 3.133.76.19
01:04:47:WU00:FS00:Requesting new work unit for slot 00: READY gpu:1:TU106-200A [GeForce RTX 2060] from 3.133.76.19
01:04:47:WU00:FS00:Connecting to 3.133.76.19:8080
01:05:08:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
01:05:08:WU00:FS00:Connecting to 3.133.76.19:80
01:05:29:ERROR:WU00:FS00:Exception: Failed to connect to 3.133.76.19:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
01:05:29:WU00:FS00:Connecting to 65.254.110.245:80
01:05:30:WU00:FS00:Assigned to work server 140.163.4.241
01:05:30:WU00:FS00:Requesting new work unit for slot 00: READY gpu:1:TU106-200A [GeForce RTX 2060] from 140.163.4.241
01:05:30:WU00:FS00:Connecting to 140.163.4.241:8080
01:05:37:WU00:FS00:Downloading 4.57MiB
01:05:40:WU00:FS00:Download complete
01:05:40:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:11743 run:0 clone:4227 gen:78 core:0x22 unit:0x000000788ca304f15e69dcd388f3322a
01:05:40:WU00:FS00:Starting
01:05:40:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\FOLD5\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 8544 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
01:05:40:WU00:FS00:Started FahCore on PID 8600
01:05:40:WU00:FS00:Core PID:3400
01:05:40:WU00:FS00:FahCore 0x22 started
01:05:40:WU00:FS00:0x22:*********************** Log Started 2020-05-15T01:05:40Z ***********************
01:05:40:WU00:FS00:0x22:*************************** Core22 Folding@home Core ***************************
01:05:40:WU00:FS00:0x22:       Type: 0x22
01:05:40:WU00:FS00:0x22:       Core: Core22
01:05:40:WU00:FS00:0x22:    Website: https://foldingathome.org/
01:05:40:WU00:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
01:05:40:WU00:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
01:05:40:WU00:FS00:0x22:             <rafal.wiewiora@choderalab.org>
01:05:40:WU00:FS00:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 8600 -checkpoint 15
01:05:40:WU00:FS00:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
01:05:40:WU00:FS00:0x22:             0 -gpu 0
01:05:40:WU00:FS00:0x22:     Config: <none>
01:05:40:WU00:FS00:0x22:************************************ Build *************************************
01:05:40:WU00:FS00:0x22:    Version: 0.0.5
01:05:40:WU00:FS00:0x22:       Date: Apr 22 2020
01:05:40:WU00:FS00:0x22:       Time: 04:42:59
01:05:40:WU00:FS00:0x22: Repository: Git
01:05:40:WU00:FS00:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
01:05:40:WU00:FS00:0x22:     Branch: HEAD
01:05:40:WU00:FS00:0x22:   Compiler: Visual C++ 2008
01:05:40:WU00:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
01:05:40:WU00:FS00:0x22:   Platform: win32 10
01:05:40:WU00:FS00:0x22:       Bits: 64
01:05:40:WU00:FS00:0x22:       Mode: Release
01:05:40:WU00:FS00:0x22:************************************ System ************************************
01:05:40:WU00:FS00:0x22:        CPU: Intel(R) Core(TM) i5-9600KF CPU @ 3.70GHz
01:05:40:WU00:FS00:0x22:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 13
01:05:40:WU00:FS00:0x22:       CPUs: 6
01:05:40:WU00:FS00:0x22:     Memory: 7.94GiB
01:05:40:WU00:FS00:0x22:Free Memory: 5.02GiB
01:05:40:WU00:FS00:0x22:    Threads: WINDOWS_THREADS
01:05:40:WU00:FS00:0x22: OS Version: 6.2
01:05:40:WU00:FS00:0x22:Has Battery: false
01:05:40:WU00:FS00:0x22: On Battery: false
01:05:40:WU00:FS00:0x22: UTC Offset: -5
01:05:40:WU00:FS00:0x22:        PID: 3400
01:05:40:WU00:FS00:0x22:        CWD: C:\Users\FOLD5\AppData\Roaming\FAHClient\work
01:05:40:WU00:FS00:0x22:         OS: Windows 10 Pro
01:05:40:WU00:FS00:0x22:    OS Arch: AMD64
01:05:40:WU00:FS00:0x22:********************************************************************************
01:05:40:WU00:FS00:0x22:Project: 11743 (Run 0, Clone 4227, Gen 78)
01:05:40:WU00:FS00:0x22:Unit: 0x000000788ca304f15e69dcd388f3322a
01:05:40:WU00:FS00:0x22:Reading tar file core.xml
01:05:40:WU00:FS00:0x22:Reading tar file integrator.xml
01:05:40:WU00:FS00:0x22:Reading tar file state.xml
01:05:40:WU00:FS00:0x22:Reading tar file system.xml
01:05:41:WU00:FS00:0x22:Digital signatures verified
01:05:41:WU00:FS00:0x22:Folding@home GPU Core22 Folding@home Core
01:05:41:WU00:FS00:0x22:Version 0.0.5
01:05:45:WU00:FS00:0x22:Completed 0 out of 2000000 steps (0%)
01:05:45:WU00:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
01:06:53:WU00:FS00:0x22:Completed 20000 out of 2000000 steps (1%)
01:08:01:WU00:FS00:0x22:Completed 40000 out of 2000000 steps (2%)
01:09:10:WU00:FS00:0x22:Completed 60000 out of 2000000 steps (3%)
01:10:19:WU00:FS00:0x22:Completed 80000 out of 2000000 steps (4%)
01:11:27:WU00:FS00:0x22:Completed 100000 out of 2000000 steps (5%)
01:12:02:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:12:02:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
01:12:05:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:12:05:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
01:12:08:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:12:08:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
01:12:08:WU00:FS00:0x22:ERROR:114: Max Retries Reached
01:12:08:WU00:FS00:0x22:Saving result file ..\logfile_01.txt
01:12:08:WU00:FS00:0x22:Saving result file badstate-0.xml
01:12:09:WU00:FS00:0x22:Saving result file badstate-1.xml
01:12:10:WU00:FS00:0x22:Saving result file badstate-2.xml
01:12:12:WU00:FS00:0x22:Saving result file checkpointState.xml
01:12:13:WU00:FS00:0x22:Saving result file checkpt.crc
01:12:13:WU00:FS00:0x22:Saving result file positions.xtc
01:12:13:WU00:FS00:0x22:Saving result file science.log
01:12:13:WU00:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
01:12:14:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
01:12:14:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:11743 run:0 clone:4227 gen:78 core:0x22 unit:0x000000788ca304f15e69dcd388f3322a
01:12:14:WU00:FS00:Uploading 4.33MiB to 140.163.4.241
01:12:14:WU00:FS00:Connecting to 140.163.4.241:8080
01:12:14:WU01:FS00:Connecting to 65.254.110.245:80
01:12:14:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
01:12:14:WU01:FS00:Connecting to 18.218.241.186:80
01:12:14:WU01:FS00:Assigned to work server 18.188.125.154
01:12:14:WU01:FS00:Requesting new work unit for slot 00: READY gpu:1:TU106-200A [GeForce RTX 2060] from 18.188.125.154
01:12:14:WU01:FS00:Connecting to 18.188.125.154:8080
01:12:16:WU01:FS00:Downloading 6.26MiB
01:12:17:WU01:FS00:Download complete
01:12:17:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13404 run:482 clone:65 gen:1 core:0x22 unit:0x0000000112bc7d9a5ebc6ff03690c873
01:12:17:WU01:FS00:Starting
01:12:17:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\FOLD5\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 8544 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
01:12:17:WU01:FS00:Started FahCore on PID 6032
01:12:17:WU01:FS00:Core PID:1208
01:12:17:WU01:FS00:FahCore 0x22 started
01:12:17:WU01:FS00:0x22:*********************** Log Started 2020-05-15T01:12:17Z ***********************
01:12:17:WU01:FS00:0x22:*************************** Core22 Folding@home Core ***************************
01:12:17:WU01:FS00:0x22:       Type: 0x22
01:12:17:WU01:FS00:0x22:       Core: Core22
01:12:17:WU01:FS00:0x22:    Website: https://foldingathome.org/
01:12:17:WU01:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
01:12:17:WU01:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
01:12:17:WU01:FS00:0x22:             <rafal.wiewiora@choderalab.org>
01:12:17:WU01:FS00:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 6032 -checkpoint 15
01:12:17:WU01:FS00:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
01:12:17:WU01:FS00:0x22:             0 -gpu 0
01:12:17:WU01:FS00:0x22:     Config: <none>
01:12:17:WU01:FS00:0x22:************************************ Build *************************************
01:12:17:WU01:FS00:0x22:    Version: 0.0.5
01:12:17:WU01:FS00:0x22:       Date: Apr 22 2020
01:12:17:WU01:FS00:0x22:       Time: 04:42:59
01:12:17:WU01:FS00:0x22: Repository: Git
01:12:17:WU01:FS00:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
01:12:17:WU01:FS00:0x22:     Branch: HEAD
01:12:17:WU01:FS00:0x22:   Compiler: Visual C++ 2008
01:12:17:WU01:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
01:12:17:WU01:FS00:0x22:   Platform: win32 10
01:12:17:WU01:FS00:0x22:       Bits: 64
01:12:17:WU01:FS00:0x22:       Mode: Release
01:12:17:WU01:FS00:0x22:************************************ System ************************************
01:12:17:WU01:FS00:0x22:        CPU: Intel(R) Core(TM) i5-9600KF CPU @ 3.70GHz
01:12:17:WU01:FS00:0x22:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 13
01:12:17:WU01:FS00:0x22:       CPUs: 6
01:12:17:WU01:FS00:0x22:     Memory: 7.94GiB
01:12:17:WU01:FS00:0x22:Free Memory: 4.99GiB
01:12:17:WU01:FS00:0x22:    Threads: WINDOWS_THREADS
01:12:17:WU01:FS00:0x22: OS Version: 6.2
01:12:17:WU01:FS00:0x22:Has Battery: false
01:12:17:WU01:FS00:0x22: On Battery: false
01:12:17:WU01:FS00:0x22: UTC Offset: -5
01:12:17:WU01:FS00:0x22:        PID: 1208
01:12:17:WU01:FS00:0x22:        CWD: C:\Users\FOLD5\AppData\Roaming\FAHClient\work
01:12:17:WU01:FS00:0x22:         OS: Windows 10 Pro
01:12:17:WU01:FS00:0x22:    OS Arch: AMD64
01:12:17:WU01:FS00:0x22:********************************************************************************
01:12:17:WU01:FS00:0x22:Project: 13404 (Run 482, Clone 65, Gen 1)
01:12:17:WU01:FS00:0x22:Unit: 0x0000000112bc7d9a5ebc6ff03690c873
01:12:17:WU01:FS00:0x22:Reading tar file core.xml
01:12:17:WU01:FS00:0x22:Reading tar file integrator.xml
01:12:17:WU01:FS00:0x22:Reading tar file state.xml
01:12:17:WU01:FS00:0x22:Reading tar file system.xml
01:12:18:WU01:FS00:0x22:Digital signatures verified
01:12:18:WU01:FS00:0x22:Folding@home GPU Core22 Folding@home Core
01:12:18:WU01:FS00:0x22:Version 0.0.5
01:12:19:WU00:FS00:Upload complete
01:12:19:WU00:FS00:Server responded WORK_ACK (400)
01:12:19:WU00:FS00:Cleaning up
01:12:26:WU01:FS00:0x22:Completed 0 out of 1000000 steps (0%)
01:12:26:WU01:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
01:14:40:WU01:FS00:0x22:Completed 10000 out of 1000000 steps (1%)
01:16:54:WU01:FS00:0x22:Completed 20000 out of 1000000 steps (2%)
01:19:08:WU01:FS00:0x22:Completed 30000 out of 1000000 steps (3%)
01:21:21:WU01:FS00:0x22:Completed 40000 out of 1000000 steps (4%)
01:23:35:WU01:FS00:0x22:Completed 50000 out of 1000000 steps (5%)
01:25:06:WU01:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:25:06:WU01:FS00:0x22:Following exception occured: Particle coordinate is nan
01:25:18:WU01:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:25:18:WU01:FS00:0x22:Following exception occured: Particle coordinate is nan
01:25:29:WU01:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:25:29:WU01:FS00:0x22:Following exception occured: Particle coordinate is nan
01:25:29:WU01:FS00:0x22:ERROR:114: Max Retries Reached
01:25:29:WU01:FS00:0x22:Saving result file ..\logfile_01.txt
01:25:29:WU01:FS00:0x22:Saving result file badstate-0.xml
01:25:29:WU01:FS00:0x22:Saving result file badstate-1.xml
01:25:29:WU01:FS00:0x22:Saving result file badstate-2.xml
01:25:29:WU01:FS00:0x22:Saving result file checkpt.crc
01:25:29:WU01:FS00:0x22:Saving result file globals.csv
01:25:29:WU01:FS00:0x22:Saving result file science.log
01:25:29:WU01:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
01:25:29:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
01:25:29:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13404 run:482 clone:65 gen:1 core:0x22 unit:0x0000000112bc7d9a5ebc6ff03690c873
01:25:29:WU01:FS00:Uploading 100.16KiB to 18.188.125.154
01:25:30:WU01:FS00:Connecting to 18.188.125.154:8080
01:25:30:WU01:FS00:Upload complete
01:25:30:WU01:FS00:Server responded WORK_ACK (400)
01:25:30:WU01:FS00:Cleaning up
01:25:30:WU00:FS00:Connecting to 65.254.110.245:80
01:25:30:WU00:FS00:Assigned to work server 13.82.98.119
01:25:30:WU00:FS00:Requesting new work unit for slot 00: READY gpu:1:TU106-200A [GeForce RTX 2060] from 13.82.98.119
01:25:30:WU00:FS00:Connecting to 13.82.98.119:8080
01:26:03:WU00:FS00:Downloading 68.54MiB
01:26:09:WU00:FS00:Download 28.72%
01:26:15:WU00:FS00:Download 54.53%
01:26:21:WU00:FS00:Download 87.99%
01:26:22:WU00:FS00:Download complete
01:26:23:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14431 run:0 clone:437 gen:65 core:0x22 unit:0x000000600d5262775e8b4d5d06bdace2
01:26:23:WU00:FS00:Starting
01:26:23:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\FOLD5\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 8544 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
01:26:23:WU00:FS00:Started FahCore on PID 7156
01:26:23:WU00:FS00:Core PID:9560
01:26:23:WU00:FS00:FahCore 0x22 started
01:26:23:WU00:FS00:0x22:*********************** Log Started 2020-05-15T01:26:23Z ***********************
01:26:23:WU00:FS00:0x22:*************************** Core22 Folding@home Core ***************************
01:26:23:WU00:FS00:0x22:       Type: 0x22
01:26:23:WU00:FS00:0x22:       Core: Core22
01:26:23:WU00:FS00:0x22:    Website: https://foldingathome.org/
01:26:23:WU00:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
01:26:23:WU00:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
01:26:23:WU00:FS00:0x22:             <rafal.wiewiora@choderalab.org>
01:26:23:WU00:FS00:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 7156 -checkpoint 15
01:26:23:WU00:FS00:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
01:26:23:WU00:FS00:0x22:             0 -gpu 0
01:26:23:WU00:FS00:0x22:     Config: <none>
01:26:23:WU00:FS00:0x22:************************************ Build *************************************
01:26:23:WU00:FS00:0x22:    Version: 0.0.5
01:26:23:WU00:FS00:0x22:       Date: Apr 22 2020
01:26:23:WU00:FS00:0x22:       Time: 04:42:59
01:26:23:WU00:FS00:0x22: Repository: Git
01:26:23:WU00:FS00:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
01:26:23:WU00:FS00:0x22:     Branch: HEAD
01:26:23:WU00:FS00:0x22:   Compiler: Visual C++ 2008
01:26:23:WU00:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
01:26:23:WU00:FS00:0x22:   Platform: win32 10
01:26:23:WU00:FS00:0x22:       Bits: 64
01:26:23:WU00:FS00:0x22:       Mode: Release
01:26:23:WU00:FS00:0x22:************************************ System ************************************
01:26:23:WU00:FS00:0x22:        CPU: Intel(R) Core(TM) i5-9600KF CPU @ 3.70GHz
01:26:23:WU00:FS00:0x22:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 13
01:26:23:WU00:FS00:0x22:       CPUs: 6
01:26:23:WU00:FS00:0x22:     Memory: 7.94GiB
01:26:23:WU00:FS00:0x22:Free Memory: 4.99GiB
01:26:23:WU00:FS00:0x22:    Threads: WINDOWS_THREADS
01:26:23:WU00:FS00:0x22: OS Version: 6.2
01:26:23:WU00:FS00:0x22:Has Battery: false
01:26:23:WU00:FS00:0x22: On Battery: false
01:26:23:WU00:FS00:0x22: UTC Offset: -5
01:26:23:WU00:FS00:0x22:        PID: 9560
01:26:23:WU00:FS00:0x22:        CWD: C:\Users\FOLD5\AppData\Roaming\FAHClient\work
01:26:23:WU00:FS00:0x22:         OS: Windows 10 Pro
01:26:23:WU00:FS00:0x22:    OS Arch: AMD64
01:26:23:WU00:FS00:0x22:********************************************************************************
01:26:23:WU00:FS00:0x22:Project: 14431 (Run 0, Clone 437, Gen 65)
01:26:23:WU00:FS00:0x22:Unit: 0x000000600d5262775e8b4d5d06bdace2
01:26:23:WU00:FS00:0x22:Reading tar file core.xml
01:26:23:WU00:FS00:0x22:Reading tar file integrator.xml
01:26:23:WU00:FS00:0x22:Reading tar file state.xml
01:26:23:WU00:FS00:0x22:Reading tar file system.xml
01:26:24:WU00:FS00:0x22:Digital signatures verified
01:26:24:WU00:FS00:0x22:Folding@home GPU Core22 Folding@home Core
01:26:24:WU00:FS00:0x22:Version 0.0.5
01:26:33:WU00:FS00:0x22:Completed 0 out of 2000000 steps (0%)
01:26:33:WU00:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
01:27:09:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:27:09:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
01:28:19:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:28:19:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
01:30:17:WU00:FS00:0x22:Completed 20000 out of 2000000 steps (1%)
01:32:16:WU00:FS00:0x22:Completed 40000 out of 2000000 steps (2%)
01:34:14:WU00:FS00:0x22:Completed 60000 out of 2000000 steps (3%)
01:34:55:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:34:55:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
01:34:55:WU00:FS00:0x22:ERROR:114: Max Retries Reached
01:34:55:WU00:FS00:0x22:Saving result file ..\logfile_01.txt
01:34:55:WU00:FS00:0x22:Saving result file badstate-0.xml
01:34:55:WU00:FS00:0x22:Saving result file badstate-1.xml
01:34:55:WU00:FS00:0x22:Saving result file badstate-2.xml
01:34:55:WU00:FS00:0x22:Saving result file checkpt.crc
01:34:55:WU00:FS00:0x22:Saving result file science.log
01:34:55:WU00:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
01:34:56:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
01:34:56:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:14431 run:0 clone:437 gen:65 core:0x22 unit:0x000000600d5262775e8b4d5d06bdace2
01:34:56:WU00:FS00:Uploading 50.92MiB to 13.82.98.119
01:34:56:WU00:FS00:Connecting to 13.82.98.119:8080
01:34:56:WU01:FS00:Connecting to 65.254.110.245:80
01:34:56:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
01:34:56:WU01:FS00:Connecting to 18.218.241.186:80
01:34:56:WU01:FS00:Assigned to work server 3.133.76.19
01:34:56:WU01:FS00:Requesting new work unit for slot 00: READY gpu:1:TU106-200A [GeForce RTX 2060] from 3.133.76.19
01:34:56:WU01:FS00:Connecting to 3.133.76.19:8080
01:35:02:WU00:FS00:Upload 9.45%
01:35:08:WU00:FS00:Upload 24.30%
01:35:14:WU00:FS00:Upload 39.15%
01:35:20:WU00:FS00:Upload 49.83%
01:35:26:WU00:FS00:Upload 65.30%
01:35:32:WU00:FS00:Upload 79.54%
01:35:38:WU00:FS00:Upload 93.28%
01:35:41:WU00:FS00:Upload complete
01:35:41:WU00:FS00:Server responded WORK_ACK (400)
01:35:41:WU00:FS00:Cleaning up
01:35:51:WU01:FS00:Downloading 19.12MiB
01:35:54:WU01:FS00:Download complete
01:35:54:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14449 run:0 clone:1690 gen:3 core:0x22 unit:0x0000000b03854c135eac4ad1e08696c7
01:35:54:WU01:FS00:Starting
01:35:54:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\FOLD5\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 8544 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
01:35:54:WU01:FS00:Started FahCore on PID 2436
01:35:54:WU01:FS00:Core PID:3588
01:35:54:WU01:FS00:FahCore 0x22 started
01:35:54:WU01:FS00:0x22:*********************** Log Started 2020-05-15T01:35:54Z ***********************
01:35:54:WU01:FS00:0x22:*************************** Core22 Folding@home Core ***************************
01:35:54:WU01:FS00:0x22:       Type: 0x22
01:35:54:WU01:FS00:0x22:       Core: Core22
01:35:54:WU01:FS00:0x22:    Website: https://foldingathome.org/
01:35:54:WU01:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
01:35:54:WU01:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
01:35:54:WU01:FS00:0x22:             <rafal.wiewiora@choderalab.org>
01:35:54:WU01:FS00:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 2436 -checkpoint 15
01:35:54:WU01:FS00:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
01:35:54:WU01:FS00:0x22:             0 -gpu 0
01:35:54:WU01:FS00:0x22:     Config: <none>
01:35:54:WU01:FS00:0x22:************************************ Build *************************************
01:35:54:WU01:FS00:0x22:    Version: 0.0.5
01:35:54:WU01:FS00:0x22:       Date: Apr 22 2020
01:35:54:WU01:FS00:0x22:       Time: 04:42:59
01:35:54:WU01:FS00:0x22: Repository: Git
01:35:54:WU01:FS00:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
01:35:54:WU01:FS00:0x22:     Branch: HEAD
01:35:54:WU01:FS00:0x22:   Compiler: Visual C++ 2008
01:35:54:WU01:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
01:35:54:WU01:FS00:0x22:   Platform: win32 10
01:35:54:WU01:FS00:0x22:       Bits: 64
01:35:54:WU01:FS00:0x22:       Mode: Release
01:35:54:WU01:FS00:0x22:************************************ System ************************************
01:35:54:WU01:FS00:0x22:        CPU: Intel(R) Core(TM) i5-9600KF CPU @ 3.70GHz
01:35:54:WU01:FS00:0x22:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 13
01:35:54:WU01:FS00:0x22:       CPUs: 6
01:35:54:WU01:FS00:0x22:     Memory: 7.94GiB
01:35:54:WU01:FS00:0x22:Free Memory: 5.00GiB
01:35:54:WU01:FS00:0x22:    Threads: WINDOWS_THREADS
01:35:54:WU01:FS00:0x22: OS Version: 6.2
01:35:54:WU01:FS00:0x22:Has Battery: false
01:35:54:WU01:FS00:0x22: On Battery: false
01:35:54:WU01:FS00:0x22: UTC Offset: -5
01:35:54:WU01:FS00:0x22:        PID: 3588
01:35:54:WU01:FS00:0x22:        CWD: C:\Users\FOLD5\AppData\Roaming\FAHClient\work
01:35:54:WU01:FS00:0x22:         OS: Windows 10 Pro
01:35:54:WU01:FS00:0x22:    OS Arch: AMD64
01:35:54:WU01:FS00:0x22:********************************************************************************
01:35:54:WU01:FS00:0x22:Project: 14449 (Run 0, Clone 1690, Gen 3)
01:35:54:WU01:FS00:0x22:Unit: 0x0000000b03854c135eac4ad1e08696c7
01:35:54:WU01:FS00:0x22:Reading tar file core.xml
01:35:54:WU01:FS00:0x22:Reading tar file integrator.xml
01:35:54:WU01:FS00:0x22:Reading tar file state.xml
01:35:55:WU01:FS00:0x22:Reading tar file system.xml
01:35:56:WU01:FS00:0x22:Digital signatures verified
01:35:56:WU01:FS00:0x22:Folding@home GPU Core22 Folding@home Core
01:35:56:WU01:FS00:0x22:Version 0.0.5
01:36:08:WU01:FS00:0x22:Completed 0 out of 2000000 steps (0%)
01:36:08:WU01:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
01:36:24:WU01:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:36:24:WU01:FS00:0x22:Following exception occured: Particle coordinate is nan
01:37:32:WU01:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:37:32:WU01:FS00:0x22:Following exception occured: Particle coordinate is nan
01:38:18:WU01:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:38:18:WU01:FS00:0x22:Following exception occured: Particle coordinate is nan
01:38:18:WU01:FS00:0x22:ERROR:114: Max Retries Reached
01:38:18:WU01:FS00:0x22:Saving result file ..\logfile_01.txt
01:38:18:WU01:FS00:0x22:Saving result file badstate-0.xml
01:38:18:WU01:FS00:0x22:Saving result file badstate-1.xml
01:38:18:WU01:FS00:0x22:Saving result file badstate-2.xml
01:38:18:WU01:FS00:0x22:Saving result file checkpt.crc
01:38:18:WU01:FS00:0x22:Saving result file science.log
01:38:18:WU01:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
01:38:19:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
01:38:19:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:14449 run:0 clone:1690 gen:3 core:0x22 unit:0x0000000b03854c135eac4ad1e08696c7
01:38:19:WU01:FS00:Uploading 1.71MiB to 3.133.76.19
01:38:19:WU01:FS00:Connecting to 3.133.76.19:8080
01:38:19:WU00:FS00:Connecting to 65.254.110.245:80
01:38:20:WU00:FS00:Assigned to work server 3.133.76.19
01:38:20:WU00:FS00:Requesting new work unit for slot 00: READY gpu:1:TU106-200A [GeForce RTX 2060] from 3.133.76.19
01:38:20:WU00:FS00:Connecting to 3.133.76.19:8080
01:38:40:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
01:38:40:WU01:FS00:Connecting to 3.133.76.19:80
01:39:01:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to connect to 3.133.76.19:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
01:39:02:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:14449 run:0 clone:1690 gen:3 core:0x22 unit:0x0000000b03854c135eac4ad1e08696c7
01:39:02:WU01:FS00:Uploading 1.71MiB to 3.133.76.19
01:39:02:WU01:FS00:Connecting to 3.133.76.19:8080
01:39:22:WU00:FS00:Downloading 17.39MiB
01:39:23:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
01:39:23:WU01:FS00:Connecting to 3.133.76.19:80
01:39:23:WU01:FS00:Upload 3.65%
01:39:24:WU00:FS00:Download complete
01:39:24:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14436 run:1823 clone:1 gen:41 core:0x22 unit:0x0000003d03854c135e9a790ebe2d20b9
01:39:24:WU00:FS00:Starting
01:39:24:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\FOLD5\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 8544 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
01:39:24:WU00:FS00:Started FahCore on PID 1536
01:39:24:WU00:FS00:Core PID:2416
01:39:24:WU00:FS00:FahCore 0x22 started
01:39:25:WU00:FS00:0x22:*********************** Log Started 2020-05-15T01:39:24Z ***********************
01:39:25:WU00:FS00:0x22:*************************** Core22 Folding@home Core ***************************
01:39:25:WU00:FS00:0x22:       Type: 0x22
01:39:25:WU00:FS00:0x22:       Core: Core22
01:39:25:WU00:FS00:0x22:    Website: https://foldingathome.org/
01:39:25:WU00:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
01:39:25:WU00:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
01:39:25:WU00:FS00:0x22:             <rafal.wiewiora@choderalab.org>
01:39:25:WU00:FS00:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 1536 -checkpoint 15
01:39:25:WU00:FS00:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
01:39:25:WU00:FS00:0x22:             0 -gpu 0
01:39:25:WU00:FS00:0x22:     Config: <none>
01:39:25:WU00:FS00:0x22:************************************ Build *************************************
01:39:25:WU00:FS00:0x22:    Version: 0.0.5
01:39:25:WU00:FS00:0x22:       Date: Apr 22 2020
01:39:25:WU00:FS00:0x22:       Time: 04:42:59
01:39:25:WU00:FS00:0x22: Repository: Git
01:39:25:WU00:FS00:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
01:39:25:WU00:FS00:0x22:     Branch: HEAD
01:39:25:WU00:FS00:0x22:   Compiler: Visual C++ 2008
01:39:25:WU00:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
01:39:25:WU00:FS00:0x22:   Platform: win32 10
01:39:25:WU00:FS00:0x22:       Bits: 64
01:39:25:WU00:FS00:0x22:       Mode: Release
01:39:25:WU00:FS00:0x22:************************************ System ************************************
01:39:25:WU00:FS00:0x22:        CPU: Intel(R) Core(TM) i5-9600KF CPU @ 3.70GHz
01:39:25:WU00:FS00:0x22:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 13
01:39:25:WU00:FS00:0x22:       CPUs: 6
01:39:25:WU00:FS00:0x22:     Memory: 7.94GiB
01:39:25:WU00:FS00:0x22:Free Memory: 5.00GiB
01:39:25:WU00:FS00:0x22:    Threads: WINDOWS_THREADS
01:39:25:WU00:FS00:0x22: OS Version: 6.2
01:39:25:WU00:FS00:0x22:Has Battery: false
01:39:25:WU00:FS00:0x22: On Battery: false
01:39:25:WU00:FS00:0x22: UTC Offset: -5
01:39:25:WU00:FS00:0x22:        PID: 2416
01:39:25:WU00:FS00:0x22:        CWD: C:\Users\FOLD5\AppData\Roaming\FAHClient\work
01:39:25:WU00:FS00:0x22:         OS: Windows 10 Pro
01:39:25:WU00:FS00:0x22:    OS Arch: AMD64
01:39:25:WU00:FS00:0x22:********************************************************************************
01:39:25:WU00:FS00:0x22:Project: 14436 (Run 1823, Clone 1, Gen 41)
01:39:25:WU00:FS00:0x22:Unit: 0x0000003d03854c135e9a790ebe2d20b9
01:39:25:WU00:FS00:0x22:Reading tar file core.xml
01:39:25:WU00:FS00:0x22:Reading tar file integrator.xml
01:39:25:WU00:FS00:0x22:Reading tar file state.xml
01:39:25:WU00:FS00:0x22:Reading tar file system.xml
01:39:26:WU00:FS00:0x22:Digital signatures verified
01:39:26:WU00:FS00:0x22:Folding@home GPU Core22 Folding@home Core
01:39:26:WU00:FS00:0x22:Version 0.0.5
01:39:37:WU00:FS00:0x22:Completed 0 out of 2000000 steps (0%)
01:39:37:WU00:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
01:39:42:WU01:FS00:Upload 7.30%
01:39:42:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
01:40:02:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:14449 run:0 clone:1690 gen:3 core:0x22 unit:0x0000000b03854c135eac4ad1e08696c7
01:40:02:WU01:FS00:Uploading 1.71MiB to 3.133.76.19
01:40:02:WU01:FS00:Connecting to 3.133.76.19:8080
01:40:23:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
01:40:23:WU01:FS00:Connecting to 3.133.76.19:80
01:40:44:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to connect to 3.133.76.19:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
01:41:03:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:41:03:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
01:41:39:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:14449 run:0 clone:1690 gen:3 core:0x22 unit:0x0000000b03854c135eac4ad1e08696c7
01:41:39:WU01:FS00:Uploading 1.71MiB to 3.133.76.19
01:41:39:WU01:FS00:Connecting to 3.133.76.19:8080
01:42:00:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
01:42:00:WU01:FS00:Connecting to 3.133.76.19:80
01:42:00:WU01:FS00:Upload 3.65%
01:42:50:WU01:FS00:Upload 10.96%
01:42:51:WU01:FS00:Upload complete
01:42:51:WU01:FS00:Server responded WORK_ACK (400)
01:42:51:WU01:FS00:Cleaning up
01:43:45:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:43:45:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
01:44:17:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:44:17:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
01:44:17:WU00:FS00:0x22:ERROR:114: Max Retries Reached
01:44:17:WU00:FS00:0x22:Saving result file ..\logfile_01.txt
01:44:17:WU00:FS00:0x22:Saving result file badstate-0.xml
01:44:17:WU00:FS00:0x22:Saving result file badstate-1.xml
01:44:18:WU00:FS00:0x22:Saving result file badstate-2.xml
01:44:18:WU00:FS00:0x22:Saving result file checkpt.crc
01:44:18:WU00:FS00:0x22:Saving result file science.log
01:44:18:WU00:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
01:44:18:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
01:44:18:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:14436 run:1823 clone:1 gen:41 core:0x22 unit:0x0000003d03854c135e9a790ebe2d20b9
01:44:18:WU00:FS00:Uploading 181.22KiB to 3.133.76.19
01:44:18:WU00:FS00:Connecting to 3.133.76.19:8080
01:44:19:WU01:FS00:Connecting to 65.254.110.245:80
01:44:19:WU01:FS00:Assigned to work server 128.252.203.2
01:44:19:WU01:FS00:Requesting new work unit for slot 00: READY gpu:1:TU106-200A [GeForce RTX 2060] from 128.252.203.2
01:44:19:WU01:FS00:Connecting to 128.252.203.2:8080
01:44:39:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
01:44:39:WU00:FS00:Connecting to 3.133.76.19:80
01:45:01:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to connect to 3.133.76.19:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
01:45:01:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:14436 run:1823 clone:1 gen:41 core:0x22 unit:0x0000003d03854c135e9a790ebe2d20b9
01:45:01:WU00:FS00:Uploading 181.22KiB to 3.133.76.19
01:45:01:WU00:FS00:Connecting to 3.133.76.19:8080
01:45:11:WU01:FS00:Downloading 33.93MiB
01:45:16:WU01:FS00:Download complete
01:45:16:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14563 run:0 clone:457 gen:38 core:0x22 unit:0x0000003480fccb025e9615b919e712a2
01:45:16:WU01:FS00:Starting
01:45:16:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\FOLD5\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 8544 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
01:45:16:WU01:FS00:Started FahCore on PID 4800
01:45:16:WU01:FS00:Core PID:3964
01:45:16:WU01:FS00:FahCore 0x22 started
01:45:17:WU01:FS00:0x22:*********************** Log Started 2020-05-15T01:45:16Z ***********************
01:45:17:WU01:FS00:0x22:*************************** Core22 Folding@home Core ***************************
01:45:17:WU01:FS00:0x22:       Type: 0x22
01:45:17:WU01:FS00:0x22:       Core: Core22
01:45:17:WU01:FS00:0x22:    Website: https://foldingathome.org/
01:45:17:WU01:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
01:45:17:WU01:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
01:45:17:WU01:FS00:0x22:             <rafal.wiewiora@choderalab.org>
01:45:17:WU01:FS00:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 4800 -checkpoint 15
01:45:17:WU01:FS00:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
01:45:17:WU01:FS00:0x22:             0 -gpu 0
01:45:17:WU01:FS00:0x22:     Config: <none>
01:45:17:WU01:FS00:0x22:************************************ Build *************************************
01:45:17:WU01:FS00:0x22:    Version: 0.0.5
01:45:17:WU01:FS00:0x22:       Date: Apr 22 2020
01:45:17:WU01:FS00:0x22:       Time: 04:42:59
01:45:17:WU01:FS00:0x22: Repository: Git
01:45:17:WU01:FS00:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
01:45:17:WU01:FS00:0x22:     Branch: HEAD
01:45:17:WU01:FS00:0x22:   Compiler: Visual C++ 2008
01:45:17:WU01:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
01:45:17:WU01:FS00:0x22:   Platform: win32 10
01:45:17:WU01:FS00:0x22:       Bits: 64
01:45:17:WU01:FS00:0x22:       Mode: Release
01:45:17:WU01:FS00:0x22:************************************ System ************************************
01:45:17:WU01:FS00:0x22:        CPU: Intel(R) Core(TM) i5-9600KF CPU @ 3.70GHz
01:45:17:WU01:FS00:0x22:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 13
01:45:17:WU01:FS00:0x22:       CPUs: 6
01:45:17:WU01:FS00:0x22:     Memory: 7.94GiB
01:45:17:WU01:FS00:0x22:Free Memory: 4.98GiB
01:45:17:WU01:FS00:0x22:    Threads: WINDOWS_THREADS
01:45:17:WU01:FS00:0x22: OS Version: 6.2
01:45:17:WU01:FS00:0x22:Has Battery: false
01:45:17:WU01:FS00:0x22: On Battery: false
01:45:17:WU01:FS00:0x22: UTC Offset: -5
01:45:17:WU01:FS00:0x22:        PID: 3964
01:45:17:WU01:FS00:0x22:        CWD: C:\Users\FOLD5\AppData\Roaming\FAHClient\work
01:45:17:WU01:FS00:0x22:         OS: Windows 10 Pro
01:45:17:WU01:FS00:0x22:    OS Arch: AMD64
01:45:17:WU01:FS00:0x22:********************************************************************************
01:45:17:WU01:FS00:0x22:Project: 14563 (Run 0, Clone 457, Gen 38)
01:45:17:WU01:FS00:0x22:Unit: 0x0000003480fccb025e9615b919e712a2
01:45:17:WU01:FS00:0x22:Reading tar file core.xml
01:45:17:WU01:FS00:0x22:Reading tar file integrator.xml
01:45:17:WU01:FS00:0x22:Reading tar file state.xml
01:45:21:WU01:FS00:0x22:Reading tar file system.xml
01:45:23:WU01:FS00:0x22:Digital signatures verified
01:45:23:WU01:FS00:0x22:Folding@home GPU Core22 Folding@home Core
01:45:23:WU01:FS00:0x22:Version 0.0.5
01:45:54:WU01:FS00:0x22:Completed 0 out of 500000 steps (0%)
01:45:54:WU01:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
01:45:59:WU00:FS00:Upload 100.00%
01:46:00:WU00:FS00:Upload complete
01:46:00:WU00:FS00:Server responded WORK_ACK (400)
01:46:00:WU00:FS00:Cleaning up
01:48:16:WU01:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:48:16:WU01:FS00:0x22:Following exception occured: Particle coordinate is nan
01:49:23:WU01:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:49:23:WU01:FS00:0x22:Following exception occured: Particle coordinate is nan
01:50:07:WU01:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:50:07:WU01:FS00:0x22:Following exception occured: Particle coordinate is nan
01:50:08:WU01:FS00:0x22:ERROR:114: Max Retries Reached
01:50:08:WU01:FS00:0x22:Saving result file ..\logfile_01.txt
01:50:08:WU01:FS00:0x22:Saving result file badstate-0.xml
01:50:18:WU01:FS00:0x22:Saving result file badstate-1.xml
01:50:28:WU01:FS00:0x22:Saving result file badstate-2.xml
01:50:38:WU01:FS00:0x22:Saving result file checkpt.crc
01:50:38:WU01:FS00:0x22:Saving result file science.log
01:50:38:WU01:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
01:50:39:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
01:50:39:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:14563 run:0 clone:457 gen:38 core:0x22 unit:0x0000003480fccb025e9615b919e712a2
01:50:39:WU01:FS00:Uploading 27.58KiB to 128.252.203.2
01:50:39:WU01:FS00:Connecting to 128.252.203.2:8080
01:50:39:WU00:FS00:Connecting to 65.254.110.245:80
01:50:39:WU00:FS00:Assigned to work server 128.252.203.2
01:50:39:WU00:FS00:Requesting new work unit for slot 00: READY gpu:1:TU106-200A [GeForce RTX 2060] from 128.252.203.2
01:50:39:WU00:FS00:Connecting to 128.252.203.2:8080
01:50:41:WU01:FS00:Upload complete
01:50:42:WU01:FS00:Server responded WORK_ACK (400)
01:50:42:WU01:FS00:Cleaning up
01:51:23:WU00:FS00:Downloading 28.39MiB
01:51:29:WU00:FS00:Download 74.62%
01:51:31:WU00:FS00:Download complete
01:51:31:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14562 run:0 clone:57 gen:43 core:0x22 unit:0x0000003f80fccb025e9615bedeab90eb
01:51:31:WU00:FS00:Starting
01:51:31:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\FOLD5\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 8544 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
01:51:31:WU00:FS00:Started FahCore on PID 8912
01:51:31:WU00:FS00:Core PID:568
01:51:31:WU00:FS00:FahCore 0x22 started
01:51:32:WU00:FS00:0x22:*********************** Log Started 2020-05-15T01:51:31Z ***********************
01:51:32:WU00:FS00:0x22:*************************** Core22 Folding@home Core ***************************
01:51:32:WU00:FS00:0x22:       Type: 0x22
01:51:32:WU00:FS00:0x22:       Core: Core22
01:51:32:WU00:FS00:0x22:    Website: https://foldingathome.org/
01:51:32:WU00:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
01:51:32:WU00:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
01:51:32:WU00:FS00:0x22:             <rafal.wiewiora@choderalab.org>
01:51:32:WU00:FS00:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 8912 -checkpoint 15
01:51:32:WU00:FS00:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
01:51:32:WU00:FS00:0x22:             0 -gpu 0
01:51:32:WU00:FS00:0x22:     Config: <none>
01:51:32:WU00:FS00:0x22:************************************ Build *************************************
01:51:32:WU00:FS00:0x22:    Version: 0.0.5
01:51:32:WU00:FS00:0x22:       Date: Apr 22 2020
01:51:32:WU00:FS00:0x22:       Time: 04:42:59
01:51:32:WU00:FS00:0x22: Repository: Git
01:51:32:WU00:FS00:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
01:51:32:WU00:FS00:0x22:     Branch: HEAD
01:51:32:WU00:FS00:0x22:   Compiler: Visual C++ 2008
01:51:32:WU00:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
01:51:32:WU00:FS00:0x22:   Platform: win32 10
01:51:32:WU00:FS00:0x22:       Bits: 64
01:51:32:WU00:FS00:0x22:       Mode: Release
01:51:32:WU00:FS00:0x22:************************************ System ************************************
01:51:32:WU00:FS00:0x22:        CPU: Intel(R) Core(TM) i5-9600KF CPU @ 3.70GHz
01:51:32:WU00:FS00:0x22:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 13
01:51:32:WU00:FS00:0x22:       CPUs: 6
01:51:32:WU00:FS00:0x22:     Memory: 7.94GiB
01:51:32:WU00:FS00:0x22:Free Memory: 5.01GiB
01:51:32:WU00:FS00:0x22:    Threads: WINDOWS_THREADS
01:51:32:WU00:FS00:0x22: OS Version: 6.2
01:51:32:WU00:FS00:0x22:Has Battery: false
01:51:32:WU00:FS00:0x22: On Battery: false
01:51:32:WU00:FS00:0x22: UTC Offset: -5
01:51:32:WU00:FS00:0x22:        PID: 568
01:51:32:WU00:FS00:0x22:        CWD: C:\Users\FOLD5\AppData\Roaming\FAHClient\work
01:51:32:WU00:FS00:0x22:         OS: Windows 10 Pro
01:51:32:WU00:FS00:0x22:    OS Arch: AMD64
01:51:32:WU00:FS00:0x22:********************************************************************************
01:51:32:WU00:FS00:0x22:Project: 14562 (Run 0, Clone 57, Gen 43)
01:51:32:WU00:FS00:0x22:Unit: 0x0000003f80fccb025e9615bedeab90eb
01:51:32:WU00:FS00:0x22:Reading tar file core.xml
01:51:32:WU00:FS00:0x22:Reading tar file integrator.xml
01:51:32:WU00:FS00:0x22:Reading tar file state.xml
01:51:35:WU00:FS00:0x22:Reading tar file system.xml
01:51:37:WU00:FS00:0x22:Digital signatures verified
01:51:37:WU00:FS00:0x22:Folding@home GPU Core22 Folding@home Core
01:51:37:WU00:FS00:0x22:Version 0.0.5
01:52:04:WU00:FS00:0x22:Completed 0 out of 500000 steps (0%)
01:52:04:WU00:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
01:53:54:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:53:54:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
01:55:30:WU00:FS00:0x22:Completed 5000 out of 500000 steps (1%)
01:57:07:WU00:FS00:0x22:Completed 10000 out of 500000 steps (2%)
01:58:27:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:58:27:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
02:00:04:WU00:FS00:0x22:Completed 5000 out of 500000 steps (1%)
02:01:40:WU00:FS00:0x22:Completed 10000 out of 500000 steps (2%)
02:01:51:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
02:01:51:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
02:01:51:WU00:FS00:0x22:ERROR:114: Max Retries Reached
02:01:51:WU00:FS00:0x22:Saving result file ..\logfile_01.txt
02:01:51:WU00:FS00:0x22:Saving result file badstate-0.xml
02:01:59:WU00:FS00:0x22:Saving result file badstate-1.xml
02:02:08:WU00:FS00:0x22:Saving result file badstate-2.xml
02:02:16:WU00:FS00:0x22:Saving result file checkpt.crc
02:02:16:WU00:FS00:0x22:Saving result file science.log
02:02:16:WU00:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
02:02:17:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
02:02:17:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:14562 run:0 clone:57 gen:43 core:0x22 unit:0x0000003f80fccb025e9615bedeab90eb
02:02:17:WU00:FS00:Uploading 23.98KiB to 128.252.203.2
02:02:17:WU00:FS00:Connecting to 128.252.203.2:8080
02:02:17:WU02:FS00:Connecting to 65.254.110.245:80
02:02:17:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
02:02:17:WU02:FS00:Connecting to 18.218.241.186:80
02:02:17:WU02:FS00:Assigned to work server 206.223.170.146
02:02:17:WU02:FS00:Requesting new work unit for slot 00: READY gpu:1:TU106-200A [GeForce RTX 2060] from 206.223.170.146
02:02:17:WU02:FS00:Connecting to 206.223.170.146:8080
02:02:19:WU00:FS00:Upload complete
02:02:19:WU00:FS00:Server responded WORK_ACK (400)
02:02:19:WU00:FS00:Cleaning up
02:02:28:WU02:FS00:Downloading 41.91MiB
02:02:34:WU02:FS00:Download 67.85%
02:02:40:WU02:FS00:Download 99.91%
02:02:40:WU02:FS00:Download complete
02:02:40:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:14253 run:1753 clone:3 gen:8 core:0x22 unit:0x00000011cedfaa925eaba95ce35b77ca
02:02:40:WU02:FS00:Starting
02:02:40:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\FOLD5\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 8544 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
02:02:40:WU02:FS00:Started FahCore on PID 7548
02:02:40:WU02:FS00:Core PID:9232
02:02:40:WU02:FS00:FahCore 0x22 started
02:02:40:WU02:FS00:0x22:*********************** Log Started 2020-05-15T02:02:40Z ***********************
02:02:40:WU02:FS00:0x22:*************************** Core22 Folding@home Core ***************************
02:02:40:WU02:FS00:0x22:       Type: 0x22
02:02:40:WU02:FS00:0x22:       Core: Core22
02:02:40:WU02:FS00:0x22:    Website: https://foldingathome.org/
02:02:40:WU02:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
02:02:40:WU02:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
02:02:40:WU02:FS00:0x22:             <rafal.wiewiora@choderalab.org>
02:02:40:WU02:FS00:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 7548 -checkpoint 15
02:02:40:WU02:FS00:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
02:02:40:WU02:FS00:0x22:             0 -gpu 0
02:02:40:WU02:FS00:0x22:     Config: <none>
02:02:40:WU02:FS00:0x22:************************************ Build *************************************
02:02:40:WU02:FS00:0x22:    Version: 0.0.5
02:02:40:WU02:FS00:0x22:       Date: Apr 22 2020
02:02:40:WU02:FS00:0x22:       Time: 04:42:59
02:02:40:WU02:FS00:0x22: Repository: Git
02:02:40:WU02:FS00:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
02:02:40:WU02:FS00:0x22:     Branch: HEAD
02:02:40:WU02:FS00:0x22:   Compiler: Visual C++ 2008
02:02:40:WU02:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
02:02:40:WU02:FS00:0x22:   Platform: win32 10
02:02:40:WU02:FS00:0x22:       Bits: 64
02:02:40:WU02:FS00:0x22:       Mode: Release
02:02:40:WU02:FS00:0x22:************************************ System ************************************
02:02:40:WU02:FS00:0x22:        CPU: Intel(R) Core(TM) i5-9600KF CPU @ 3.70GHz
02:02:40:WU02:FS00:0x22:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 13
02:02:40:WU02:FS00:0x22:       CPUs: 6
02:02:40:WU02:FS00:0x22:     Memory: 7.94GiB
02:02:40:WU02:FS00:0x22:Free Memory: 4.98GiB
02:02:40:WU02:FS00:0x22:    Threads: WINDOWS_THREADS
02:02:40:WU02:FS00:0x22: OS Version: 6.2
02:02:40:WU02:FS00:0x22:Has Battery: false
02:02:40:WU02:FS00:0x22: On Battery: false
02:02:40:WU02:FS00:0x22: UTC Offset: -5
02:02:40:WU02:FS00:0x22:        PID: 9232
02:02:40:WU02:FS00:0x22:        CWD: C:\Users\FOLD5\AppData\Roaming\FAHClient\work
02:02:40:WU02:FS00:0x22:         OS: Windows 10 Pro
02:02:40:WU02:FS00:0x22:    OS Arch: AMD64
02:02:40:WU02:FS00:0x22:********************************************************************************
02:02:40:WU02:FS00:0x22:Project: 14253 (Run 1753, Clone 3, Gen 8)
02:02:40:WU02:FS00:0x22:Unit: 0x00000011cedfaa925eaba95ce35b77ca
02:02:40:WU02:FS00:0x22:Reading tar file core.xml
02:02:40:WU02:FS00:0x22:Reading tar file integrator.xml
02:02:40:WU02:FS00:0x22:Reading tar file state.xml
02:02:42:WU02:FS00:0x22:Reading tar file system.xml
02:02:43:WU02:FS00:0x22:Digital signatures verified
02:02:43:WU02:FS00:0x22:Folding@home GPU Core22 Folding@home Core
02:02:43:WU02:FS00:0x22:Version 0.0.5
02:03:13:WU02:FS00:0x22:Completed 0 out of 500000 steps (0%)
02:03:13:WU02:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
02:05:13:WU02:FS00:0x22:Completed 5000 out of 500000 steps (1%)
02:05:17:WU02:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
02:05:17:WU02:FS00:0x22:Following exception occured: Particle coordinate is nan
02:05:59:WU02:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
02:05:59:WU02:FS00:0x22:Following exception occured: Particle coordinate is nan
02:06:23:WU02:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
02:06:23:WU02:FS00:0x22:Following exception occured: Particle coordinate is nan
02:06:23:WU02:FS00:0x22:ERROR:114: Max Retries Reached
02:06:23:WU02:FS00:0x22:Saving result file ..\logfile_01.txt
02:06:23:WU02:FS00:0x22:Saving result file badstate-0.xml
02:06:24:WU02:FS00:0x22:Saving result file badstate-1.xml
02:06:24:WU02:FS00:0x22:Saving result file badstate-2.xml
02:06:24:WU02:FS00:0x22:Saving result file checkpt.crc
02:06:24:WU02:FS00:0x22:Saving result file science.log
02:06:24:WU02:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
02:06:25:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
02:06:25:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:14253 run:1753 clone:3 gen:8 core:0x22 unit:0x00000011cedfaa925eaba95ce35b77ca
02:06:25:WU02:FS00:Uploading 420.80KiB to 206.223.170.146
02:06:25:WU02:FS00:Connecting to 206.223.170.146:8080
02:06:26:WU02:FS00:Upload complete
02:06:26:WU02:FS00:Server responded WORK_ACK (400)
02:06:26:WU02:FS00:Cleaning up
******************************* Date: 2020-05-15 *******************************
******************************* Date: 2020-05-15 *******************************
******************************* Date: 2020-05-15 *******************************
19:04:25:FS00:Paused
19:04:32:FS00:Unpaused
19:04:32:WU00:FS00:Connecting to 65.254.110.245:80
19:04:33:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
19:04:33:WU00:FS00:Connecting to 18.218.241.186:80
19:04:33:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
19:04:33:WU00:FS00:Connecting to 65.254.110.245:80
19:04:34:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
19:04:34:WU00:FS00:Connecting to 18.218.241.186:80
19:04:34:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
19:04:34:ERROR:WU00:FS00:Exception: Could not get an assignment
19:04:34:WU00:FS00:Connecting to 65.254.110.245:80
19:04:35:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
19:04:35:WU00:FS00:Connecting to 18.218.241.186:80
19:04:35:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
19:04:35:WU00:FS00:Connecting to 65.254.110.245:80
19:04:35:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
19:04:35:WU00:FS00:Connecting to 18.218.241.186:80
19:04:35:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
19:04:35:ERROR:WU00:FS00:Exception: Could not get an assignment
Top
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Failed???

Post by Neil-B »

There are a number of threads about projects 13404 and 13405 … These are experimental projects trialling a different approach in support of some urgent work … They have a higher failure rate than traditional WUs - but even the failures are valuable science.
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070
(Green/Bold = Active)
Top
Harpo
Posts: 29
Joined: Thu Mar 26, 2020 9:10 pm

Re: Failed???

Post by Harpo »

I posted as much of the log as this forum would accept, it looks like this project kept that card failing all night.

I stress tested the card just to be sure, on 1080p Furmark it shows 87% load at 97%tdp (165w), temp is stable at 68°C, stock clocks.

So is the problem with the project or my gpu?

Code: Select all

01:11:27:WU00:FS00:0x22:Completed 100000 out of 2000000 steps (5%)
01:12:02:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:12:02:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
01:12:05:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:12:05:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
01:12:08:WU00:FS00:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
01:12:08:WU00:FS00:0x22:Following exception occured: Particle coordinate is nan
01:12:08:WU00:FS00:0x22:ERROR:114: Max Retries Reached
01:12:08:WU00:FS00:0x22:Saving result file ..\logfile_01.txt
01:12:08:WU00:FS00:0x22:Saving result file badstate-0.xml
01:12:09:WU00:FS00:0x22:Saving result file badstate-1.xml
01:12:10:WU00:FS00:0x22:Saving result file badstate-2.xml
01:12:12:WU00:FS00:0x22:Saving result file checkpointState.xml
01:12:13:WU00:FS00:0x22:Saving result file checkpt.crc
01:12:13:WU00:FS00:0x22:Saving result file positions.xtc
01:12:13:WU00:FS00:0x22:Saving result file science.log
01:12:13:WU00:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
01:12:14:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
01:12:14:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:11743 run:0 clone:4227 gen:78 core:0x22
Top
PantherX
Site Moderator
Posts: 7020
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:
Contact PantherX

Re: Failed???

Post by PantherX »

Please note that I checked only 1 WU project:13404 run:695 clone:11 gen:0 and it was successfully folded by another donor: https://apps.foldingathome.org/wu#proje ... e=11&gen=0

Can you please provide more details of the GPU? Is it factory overclocked, what vendor, what model, how old?

FYI, Furmark is a poor stress test for folding as folding is more stressful than running Furmark regardless of what setting you're using on Furmark.
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Top
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Failed???

Post by Neil-B »

Apologies ... I had not spotted that there were other projects as well as 13404 ... and PantherX got his response in before mine below :)

Is you GPU at stock speeds? ... If not dialling it back to stock might be a good first step in the troubleshooting journey ... iirc Furmark isn't a particularly representative test when checking a card for FAH use as it works in a totally different manner ... but I'll let our GPU gurus advise on how best to identify and resolve your issues - hopefully one of them will weigh in shortly.
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070
(Green/Bold = Active)
Top
Harpo
Posts: 29
Joined: Thu Mar 26, 2020 9:10 pm

Re: Failed???

Post by Harpo »

The card is an EVGA GeForce RTX 2060 KO, less than a month old. Log shows it's never been over 71°C in the past few days, clocks are 1905/6800 under load, custom fan curve, NO overclock.
Case is a Thermaltake Commander C34 TG (2x200mm intake fans, 1x120mm exhaust, custom curve).
It is the lower card of two, above it is a GTX1660ti.
Here's a benchmark, run today:

FAHBench (OpenCL, dhfr, 23558 atoms): 109.928 single precision, 10.4712 double precision
Top
PantherX
Site Moderator
Posts: 7020
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:
Contact PantherX

Re: Failed???

Post by PantherX »

Unfortunately, FAHBench hasn't been updated to use the new FahCore_22 feature set :(

I am assuming that you have ruled out RAM, CPU and other system components, right?
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Top
JohnChodera
Pande Group Member
Posts: 470
Joined: Fri Feb 22, 2013 9:59 pm

Re: Failed???

Post by JohnChodera »

> There are a number of threads about projects 13404 and 13405 … These are experimental projects trialling a different approach in support of some urgent work … They have a higher failure rate than traditional WUs - but even the failures are valuable science.

@Neil-B: Thanks for chiming in here! This is exactly right!

We're working on refining our methods to reduce failure rates for future projects in this series, but we're currently still generating a great deal of data that is still super valuable!

> Unfortunately, FAHBench hasn't been updated to use the new FahCore_22 feature set :(

We're working on changing that---stay tuned!

Thanks so much for your patience, everybody!

~ John Chodera // MSKCC
Top
Harpo
Posts: 29
Joined: Thu Mar 26, 2020 9:10 pm

Re: Failed???

Post by Harpo »

PantherX wrote:I am assuming that you have ruled out RAM, CPU and other system components, right?
Yes, it all checks out. There is another card in the rig (1660ti) that is running fine, too. I am wondering about the slot though... I assumed the mb would do two x8 slots, but on checking I find that the 2nd slot can only do PCIe 3.0 x4. The card is showing an estimated PPD of 1013262, so I don't think this is an issue. I did reduce the power to 95%, the 2060 is only drawing 140w now.
Top
Post Reply

Return to “New Donors start here”