I am having trouble getting my two GPUs to both fold.

If you think it might be a driver problem, see viewforum.php?f=79

Moderators: Site Moderators, PandeGroup

Re: I am having trouble getting my two GPUs to both fold.

Postby 7im » Tue Aug 02, 2016 10:13 pm

The client installer should have done that for you automatically. Odd that it didn't do that, and you had to add the GPU slot manually. Good to hear it is folding again.
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
User avatar
7im
 
Posts: 14648
Joined: Thu Nov 29, 2007 4:30 pm
Location: Arizona

Re: I am having trouble getting my two GPUs to both fold.

Postby ChristianVirtual » Tue Aug 02, 2016 10:57 pm

No problem; glad to see its working now.
And get well soon !
ImageImage
Please contribute your logs to http://ppd.fahmm.net
User avatar
ChristianVirtual
 
Posts: 1512
Joined: Tue May 28, 2013 12:14 pm
Location: 日本 東京

Re: I am having trouble getting my two GPUs to both fold.

Postby Bosses_Boss » Wed Aug 03, 2016 6:31 pm

So now I got a different problem, but still related. My GPUs, specifically the 980 Ti aren't folding at my expected 1.2 Million PPD that I saw before sleeping.

Images to help explain my problem - https://imgur.com/a/bjWau

Basically the 980 Ti in my system, seems to have buggered up in folding during the night. But from my monitoring programs I don't see why.

Both GPUs are in PCI-E 8x. Would this be an issue?

Both are powered by a AXi 1500w.

They both work just fine when alone in the system but not together. Tested this just this morning.

I have tried to provoke the 980 Ti to get it's clock speed up using GPU-Z and Cuda-Z but the clocks don't increase but I see the GPU get to 100% load on the core in HWMonitor.

Thank you for looking at my problem once again. I am out right now for about another hour. But wanted to ask y'all for help while I was out. I plan to do a fresh install of NV driver but roll back to a less crashy one see if that helps. I will also switch to have the 980 Ti above the 1070 and see if that makes the 10. And I will try other cables or connections to the PSU. But that is all I got.

I just skimmed over the log and saw errors at the end of it. Is it my CPU OC? Or would those errors be the GPU OCs? huh

Any other ideas?

Thank you in advance!

My log right now in full

Code: Select all
*********************** Log Started 2016-08-03T03:32:36Z ***********************
03:32:36:************************* Folding@home Client *************************
03:32:36:      Website: http://folding.stanford.edu/
03:32:36:    Copyright: (c) 2009-2014 Stanford University
03:32:36:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
03:32:36:         Args: --open-web-control
03:32:36:       Config: C:/Users/Bosses/AppData/Roaming/FAHClient/config.xml
03:32:36:******************************** Build ********************************
03:32:36:      Version: 7.4.4
03:32:36:         Date: Mar 4 2014
03:32:36:         Time: 20:26:54
03:32:36:      SVN Rev: 4130
03:32:36:       Branch: fah/trunk/client
03:32:36:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
03:32:36:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
03:32:36:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
03:32:36:     Platform: win32 XP
03:32:36:         Bits: 32
03:32:36:         Mode: Release
03:32:36:******************************* System ********************************
03:32:36:          CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
03:32:36:       CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
03:32:36:         CPUs: 12
03:32:36:       Memory: 31.90GiB
03:32:36:  Free Memory: 25.84GiB
03:32:36:      Threads: WINDOWS_THREADS
03:32:36:   OS Version: 6.2
03:32:36:  Has Battery: false
03:32:36:   On Battery: false
03:32:36:   UTC Offset: -6
03:32:36:          PID: 5552
03:32:36:          CWD: C:/Users/Bosses/AppData/Roaming/FAHClient
03:32:36:           OS: Windows 10 Home
03:32:36:      OS Arch: AMD64
03:32:36:         GPUs: 3
03:32:36:        GPU 0: UNSUPPORTED: NV3 [PCI]
03:32:36:        GPU 1: NVIDIA:5 GP104 [GeForce GTX 1070]
03:32:36:        GPU 2: NVIDIA:5 GM200 [GeForce GTX 980 Ti]
03:32:36:         CUDA: 6.1
03:32:36:  CUDA Driver: 8000
03:32:36:Win32 Service: false
03:32:36:***********************************************************************
03:32:36:<config>
03:32:36:  <!-- Folding Core -->
03:32:36:  <checkpoint v='5'/>
03:32:36:
03:32:36:  <!-- Network -->
03:32:36:  <proxy v=':8080'/>
03:32:36:
03:32:36:  <!-- Slot Control -->
03:32:36:  <power v='full'/>
03:32:36:
03:32:36:  <!-- User Information -->
03:32:36:  <passkey v='********************************'/>
03:32:36:  <team v='225605'/>
03:32:36:  <user v='Bosses_Boss'/>
03:32:36:
03:32:36:  <!-- Folding Slots -->
03:32:36:  <slot id='0' type='CPU'>
03:32:36:    <cpus v='5'/>
03:32:36:    <pause-on-start v='true'/>
03:32:36:  </slot>
03:32:36:  <slot id='1' type='GPU'/>
03:32:36:  <slot id='2' type='GPU'/>
03:32:36:</config>
03:32:36:Trying to access database...
03:32:36:Successfully acquired database lock
03:32:36:Enabled folding slot 00: PAUSED cpu:5 (by user)
03:32:36:Enabled folding slot 01: READY gpu:1:GP104 [GeForce GTX 1070]
03:32:36:Enabled folding slot 02: READY gpu:2:GM200 [GeForce GTX 980 Ti]
03:32:36:WU01:FS01:Starting
03:32:36:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Bosses/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 704 -lifeline 5552 -checkpoint 5 -gpu 0 -gpu-vendor nvidia
03:32:36:WU01:FS01:Started FahCore on PID 896
03:32:36:WU01:FS01:Core PID:1048
03:32:36:WU01:FS01:FahCore 0x21 started
03:32:36:WU02:FS02:Starting
03:32:36:WU02:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Bosses/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 704 -lifeline 5552 -checkpoint 5 -gpu 1 -gpu-vendor nvidia
03:32:36:WU02:FS02:Started FahCore on PID 12524
03:32:36:WU02:FS02:Core PID:6824
03:32:36:WU02:FS02:FahCore 0x21 started
03:32:37:WU01:FS01:0x21:*********************** Log Started 2016-08-03T03:32:36Z ***********************
03:32:37:WU01:FS01:0x21:Project: 10494 (Run 8, Clone 39, Gen 154)
03:32:37:WU01:FS01:0x21:Unit: 0x000000cc8ca304f555de941d8a2d2d5b
03:32:37:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
03:32:37:WU01:FS01:0x21:Machine: 1
03:32:37:WU01:FS01:0x21:Digital signatures verified
03:32:37:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
03:32:37:WU01:FS01:0x21:Version 0.0.17
03:32:37:WU01:FS01:0x21:  Found a checkpoint file
03:32:37:WU02:FS02:0x21:*********************** Log Started 2016-08-03T03:32:36Z ***********************
03:32:37:WU02:FS02:0x21:Project: 11706 (Run 0, Clone 406, Gen 36)
03:32:37:WU02:FS02:0x21:Unit: 0x0000002b8ca304f356ff2af27981eb70
03:32:37:WU02:FS02:0x21:CPU: 0x00000000000000000000000000000000
03:32:37:WU02:FS02:0x21:Machine: 2
03:32:37:WU02:FS02:0x21:Digital signatures verified
03:32:37:WU02:FS02:0x21:Folding@home GPU Core21 Folding@home Core
03:32:37:WU02:FS02:0x21:Version 0.0.17
03:32:38:11:127.0.0.1:New Web connection
03:32:44:WU02:FS02:0x21:Completed 0 out of 5000000 steps (0%)
03:32:44:WU02:FS02:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
03:33:09:WU01:FS01:0x21:Completed 350000 out of 1000000 steps (35%)
03:33:09:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
03:36:10:WU01:FS01:0x21:Completed 360000 out of 1000000 steps (36%)
03:37:00:Removing old file 'configs/config-20160802-135111.xml'
03:37:00:Saving configuration to config.xml
03:37:00:<config>
03:37:00:  <!-- Folding Core -->
03:37:00:  <checkpoint v='5'/>
03:37:00:
03:37:00:  <!-- Network -->
03:37:00:  <proxy v=':8080'/>
03:37:00:
03:37:00:  <!-- Slot Control -->
03:37:00:  <power v='full'/>
03:37:00:
03:37:00:  <!-- User Information -->
03:37:00:  <passkey v='********************************'/>
03:37:00:  <team v='225605'/>
03:37:00:  <user v='Bosses_Boss'/>
03:37:00:
03:37:00:  <!-- Folding Slots -->
03:37:00:  <slot id='0' type='CPU'>
03:37:00:    <cpus v='5'/>
03:37:00:    <pause-on-start v='true'/>
03:37:00:  </slot>
03:37:00:  <slot id='1' type='GPU'/>
03:37:00:  <slot id='2' type='GPU'/>
03:37:00:</config>
03:37:16:Removing old file 'configs/config-20160802-135138.xml'
03:37:16:Saving configuration to config.xml
03:37:16:<config>
03:37:16:  <!-- Folding Core -->
03:37:16:  <checkpoint v='5'/>
03:37:16:
03:37:16:  <!-- Network -->
03:37:16:  <proxy v=':8080'/>
03:37:16:
03:37:16:  <!-- Slot Control -->
03:37:16:  <power v='full'/>
03:37:16:
03:37:16:  <!-- User Information -->
03:37:16:  <passkey v='********************************'/>
03:37:16:  <team v='225605'/>
03:37:16:  <user v='Bosses_Boss'/>
03:37:16:
03:37:16:  <!-- Folding Slots -->
03:37:16:  <slot id='0' type='CPU'>
03:37:16:    <cpus v='5'/>
03:37:16:    <pause-on-start v='true'/>
03:37:16:  </slot>
03:37:16:  <slot id='1' type='GPU'/>
03:37:16:  <slot id='2' type='GPU'/>
03:37:16:</config>
03:39:10:WU01:FS01:0x21:Completed 370000 out of 1000000 steps (37%)
03:41:49:WU02:FS02:0x21:Completed 50000 out of 5000000 steps (1%)
03:42:15:WU01:FS01:0x21:Completed 380000 out of 1000000 steps (38%)
03:45:20:WU01:FS01:0x21:Completed 390000 out of 1000000 steps (39%)
03:48:24:WU01:FS01:0x21:Completed 400000 out of 1000000 steps (40%)
03:50:54:WU02:FS02:0x21:Completed 100000 out of 5000000 steps (2%)
03:51:39:WU01:FS01:0x21:Completed 410000 out of 1000000 steps (41%)
03:54:43:WU01:FS01:0x21:Completed 420000 out of 1000000 steps (42%)
03:57:48:WU01:FS01:0x21:Completed 430000 out of 1000000 steps (43%)
04:00:00:WU02:FS02:0x21:Completed 150000 out of 5000000 steps (3%)
04:00:53:WU01:FS01:0x21:Completed 440000 out of 1000000 steps (44%)
04:03:57:WU01:FS01:0x21:Completed 450000 out of 1000000 steps (45%)
04:07:12:WU01:FS01:0x21:Completed 460000 out of 1000000 steps (46%)
04:09:05:WU02:FS02:0x21:Completed 200000 out of 5000000 steps (4%)
04:10:17:WU01:FS01:0x21:Completed 470000 out of 1000000 steps (47%)
04:13:21:WU01:FS01:0x21:Completed 480000 out of 1000000 steps (48%)
04:16:26:WU01:FS01:0x21:Completed 490000 out of 1000000 steps (49%)
04:18:10:WU02:FS02:0x21:Completed 250000 out of 5000000 steps (5%)
04:19:30:WU01:FS01:0x21:Completed 500000 out of 1000000 steps (50%)
04:22:46:WU01:FS01:0x21:Completed 510000 out of 1000000 steps (51%)
04:25:50:WU01:FS01:0x21:Completed 520000 out of 1000000 steps (52%)
04:27:18:WU02:FS02:0x21:Completed 300000 out of 5000000 steps (6%)
04:28:55:WU01:FS01:0x21:Completed 530000 out of 1000000 steps (53%)
04:32:00:WU01:FS01:0x21:Completed 540000 out of 1000000 steps (54%)
04:35:05:WU01:FS01:0x21:Completed 550000 out of 1000000 steps (55%)
04:36:23:WU02:FS02:0x21:Completed 350000 out of 5000000 steps (7%)
04:38:20:WU01:FS01:0x21:Completed 560000 out of 1000000 steps (56%)
04:41:25:WU01:FS01:0x21:Completed 570000 out of 1000000 steps (57%)
04:44:30:WU01:FS01:0x21:Completed 580000 out of 1000000 steps (58%)
04:45:29:WU02:FS02:0x21:Completed 400000 out of 5000000 steps (8%)
04:47:35:WU01:FS01:0x21:Completed 590000 out of 1000000 steps (59%)
04:50:40:WU01:FS01:0x21:Completed 600000 out of 1000000 steps (60%)
04:53:55:WU01:FS01:0x21:Completed 610000 out of 1000000 steps (61%)
04:54:34:WU02:FS02:0x21:Completed 450000 out of 5000000 steps (9%)
04:57:00:WU01:FS01:0x21:Completed 620000 out of 1000000 steps (62%)
05:00:06:WU01:FS01:0x21:Completed 630000 out of 1000000 steps (63%)
05:03:10:WU01:FS01:0x21:Completed 640000 out of 1000000 steps (64%)
05:03:39:WU02:FS02:0x21:Completed 500000 out of 5000000 steps (10%)
05:06:16:WU01:FS01:0x21:Completed 650000 out of 1000000 steps (65%)
05:09:31:WU01:FS01:0x21:Completed 660000 out of 1000000 steps (66%)
05:12:36:WU01:FS01:0x21:Completed 670000 out of 1000000 steps (67%)
05:12:47:WU02:FS02:0x21:Completed 550000 out of 5000000 steps (11%)
05:15:40:WU01:FS01:0x21:Completed 680000 out of 1000000 steps (68%)
05:18:45:WU01:FS01:0x21:Completed 690000 out of 1000000 steps (69%)
05:21:50:WU01:FS01:0x21:Completed 700000 out of 1000000 steps (70%)
05:21:52:WU02:FS02:0x21:Completed 600000 out of 5000000 steps (12%)
05:25:05:WU01:FS01:0x21:Completed 710000 out of 1000000 steps (71%)
05:28:10:WU01:FS01:0x21:Completed 720000 out of 1000000 steps (72%)
05:30:58:WU02:FS02:0x21:Completed 650000 out of 5000000 steps (13%)
05:31:15:WU01:FS01:0x21:Completed 730000 out of 1000000 steps (73%)
05:34:19:WU01:FS01:0x21:Completed 740000 out of 1000000 steps (74%)
05:37:24:WU01:FS01:0x21:Completed 750000 out of 1000000 steps (75%)
05:40:03:WU02:FS02:0x21:Completed 700000 out of 5000000 steps (14%)
05:40:39:WU01:FS01:0x21:Completed 760000 out of 1000000 steps (76%)
05:43:44:WU01:FS01:0x21:Completed 770000 out of 1000000 steps (77%)
05:46:49:WU01:FS01:0x21:Completed 780000 out of 1000000 steps (78%)
05:49:08:WU02:FS02:0x21:Completed 750000 out of 5000000 steps (15%)
05:49:54:WU01:FS01:0x21:Completed 790000 out of 1000000 steps (79%)
05:52:58:WU01:FS01:0x21:Completed 800000 out of 1000000 steps (80%)
05:56:13:WU01:FS01:0x21:Completed 810000 out of 1000000 steps (81%)
05:58:16:WU02:FS02:0x21:Completed 800000 out of 5000000 steps (16%)
05:59:18:WU01:FS01:0x21:Completed 820000 out of 1000000 steps (82%)
06:02:23:WU01:FS01:0x21:Completed 830000 out of 1000000 steps (83%)
06:05:27:WU01:FS01:0x21:Completed 840000 out of 1000000 steps (84%)
06:07:21:WU02:FS02:0x21:Completed 850000 out of 5000000 steps (17%)
06:08:32:WU01:FS01:0x21:Completed 850000 out of 1000000 steps (85%)
06:11:47:WU01:FS01:0x21:Completed 860000 out of 1000000 steps (86%)
06:14:52:WU01:FS01:0x21:Completed 870000 out of 1000000 steps (87%)
06:16:27:WU02:FS02:0x21:Completed 900000 out of 5000000 steps (18%)
06:17:56:WU01:FS01:0x21:Completed 880000 out of 1000000 steps (88%)
06:21:01:WU01:FS01:0x21:Completed 890000 out of 1000000 steps (89%)
06:24:05:WU01:FS01:0x21:Completed 900000 out of 1000000 steps (90%)
06:25:32:WU02:FS02:0x21:Completed 950000 out of 5000000 steps (19%)
06:27:20:WU01:FS01:0x21:Completed 910000 out of 1000000 steps (91%)
06:30:25:WU01:FS01:0x21:Completed 920000 out of 1000000 steps (92%)
06:33:30:WU01:FS01:0x21:Completed 930000 out of 1000000 steps (93%)
06:34:37:WU02:FS02:0x21:Completed 1000000 out of 5000000 steps (20%)
06:36:35:WU01:FS01:0x21:Completed 940000 out of 1000000 steps (94%)
06:39:40:WU01:FS01:0x21:Completed 950000 out of 1000000 steps (95%)
06:42:55:WU01:FS01:0x21:Completed 960000 out of 1000000 steps (96%)
06:43:45:WU02:FS02:0x21:Completed 1050000 out of 5000000 steps (21%)
06:45:59:WU01:FS01:0x21:Completed 970000 out of 1000000 steps (97%)
06:49:04:WU01:FS01:0x21:Completed 980000 out of 1000000 steps (98%)
06:52:08:WU01:FS01:0x21:Completed 990000 out of 1000000 steps (99%)
06:52:09:WU00:FS01:Connecting to 171.67.108.45:80
06:52:09:WU00:FS01:Assigned to work server 140.163.4.245
06:52:09:WU00:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:GP104 [GeForce GTX 1070] from 140.163.4.245
06:52:09:WU00:FS01:Connecting to 140.163.4.245:8080
06:52:10:WU00:FS01:Downloading 6.04MiB
06:52:12:WU00:FS01:Download complete
06:52:12:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:10493 run:8 clone:10 gen:114 core:0x21 unit:0x000000aa8ca304f555d616c609ba6a68
06:52:51:WU02:FS02:0x21:Completed 1100000 out of 5000000 steps (22%)
06:55:13:WU01:FS01:0x21:Completed 1000000 out of 1000000 steps (100%)
06:55:24:WU01:FS01:0x21:Saving result file logfile_01.txt
06:55:24:WU01:FS01:0x21:Saving result file checkpointState.xml
06:55:30:WU01:FS01:0x21:Saving result file checkpt.crc
06:55:30:WU01:FS01:0x21:Saving result file log.txt
06:55:30:WU01:FS01:0x21:Saving result file positions.xtc
06:55:33:WU01:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
06:55:33:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
06:55:33:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10494 run:8 clone:39 gen:154 core:0x21 unit:0x000000cc8ca304f555de941d8a2d2d5b
06:55:33:WU01:FS01:Uploading 35.92MiB to 140.163.4.245
06:55:33:WU01:FS01:Connecting to 140.163.4.245:8080
06:55:33:WU00:FS01:Starting
06:55:33:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Bosses/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 00 -suffix 01 -version 704 -lifeline 5552 -checkpoint 5 -gpu 0 -gpu-vendor nvidia
06:55:33:WU00:FS01:Started FahCore on PID 9484
06:55:33:WU00:FS01:Core PID:5644
06:55:33:WU00:FS01:FahCore 0x21 started
06:55:34:WU00:FS01:0x21:*********************** Log Started 2016-08-03T06:55:33Z ***********************
06:55:34:WU00:FS01:0x21:Project: 10493 (Run 8, Clone 10, Gen 114)
06:55:34:WU00:FS01:0x21:Unit: 0x000000aa8ca304f555d616c609ba6a68
06:55:34:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
06:55:34:WU00:FS01:0x21:Machine: 1
06:55:34:WU00:FS01:0x21:Reading tar file core.xml
06:55:34:WU00:FS01:0x21:Reading tar file system.xml
06:55:34:WU00:FS01:0x21:Reading tar file integrator.xml
06:55:34:WU00:FS01:0x21:Reading tar file state.xml
06:55:35:WU00:FS01:0x21:Digital signatures verified
06:55:35:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
06:55:35:WU00:FS01:0x21:Version 0.0.17
06:55:39:WU01:FS01:Upload 9.74%
06:55:44:WU00:FS01:0x21:Completed 0 out of 5000000 steps (0%)
06:55:44:WU00:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
06:55:45:WU01:FS01:Upload 22.62%
06:55:51:WU01:FS01:Upload 35.50%
06:55:57:WU01:FS01:Upload 48.55%
06:56:03:WU01:FS01:Upload 61.43%
06:56:09:WU01:FS01:Upload 74.30%
06:56:15:WU01:FS01:Upload 87.18%
06:56:49:WU01:FS01:Upload complete
06:56:49:WU01:FS01:Server responded WORK_ACK (400)
06:56:49:WU01:FS01:Final credit estimate, 134207.00 points
06:56:49:WU01:FS01:Cleaning up
06:59:02:WU00:FS01:0x21:Completed 50000 out of 5000000 steps (1%)
07:01:56:WU02:FS02:0x21:Completed 1150000 out of 5000000 steps (23%)
07:02:19:WU00:FS01:0x21:Completed 100000 out of 5000000 steps (2%)
07:05:38:WU00:FS01:0x21:Completed 150000 out of 5000000 steps (3%)
07:08:55:WU00:FS01:0x21:Completed 200000 out of 5000000 steps (4%)
07:11:02:WU02:FS02:0x21:Completed 1200000 out of 5000000 steps (24%)
07:12:12:WU00:FS01:0x21:Completed 250000 out of 5000000 steps (5%)
07:15:32:WU00:FS01:0x21:Completed 300000 out of 5000000 steps (6%)
07:18:49:WU00:FS01:0x21:Completed 350000 out of 5000000 steps (7%)
07:20:07:WU02:FS02:0x21:Completed 1250000 out of 5000000 steps (25%)
07:22:09:WU00:FS01:0x21:Completed 400000 out of 5000000 steps (8%)
07:25:26:WU00:FS01:0x21:Completed 450000 out of 5000000 steps (9%)
07:28:43:WU00:FS01:0x21:Completed 500000 out of 5000000 steps (10%)
07:29:15:WU02:FS02:0x21:Completed 1300000 out of 5000000 steps (26%)
07:32:03:WU00:FS01:0x21:Completed 550000 out of 5000000 steps (11%)
07:35:19:WU00:FS01:0x21:Completed 600000 out of 5000000 steps (12%)
07:38:20:WU02:FS02:0x21:Completed 1350000 out of 5000000 steps (27%)
07:38:39:WU00:FS01:0x21:Completed 650000 out of 5000000 steps (13%)
07:41:56:WU00:FS01:0x21:Completed 700000 out of 5000000 steps (14%)
07:45:13:WU00:FS01:0x21:Completed 750000 out of 5000000 steps (15%)
07:47:26:WU02:FS02:0x21:Completed 1400000 out of 5000000 steps (28%)
07:48:33:WU00:FS01:0x21:Completed 800000 out of 5000000 steps (16%)
07:51:50:WU00:FS01:0x21:Completed 850000 out of 5000000 steps (17%)
07:55:10:WU00:FS01:0x21:Completed 900000 out of 5000000 steps (18%)
07:56:32:WU02:FS02:0x21:Completed 1450000 out of 5000000 steps (29%)
07:58:27:WU00:FS01:0x21:Completed 950000 out of 5000000 steps (19%)
08:01:44:WU00:FS01:0x21:Completed 1000000 out of 5000000 steps (20%)
08:05:04:WU00:FS01:0x21:Completed 1050000 out of 5000000 steps (21%)
08:05:37:WU02:FS02:0x21:Completed 1500000 out of 5000000 steps (30%)
08:08:21:WU00:FS01:0x21:Completed 1100000 out of 5000000 steps (22%)
08:11:41:WU00:FS01:0x21:Completed 1150000 out of 5000000 steps (23%)
08:14:45:WU02:FS02:0x21:Completed 1550000 out of 5000000 steps (31%)
08:14:58:WU00:FS01:0x21:Completed 1200000 out of 5000000 steps (24%)
08:18:15:WU00:FS01:0x21:Completed 1250000 out of 5000000 steps (25%)
08:21:35:WU00:FS01:0x21:Completed 1300000 out of 5000000 steps (26%)
08:23:50:WU02:FS02:0x21:Completed 1600000 out of 5000000 steps (32%)
08:24:52:WU00:FS01:0x21:Completed 1350000 out of 5000000 steps (27%)
08:28:12:WU00:FS01:0x21:Completed 1400000 out of 5000000 steps (28%)
08:31:28:WU00:FS01:0x21:Completed 1450000 out of 5000000 steps (29%)
08:32:56:WU02:FS02:0x21:Completed 1650000 out of 5000000 steps (33%)
08:34:45:WU00:FS01:0x21:Completed 1500000 out of 5000000 steps (30%)
08:38:05:WU00:FS01:0x21:Completed 1550000 out of 5000000 steps (31%)
08:41:22:WU00:FS01:0x21:Completed 1600000 out of 5000000 steps (32%)
08:42:01:WU02:FS02:0x21:Completed 1700000 out of 5000000 steps (34%)
08:44:42:WU00:FS01:0x21:Completed 1650000 out of 5000000 steps (33%)
08:47:59:WU00:FS01:0x21:Completed 1700000 out of 5000000 steps (34%)
08:51:07:WU02:FS02:0x21:Completed 1750000 out of 5000000 steps (35%)
08:51:16:WU00:FS01:0x21:Completed 1750000 out of 5000000 steps (35%)
08:54:36:WU00:FS01:0x21:Completed 1800000 out of 5000000 steps (36%)
08:57:53:WU00:FS01:0x21:Completed 1850000 out of 5000000 steps (37%)
09:00:15:WU02:FS02:0x21:Completed 1800000 out of 5000000 steps (36%)
09:01:13:WU00:FS01:0x21:Completed 1900000 out of 5000000 steps (38%)
09:04:30:WU00:FS01:0x21:Completed 1950000 out of 5000000 steps (39%)
09:07:47:WU00:FS01:0x21:Completed 2000000 out of 5000000 steps (40%)
09:09:21:WU02:FS02:0x21:Completed 1850000 out of 5000000 steps (37%)
09:11:07:WU00:FS01:0x21:Completed 2050000 out of 5000000 steps (41%)
09:14:24:WU00:FS01:0x21:Completed 2100000 out of 5000000 steps (42%)
09:17:44:WU00:FS01:0x21:Completed 2150000 out of 5000000 steps (43%)
09:18:26:WU02:FS02:0x21:Completed 1900000 out of 5000000 steps (38%)
09:21:01:WU00:FS01:0x21:Completed 2200000 out of 5000000 steps (44%)
09:24:18:WU00:FS01:0x21:Completed 2250000 out of 5000000 steps (45%)
09:27:32:WU02:FS02:0x21:Completed 1950000 out of 5000000 steps (39%)
09:27:38:WU00:FS01:0x21:Completed 2300000 out of 5000000 steps (46%)
09:30:55:WU00:FS01:0x21:Completed 2350000 out of 5000000 steps (47%)
******************************* Date: 2016-08-03 *******************************
09:34:15:WU00:FS01:0x21:Completed 2400000 out of 5000000 steps (48%)
09:36:37:WU02:FS02:0x21:Completed 2000000 out of 5000000 steps (40%)
09:37:32:WU00:FS01:0x21:Completed 2450000 out of 5000000 steps (49%)
09:40:49:WU00:FS01:0x21:Completed 2500000 out of 5000000 steps (50%)
09:44:09:WU00:FS01:0x21:Completed 2550000 out of 5000000 steps (51%)
09:45:45:WU02:FS02:0x21:Completed 2050000 out of 5000000 steps (41%)
09:47:25:WU00:FS01:0x21:Completed 2600000 out of 5000000 steps (52%)
09:50:45:WU00:FS01:0x21:Completed 2650000 out of 5000000 steps (53%)
09:54:02:WU00:FS01:0x21:Completed 2700000 out of 5000000 steps (54%)
09:54:50:WU02:FS02:0x21:Completed 2100000 out of 5000000 steps (42%)
09:57:19:WU00:FS01:0x21:Completed 2750000 out of 5000000 steps (55%)
10:00:39:WU00:FS01:0x21:Completed 2800000 out of 5000000 steps (56%)
10:03:56:WU00:FS01:0x21:Completed 2850000 out of 5000000 steps (57%)
10:03:56:WU02:FS02:0x21:Completed 2150000 out of 5000000 steps (43%)
10:07:16:WU00:FS01:0x21:Completed 2900000 out of 5000000 steps (58%)
10:10:32:WU00:FS01:0x21:Completed 2950000 out of 5000000 steps (59%)
10:13:02:WU02:FS02:0x21:Completed 2200000 out of 5000000 steps (44%)
10:13:50:WU00:FS01:0x21:Completed 3000000 out of 5000000 steps (60%)
10:17:09:WU00:FS01:0x21:Completed 3050000 out of 5000000 steps (61%)
10:20:26:WU00:FS01:0x21:Completed 3100000 out of 5000000 steps (62%)
10:22:07:WU02:FS02:0x21:Completed 2250000 out of 5000000 steps (45%)
10:23:46:WU00:FS01:0x21:Completed 3150000 out of 5000000 steps (63%)
10:27:04:WU00:FS01:0x21:Completed 3200000 out of 5000000 steps (64%)
10:30:23:WU00:FS01:0x21:Completed 3250000 out of 5000000 steps (65%)
10:31:16:WU02:FS02:0x21:Completed 2300000 out of 5000000 steps (46%)
10:33:42:WU00:FS01:0x21:Completed 3300000 out of 5000000 steps (66%)
10:34:50:Removing old file 'configs/config-20160802-155715.xml'
10:34:50:Saving configuration to config.xml
10:34:50:<config>
10:34:50:  <!-- Folding Core -->
10:34:50:  <checkpoint v='5'/>
10:34:50:  <core-priority v='low'/>
10:34:50:
10:34:50:  <!-- Network -->
10:34:50:  <proxy v=':8080'/>
10:34:50:
10:34:50:  <!-- Slot Control -->
10:34:50:  <power v='full'/>
10:34:50:
10:34:50:  <!-- User Information -->
10:34:50:  <passkey v='********************************'/>
10:34:50:  <team v='225605'/>
10:34:50:  <user v='Bosses_Boss'/>
10:34:50:
10:34:50:  <!-- Folding Slots -->
10:34:50:  <slot id='0' type='CPU'>
10:34:50:    <cpus v='5'/>
10:34:50:    <pause-on-start v='true'/>
10:34:50:  </slot>
10:34:50:  <slot id='1' type='GPU'/>
10:34:50:  <slot id='2' type='GPU'/>
10:34:50:</config>
10:35:32:Removing old file 'configs/config-20160802-155746.xml'
10:35:32:Saving configuration to config.xml
10:35:32:<config>
10:35:32:  <!-- Folding Core -->
10:35:32:  <checkpoint v='5'/>
10:35:32:  <core-priority v='low'/>
10:35:32:
10:35:32:  <!-- Network -->
10:35:32:  <proxy v=':8080'/>
10:35:32:
10:35:32:  <!-- Slot Control -->
10:35:32:  <power v='full'/>
10:35:32:
10:35:32:  <!-- User Information -->
10:35:32:  <passkey v='********************************'/>
10:35:32:  <team v='225605'/>
10:35:32:  <user v='Bosses_Boss'/>
10:35:32:
10:35:32:  <!-- Folding Slots -->
10:35:32:  <slot id='0' type='CPU'>
10:35:32:    <cpus v='5'/>
10:35:32:    <pause-on-start v='true'/>
10:35:32:  </slot>
10:35:32:  <slot id='1' type='GPU'/>
10:35:32:  <slot id='2' type='GPU'/>
10:35:32:</config>
10:37:01:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
10:40:28:WU02:FS02:0x21:Completed 2350000 out of 5000000 steps (47%)
10:43:41:WU00:FS01:0x21:Completed 3350000 out of 5000000 steps (67%)
10:47:05:WU00:FS01:0x21:Completed 3400000 out of 5000000 steps (68%)
10:49:41:WU02:FS02:0x21:Completed 2400000 out of 5000000 steps (48%)
10:50:27:WU00:FS01:0x21:Completed 3450000 out of 5000000 steps (69%)
10:53:45:WU00:FS01:0x21:Completed 3500000 out of 5000000 steps (70%)
10:57:09:WU00:FS01:0x21:Completed 3550000 out of 5000000 steps (71%)
10:58:57:WU02:FS02:0x21:Completed 2450000 out of 5000000 steps (49%)
11:00:26:WU00:FS01:0x21:Completed 3600000 out of 5000000 steps (72%)
11:03:45:WU00:FS01:0x21:Completed 3650000 out of 5000000 steps (73%)
11:07:02:WU00:FS01:0x21:Completed 3700000 out of 5000000 steps (74%)
11:08:09:WU02:FS02:0x21:Completed 2500000 out of 5000000 steps (50%)
11:10:21:WU00:FS01:0x21:Completed 3750000 out of 5000000 steps (75%)
11:11:13:FS01:Shutting core down
11:11:13:FS02:Shutting core down
11:11:13:WU00:FS01:0x21:WARNING:Console control signal 1 on PID 5644
11:11:13:WU00:FS01:0x21:Exiting, please wait. . .
11:11:13:WU02:FS02:0x21:WARNING:Console control signal 1 on PID 6824
11:11:13:WU02:FS02:0x21:Exiting, please wait. . .
11:11:13:WU02:FS02:0x21:Folding@home Core Shutdown: INTERRUPTED
11:11:14:WU00:FS01:0x21:Folding@home Core Shutdown: INTERRUPTED
11:11:14:WU02:FS02:FahCore returned: INTERRUPTED (102 = 0x66)
11:11:14:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
11:12:08:Removing old file 'configs/config-20160802-162412.xml'
11:12:08:Saving configuration to config.xml
11:12:08:<config>
11:12:08:  <!-- Folding Core -->
11:12:08:  <checkpoint v='5'/>
11:12:08:  <core-priority v='low'/>
11:12:08:
11:12:08:  <!-- Network -->
11:12:08:  <proxy v=':8080'/>
11:12:08:
11:12:08:  <!-- Slot Control -->
11:12:08:  <power v='light'/>
11:12:08:
11:12:08:  <!-- User Information -->
11:12:08:  <passkey v='********************************'/>
11:12:08:  <team v='225605'/>
11:12:08:  <user v='Bosses_Boss'/>
11:12:08:
11:12:08:  <!-- Folding Slots -->
11:12:08:  <slot id='0' type='CPU'>
11:12:08:    <cpus v='5'/>
11:12:08:    <pause-on-start v='true'/>
11:12:08:  </slot>
11:12:08:  <slot id='1' type='GPU'/>
11:12:08:  <slot id='2' type='GPU'/>
11:12:08:</config>
11:23:12:FS00:Unpaused
11:23:12:WU01:FS00:Connecting to 171.67.108.45:8080
11:23:13:WARNING:WU01:FS00:Failed to get assignment from '171.67.108.45:8080': Empty work server assignment
11:23:13:WU01:FS00:Connecting to 171.64.65.35:80
11:23:14:WARNING:WU01:FS00:Failed to get assignment from '171.64.65.35:80': Empty work server assignment
11:23:14:ERROR:WU01:FS00:Exception: Could not get an assignment
11:23:14:WU01:FS00:Connecting to 171.67.108.45:8080
11:23:14:WARNING:WU01:FS00:Failed to get assignment from '171.67.108.45:8080': Empty work server assignment
11:23:14:WU01:FS00:Connecting to 171.64.65.35:80
11:23:15:WARNING:WU01:FS00:Failed to get assignment from '171.64.65.35:80': Empty work server assignment
11:23:15:ERROR:WU01:FS00:Exception: Could not get an assignment
11:23:16:FS00:Paused
11:23:19:Removing old file 'configs/config-20160802-163753.xml'
11:23:19:Saving configuration to config.xml
11:23:19:<config>
11:23:19:  <!-- Folding Core -->
11:23:19:  <checkpoint v='5'/>
11:23:19:  <core-priority v='low'/>
11:23:19:
11:23:19:  <!-- Network -->
11:23:19:  <proxy v=':8080'/>
11:23:19:
11:23:19:  <!-- Slot Control -->
11:23:19:  <power v='light'/>
11:23:19:
11:23:19:  <!-- User Information -->
11:23:19:  <passkey v='********************************'/>
11:23:19:  <team v='225605'/>
11:23:19:  <user v='Bosses_Boss'/>
11:23:19:
11:23:19:  <!-- Folding Slots -->
11:23:19:  <slot id='0' type='CPU'>
11:23:19:    <cpus v='5'/>
11:23:19:    <pause-on-start v='true'/>
11:23:19:    <paused v='true'/>
11:23:19:  </slot>
11:23:19:  <slot id='1' type='GPU'/>
11:23:19:  <slot id='2' type='GPU'/>
11:23:19:</config>
11:23:29:WU02:FS02:Starting
11:23:29:WU02:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Bosses/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 704 -lifeline 5552 -checkpoint 5 -gpu 1 -gpu-vendor nvidia
11:23:29:WU02:FS02:Started FahCore on PID 14624
11:23:29:WU02:FS02:Core PID:8092
11:23:29:WU02:FS02:FahCore 0x21 started
11:23:29:WU00:FS01:Starting
11:23:29:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Bosses/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 00 -suffix 01 -version 704 -lifeline 5552 -checkpoint 5 -gpu 0 -gpu-vendor nvidia
11:23:29:WU00:FS01:Started FahCore on PID 8084
11:23:29:WU00:FS01:Core PID:12512
11:23:29:WU00:FS01:FahCore 0x21 started
11:23:30:WU02:FS02:0x21:*********************** Log Started 2016-08-03T11:23:29Z ***********************
11:23:30:WU02:FS02:0x21:Project: 11706 (Run 0, Clone 406, Gen 36)
11:23:30:WU02:FS02:0x21:Unit: 0x0000002b8ca304f356ff2af27981eb70
11:23:30:WU02:FS02:0x21:CPU: 0x00000000000000000000000000000000
11:23:30:WU02:FS02:0x21:Machine: 2
11:23:30:WU02:FS02:0x21:Digital signatures verified
11:23:30:WU02:FS02:0x21:Folding@home GPU Core21 Folding@home Core
11:23:30:WU02:FS02:0x21:Version 0.0.17
11:23:30:WU02:FS02:0x21:  Found a checkpoint file
11:23:30:WU00:FS01:0x21:*********************** Log Started 2016-08-03T11:23:29Z ***********************
11:23:30:WU00:FS01:0x21:Project: 10493 (Run 8, Clone 10, Gen 114)
11:23:30:WU00:FS01:0x21:Unit: 0x000000aa8ca304f555d616c609ba6a68
11:23:30:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:23:30:WU00:FS01:0x21:Machine: 1
11:23:30:WU00:FS01:0x21:Digital signatures verified
11:23:30:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:23:30:WU00:FS01:0x21:Version 0.0.17
11:23:30:WU00:FS01:0x21:  Found a checkpoint file
11:23:37:WU02:FS02:0x21:Completed 2500000 out of 5000000 steps (50%)
11:23:37:WU02:FS02:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
11:23:39:WU00:FS01:0x21:Completed 3750000 out of 5000000 steps (75%)
11:23:39:WU00:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
11:24:20:Removing old file 'configs/config-20160802-172027.xml'
11:24:20:Saving configuration to config.xml
11:24:20:<config>
11:24:20:  <!-- Folding Core -->
11:24:20:  <checkpoint v='5'/>
11:24:20:  <core-priority v='low'/>
11:24:20:
11:24:20:  <!-- Network -->
11:24:20:  <proxy v=':8080'/>
11:24:20:
11:24:20:  <!-- Slot Control -->
11:24:20:  <power v='full'/>
11:24:20:
11:24:20:  <!-- User Information -->
11:24:20:  <passkey v='********************************'/>
11:24:20:  <team v='225605'/>
11:24:20:  <user v='Bosses_Boss'/>
11:24:20:
11:24:20:  <!-- Folding Slots -->
11:24:20:  <slot id='0' type='CPU'>
11:24:20:    <cpus v='5'/>
11:24:20:    <pause-on-start v='true'/>
11:24:20:    <paused v='true'/>
11:24:20:  </slot>
11:24:20:  <slot id='1' type='GPU'/>
11:24:20:  <slot id='2' type='GPU'/>
11:24:20:</config>
11:26:56:WU00:FS01:0x21:Completed 3800000 out of 5000000 steps (76%)
11:30:14:WU00:FS01:0x21:Completed 3850000 out of 5000000 steps (77%)
11:32:51:WU02:FS02:0x21:Completed 2550000 out of 5000000 steps (51%)
11:33:35:WU00:FS01:0x21:Completed 3900000 out of 5000000 steps (78%)
11:36:52:WU00:FS01:0x21:Completed 3950000 out of 5000000 steps (79%)
11:40:11:WU00:FS01:0x21:Completed 4000000 out of 5000000 steps (80%)
11:42:06:WU02:FS02:0x21:Completed 2600000 out of 5000000 steps (52%)
11:43:32:WU00:FS01:0x21:Completed 4050000 out of 5000000 steps (81%)
11:46:50:WU00:FS01:0x21:Completed 4100000 out of 5000000 steps (82%)
11:50:12:WU00:FS01:0x21:Completed 4150000 out of 5000000 steps (83%)
11:51:21:WU02:FS02:0x21:Completed 2650000 out of 5000000 steps (53%)
11:53:31:WU00:FS01:0x21:Completed 4200000 out of 5000000 steps (84%)
11:53:48:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
11:55:27:WU00:FS01:0x21:Completed 4150000 out of 5000000 steps (83%)
11:58:45:WU00:FS01:0x21:Completed 4200000 out of 5000000 steps (84%)
12:00:35:WU02:FS02:0x21:Completed 2700000 out of 5000000 steps (54%)
12:02:02:WU00:FS01:0x21:Completed 4250000 out of 5000000 steps (85%)
12:05:24:WU00:FS01:0x21:Completed 4300000 out of 5000000 steps (86%)
12:08:44:WU00:FS01:0x21:Completed 4350000 out of 5000000 steps (87%)
12:09:50:WU02:FS02:0x21:Completed 2750000 out of 5000000 steps (55%)
12:12:07:WU00:FS01:0x21:Completed 4400000 out of 5000000 steps (88%)
12:15:26:WU00:FS01:0x21:Completed 4450000 out of 5000000 steps (89%)
12:18:46:WU00:FS01:0x21:Completed 4500000 out of 5000000 steps (90%)
12:19:07:WU02:FS02:0x21:Completed 2800000 out of 5000000 steps (56%)
12:22:09:WU00:FS01:0x21:Completed 4550000 out of 5000000 steps (91%)
12:25:29:WU00:FS01:0x21:Completed 4600000 out of 5000000 steps (92%)
12:28:21:WU02:FS02:0x21:Completed 2850000 out of 5000000 steps (57%)
12:28:53:WU00:FS01:0x21:Completed 4650000 out of 5000000 steps (93%)
12:32:17:WU00:FS01:0x21:Completed 4700000 out of 5000000 steps (94%)
12:35:38:WU00:FS01:0x21:Completed 4750000 out of 5000000 steps (95%)
12:37:49:WU02:FS02:0x21:Completed 2900000 out of 5000000 steps (58%)
12:39:02:WU00:FS01:0x21:Completed 4800000 out of 5000000 steps (96%)
12:42:32:WU00:FS01:0x21:Completed 4850000 out of 5000000 steps (97%)
12:46:11:WU00:FS01:0x21:Completed 4900000 out of 5000000 steps (98%)
12:49:08:WU02:FS02:0x21:Completed 2950000 out of 5000000 steps (59%)
12:49:43:WU00:FS01:0x21:Completed 4950000 out of 5000000 steps (99%)
12:49:43:WU03:FS01:Connecting to 171.67.108.45:80
12:49:44:WU03:FS01:Assigned to work server 171.67.108.155
12:49:44:WU03:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:GP104 [GeForce GTX 1070] from 171.67.108.155
12:49:44:WU03:FS01:Connecting to 171.67.108.155:8080
12:49:45:WU03:FS01:Downloading 908.85KiB
12:49:46:WU03:FS01:Download complete
12:49:46:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:9660 run:1 clone:17 gen:160 core:0x18 unit:0x000000b5ab436c9b56de69baf9debdb8
12:53:03:WU00:FS01:0x21:Completed 5000000 out of 5000000 steps (100%)
12:53:06:WU00:FS01:0x21:Saving result file logfile_01.txt
12:53:06:WU00:FS01:0x21:Saving result file badstate-0.xml
12:53:08:WU00:FS01:0x21:Saving result file checkpointState.xml
12:53:10:WU00:FS01:0x21:Saving result file checkpt.crc
12:53:10:WU00:FS01:0x21:Saving result file log.txt
12:53:10:WU00:FS01:0x21:Saving result file positions.xtc
12:53:12:WU00:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
12:53:12:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
12:53:12:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:10493 run:8 clone:10 gen:114 core:0x21 unit:0x000000aa8ca304f555d616c609ba6a68
12:53:12:WU00:FS01:Uploading 16.06MiB to 140.163.4.245
12:53:12:WU03:FS01:Starting
12:53:12:WU00:FS01:Connecting to 140.163.4.245:8080
12:53:12:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Bosses/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 03 -suffix 01 -version 704 -lifeline 5552 -checkpoint 5 -gpu 0 -gpu-vendor nvidia
12:53:12:WU03:FS01:Started FahCore on PID 8500
12:53:12:WU03:FS01:Core PID:6804
12:53:12:WU03:FS01:FahCore 0x18 started
12:53:13:WU03:FS01:0x18:*********************** Log Started 2016-08-03T12:53:13Z ***********************
12:53:13:WU03:FS01:0x18:Project: 9660 (Run 1, Clone 17, Gen 160)
12:53:13:WU03:FS01:0x18:Unit: 0x000000b5ab436c9b56de69baf9debdb8
12:53:13:WU03:FS01:0x18:CPU: 0x00000000000000000000000000000000
12:53:13:WU03:FS01:0x18:Machine: 1
12:53:13:WU03:FS01:0x18:Reading tar file core.xml
12:53:13:WU03:FS01:0x18:Reading tar file integrator.xml
12:53:13:WU03:FS01:0x18:Reading tar file state.xml
12:53:13:WU03:FS01:0x18:Reading tar file system.xml
12:53:13:WU03:FS01:0x18:Digital signatures verified
12:53:13:WU03:FS01:0x18:Folding@home GPU core18
12:53:13:WU03:FS01:0x18:Version 0.0.4
12:53:18:WU00:FS01:Upload 21.40%
12:53:22:WU03:FS01:0x18:Completed 0 out of 2000000 steps (0%)
12:53:22:WU03:FS01:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
12:53:24:WU00:FS01:Upload 44.35%
12:53:30:WU00:FS01:Upload 68.86%
12:53:36:WU00:FS01:Upload 97.27%
12:53:44:WU03:FS01:0x18:Completed 20000 out of 2000000 steps (1%)
12:53:46:WU00:FS01:Upload complete
12:53:46:WU00:FS01:Server responded WORK_ACK (400)
12:53:46:WU00:FS01:Final credit estimate, 149672.00 points
12:53:46:WU00:FS01:Cleaning up
12:54:03:WU03:FS01:0x18:Completed 40000 out of 2000000 steps (2%)
12:54:22:WU03:FS01:0x18:Completed 60000 out of 2000000 steps (3%)
12:54:42:WU03:FS01:0x18:Completed 80000 out of 2000000 steps (4%)
12:55:01:WU03:FS01:0x18:Completed 100000 out of 2000000 steps (5%)
12:55:23:WU03:FS01:0x18:Completed 120000 out of 2000000 steps (6%)
12:55:43:WU03:FS01:0x18:Completed 140000 out of 2000000 steps (7%)
12:56:02:WU03:FS01:0x18:Completed 160000 out of 2000000 steps (8%)
12:56:21:WU03:FS01:0x18:Completed 180000 out of 2000000 steps (9%)
12:56:40:WU03:FS01:0x18:Completed 200000 out of 2000000 steps (10%)
12:57:03:WU03:FS01:0x18:Completed 220000 out of 2000000 steps (11%)
12:57:22:WU03:FS01:0x18:Completed 240000 out of 2000000 steps (12%)
12:57:42:WU03:FS01:0x18:Completed 260000 out of 2000000 steps (13%)
12:58:01:WU03:FS01:0x18:Completed 280000 out of 2000000 steps (14%)
12:58:21:WU03:FS01:0x18:Completed 300000 out of 2000000 steps (15%)
12:58:28:WU02:FS02:0x21:Completed 3000000 out of 5000000 steps (60%)
12:58:43:WU03:FS01:0x18:Completed 320000 out of 2000000 steps (16%)
12:59:02:WU03:FS01:0x18:Completed 340000 out of 2000000 steps (17%)
12:59:22:WU03:FS01:0x18:Completed 360000 out of 2000000 steps (18%)
12:59:41:WU03:FS01:0x18:Completed 380000 out of 2000000 steps (19%)
13:00:00:WU03:FS01:0x18:Completed 400000 out of 2000000 steps (20%)
13:00:23:WU03:FS01:0x18:Completed 420000 out of 2000000 steps (21%)
13:00:42:WU03:FS01:0x18:Completed 440000 out of 2000000 steps (22%)
13:01:01:WU03:FS01:0x18:Completed 460000 out of 2000000 steps (23%)
13:01:21:WU03:FS01:0x18:Completed 480000 out of 2000000 steps (24%)
13:01:40:WU03:FS01:0x18:Completed 500000 out of 2000000 steps (25%)
13:02:02:WU03:FS01:0x18:Completed 520000 out of 2000000 steps (26%)
13:02:21:WU03:FS01:0x18:Completed 540000 out of 2000000 steps (27%)
13:02:41:WU03:FS01:0x18:Completed 560000 out of 2000000 steps (28%)
13:03:00:WU03:FS01:0x18:Completed 580000 out of 2000000 steps (29%)
13:03:19:WU03:FS01:0x18:Completed 600000 out of 2000000 steps (30%)
13:03:42:WU03:FS01:0x18:Completed 620000 out of 2000000 steps (31%)
13:04:01:WU03:FS01:0x18:Completed 640000 out of 2000000 steps (32%)
13:04:20:WU03:FS01:0x18:Completed 660000 out of 2000000 steps (33%)
13:04:39:WU03:FS01:0x18:Completed 680000 out of 2000000 steps (34%)
13:04:59:WU03:FS01:0x18:Completed 700000 out of 2000000 steps (35%)
13:05:21:WU03:FS01:0x18:Completed 720000 out of 2000000 steps (36%)
13:05:40:WU03:FS01:0x18:Completed 740000 out of 2000000 steps (37%)
13:06:00:WU03:FS01:0x18:Completed 760000 out of 2000000 steps (38%)
13:06:19:WU03:FS01:0x18:Completed 780000 out of 2000000 steps (39%)
13:06:37:WU03:FS01:0x18:Completed 800000 out of 2000000 steps (40%)
13:07:00:WU03:FS01:0x18:Completed 820000 out of 2000000 steps (41%)
13:07:18:WU03:FS01:0x18:Completed 840000 out of 2000000 steps (42%)
13:07:37:WU03:FS01:0x18:Completed 860000 out of 2000000 steps (43%)
13:07:43:WU02:FS02:0x21:Completed 3050000 out of 5000000 steps (61%)
13:07:56:WU03:FS01:0x18:Completed 880000 out of 2000000 steps (44%)
13:08:15:WU03:FS01:0x18:Completed 900000 out of 2000000 steps (45%)
13:08:37:WU03:FS01:0x18:Completed 920000 out of 2000000 steps (46%)
13:08:56:WU03:FS01:0x18:Completed 940000 out of 2000000 steps (47%)
13:09:15:WU03:FS01:0x18:Completed 960000 out of 2000000 steps (48%)
13:09:34:WU03:FS01:0x18:Completed 980000 out of 2000000 steps (49%)
13:09:53:WU03:FS01:0x18:Completed 1000000 out of 2000000 steps (50%)
13:10:15:WU03:FS01:0x18:Completed 1020000 out of 2000000 steps (51%)
13:10:34:WU03:FS01:0x18:Completed 1040000 out of 2000000 steps (52%)
13:10:53:WU03:FS01:0x18:Completed 1060000 out of 2000000 steps (53%)
13:11:12:WU03:FS01:0x18:Completed 1080000 out of 2000000 steps (54%)
13:11:31:WU03:FS01:0x18:Completed 1100000 out of 2000000 steps (55%)
13:11:53:WU03:FS01:0x18:Completed 1120000 out of 2000000 steps (56%)
13:12:12:WU03:FS01:0x18:Completed 1140000 out of 2000000 steps (57%)
13:12:31:WU03:FS01:0x18:Completed 1160000 out of 2000000 steps (58%)
13:12:50:WU03:FS01:0x18:Completed 1180000 out of 2000000 steps (59%)
13:13:09:WU03:FS01:0x18:Completed 1200000 out of 2000000 steps (60%)
13:13:31:WU03:FS01:0x18:Completed 1220000 out of 2000000 steps (61%)
13:13:50:WU03:FS01:0x18:Completed 1240000 out of 2000000 steps (62%)
13:14:09:WU03:FS01:0x18:Completed 1260000 out of 2000000 steps (63%)
13:14:28:WU03:FS01:0x18:Completed 1280000 out of 2000000 steps (64%)
13:14:47:WU03:FS01:0x18:Completed 1300000 out of 2000000 steps (65%)
13:15:08:WU03:FS01:0x18:Completed 1320000 out of 2000000 steps (66%)
13:15:27:WU03:FS01:0x18:Completed 1340000 out of 2000000 steps (67%)
13:15:46:WU03:FS01:0x18:Completed 1360000 out of 2000000 steps (68%)
13:16:05:WU03:FS01:0x18:Completed 1380000 out of 2000000 steps (69%)
13:16:24:WU03:FS01:0x18:Completed 1400000 out of 2000000 steps (70%)
13:16:46:WU03:FS01:0x18:Completed 1420000 out of 2000000 steps (71%)
13:16:55:WU02:FS02:0x21:Completed 3100000 out of 5000000 steps (62%)
13:17:05:WU03:FS01:0x18:Completed 1440000 out of 2000000 steps (72%)
13:17:24:WU03:FS01:0x18:Completed 1460000 out of 2000000 steps (73%)
13:17:43:WU03:FS01:0x18:Completed 1480000 out of 2000000 steps (74%)
13:18:02:WU03:FS01:0x18:Completed 1500000 out of 2000000 steps (75%)
13:18:24:WU03:FS01:0x18:Completed 1520000 out of 2000000 steps (76%)
13:18:43:WU03:FS01:0x18:Completed 1540000 out of 2000000 steps (77%)
13:19:02:WU03:FS01:0x18:Completed 1560000 out of 2000000 steps (78%)
13:19:21:WU03:FS01:0x18:Completed 1580000 out of 2000000 steps (79%)
13:19:40:WU03:FS01:0x18:Completed 1600000 out of 2000000 steps (80%)
13:20:02:WU03:FS01:0x18:Completed 1620000 out of 2000000 steps (81%)
13:20:38:WU03:FS01:0x18:Completed 1640000 out of 2000000 steps (82%)
13:22:23:WU03:FS01:0x18:Completed 1660000 out of 2000000 steps (83%)
13:24:05:WU03:FS01:0x18:Completed 1680000 out of 2000000 steps (84%)
13:25:48:WU03:FS01:0x18:Completed 1700000 out of 2000000 steps (85%)
13:25:49:WU03:FS01:0x18:Bad State detected... attempting to resume from last good checkpoint
13:26:08:WU03:FS01:0x18:Completed 1620000 out of 2000000 steps (81%)
13:26:20:WU02:FS02:0x21:Completed 3150000 out of 5000000 steps (63%)
13:26:27:WU03:FS01:0x18:Completed 1640000 out of 2000000 steps (82%)
13:26:47:WU03:FS01:0x18:Completed 1660000 out of 2000000 steps (83%)
13:27:06:WU03:FS01:0x18:Completed 1680000 out of 2000000 steps (84%)
13:27:25:WU03:FS01:0x18:Completed 1700000 out of 2000000 steps (85%)
13:27:47:WU03:FS01:0x18:Completed 1720000 out of 2000000 steps (86%)
13:28:07:WU03:FS01:0x18:Completed 1740000 out of 2000000 steps (87%)
13:28:26:WU03:FS01:0x18:Completed 1760000 out of 2000000 steps (88%)
13:28:45:WU03:FS01:0x18:Completed 1780000 out of 2000000 steps (89%)
13:29:04:WU03:FS01:0x18:Completed 1800000 out of 2000000 steps (90%)
13:29:26:WU03:FS01:0x18:Completed 1820000 out of 2000000 steps (91%)
13:29:45:WU03:FS01:0x18:Completed 1840000 out of 2000000 steps (92%)
13:30:04:WU03:FS01:0x18:Completed 1860000 out of 2000000 steps (93%)
13:30:23:WU03:FS01:0x18:Completed 1880000 out of 2000000 steps (94%)
13:30:42:WU03:FS01:0x18:Completed 1900000 out of 2000000 steps (95%)
13:31:04:WU03:FS01:0x18:Completed 1920000 out of 2000000 steps (96%)
13:31:23:WU03:FS01:0x18:Completed 1940000 out of 2000000 steps (97%)
13:31:42:WU03:FS01:0x18:Completed 1960000 out of 2000000 steps (98%)
13:32:02:WU03:FS01:0x18:Completed 1980000 out of 2000000 steps (99%)
13:32:21:WU03:FS01:0x18:Completed 2000000 out of 2000000 steps (100%)
13:32:21:WU00:FS01:Connecting to 171.67.108.45:80
13:32:21:WU00:FS01:Assigned to work server 171.64.65.84
13:32:21:WU00:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:GP104 [GeForce GTX 1070] from 171.64.65.84
13:32:21:WU00:FS01:Connecting to 171.64.65.84:8080
13:32:22:WU00:FS01:Downloading 2.88MiB
13:32:24:WU03:FS01:0x18:Saving result file logfile_01.txt
13:32:24:WU03:FS01:0x18:Saving result file checkpointState.xml
13:32:24:WU03:FS01:0x18:Saving result file checkpt.crc
13:32:24:WU03:FS01:0x18:Saving result file log.txt
13:32:24:WU03:FS01:0x18:Saving result file positions.xtc
13:32:24:WU03:FS01:0x18:Folding@home Core Shutdown: FINISHED_UNIT
13:32:24:WU00:FS01:Download complete
13:32:25:WU03:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
13:32:25:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:9660 run:1 clone:17 gen:160 core:0x18 unit:0x000000b5ab436c9b56de69baf9debdb8
13:32:25:WU03:FS01:Uploading 1.74MiB to 171.67.108.155
13:32:25:WU03:FS01:Connecting to 171.67.108.155:8080
13:32:25:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9158 run:96 clone:0 gen:442 core:0x18 unit:0x000001ebab4041545673c9cb5d9bb0c6
13:32:25:WU00:FS01:Starting
13:32:25:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Bosses/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 00 -suffix 01 -version 704 -lifeline 5552 -checkpoint 5 -gpu 0 -gpu-vendor nvidia
13:32:25:WU00:FS01:Started FahCore on PID 11940
13:32:25:WU00:FS01:Core PID:9748
13:32:25:WU00:FS01:FahCore 0x18 started
13:32:25:WU00:FS01:0x18:*********************** Log Started 2016-08-03T13:32:25Z ***********************
13:32:25:WU00:FS01:0x18:Project: 9158 (Run 96, Clone 0, Gen 442)
13:32:25:WU00:FS01:0x18:Unit: 0x000001ebab4041545673c9cb5d9bb0c6
13:32:25:WU00:FS01:0x18:CPU: 0x00000000000000000000000000000000
13:32:25:WU00:FS01:0x18:Machine: 1
13:32:25:WU00:FS01:0x18:Reading tar file core.xml
13:32:25:WU00:FS01:0x18:Reading tar file system.xml
13:32:25:WU00:FS01:0x18:Reading tar file integrator.xml
13:32:25:WU00:FS01:0x18:Reading tar file state.xml
13:32:26:WU00:FS01:0x18:Digital signatures verified
13:32:26:WU00:FS01:0x18:Folding@home GPU core18
13:32:26:WU00:FS01:0x18:Version 0.0.4
13:32:28:WU03:FS01:Upload complete
13:32:28:WU03:FS01:Server responded WORK_ACK (400)
13:32:28:WU03:FS01:Final credit estimate, 11184.00 points
13:32:28:WU03:FS01:Cleaning up
13:32:44:WU00:FS01:0x18:Completed 0 out of 2500000 steps (0%)
13:32:44:WU00:FS01:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
13:33:49:WU00:FS01:0x18:Completed 25000 out of 2500000 steps (1%)
13:34:48:WU00:FS01:0x18:Completed 50000 out of 2500000 steps (2%)
13:35:34:WU02:FS02:0x21:Completed 3200000 out of 5000000 steps (64%)
13:35:47:WU00:FS01:0x18:Completed 75000 out of 2500000 steps (3%)
13:36:46:WU00:FS01:0x18:Completed 100000 out of 2500000 steps (4%)
13:37:53:WU00:FS01:0x18:Completed 125000 out of 2500000 steps (5%)
13:38:52:WU00:FS01:0x18:Completed 150000 out of 2500000 steps (6%)
snipped out.
15:10:51:WU00:FS01:0x18:Completed 2400000 out of 2500000 steps (96%)
15:12:01:WU00:FS01:0x18:Completed 2425000 out of 2500000 steps (97%)
15:12:23:WU02:FS02:0x21:Completed 3700000 out of 5000000 steps (74%)
15:13:03:WU00:FS01:0x18:Completed 2450000 out of 2500000 steps (98%)
15:14:05:WU00:FS01:0x18:Completed 2475000 out of 2500000 steps (99%)
15:15:07:WU00:FS01:0x18:Completed 2500000 out of 2500000 steps (100%)
15:15:08:WU03:FS01:Connecting to 171.67.108.45:80
15:15:09:WU03:FS01:Assigned to work server 140.163.4.245
15:15:09:WU03:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:GP104 [GeForce GTX 1070] from 140.163.4.245
15:15:09:WU03:FS01:Connecting to 140.163.4.245:8080
15:15:09:WU03:FS01:Downloading 14.49MiB
15:15:13:WU03:FS01:Download complete
15:15:13:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:10496 run:41 clone:7 gen:7 core:0x21 unit:0x000000098ca304f556bba91320b495b9
15:15:14:WU00:FS01:0x18:Saving result file logfile_01.txt
15:15:14:WU00:FS01:0x18:Saving result file checkpointState.xml
15:15:15:WU00:FS01:0x18:Saving result file checkpt.crc
15:15:15:WU00:FS01:0x18:Saving result file log.txt
15:15:15:WU00:FS01:0x18:Saving result file positions.xtc
15:15:16:WU00:FS01:0x18:Folding@home Core Shutdown: FINISHED_UNIT
15:15:16:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
15:15:16:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:9158 run:96 clone:0 gen:442 core:0x18 unit:0x000001ebab4041545673c9cb5d9bb0c6
15:15:16:WU00:FS01:Uploading 5.44MiB to 171.64.65.84
15:15:16:WU00:FS01:Connecting to 171.64.65.84:8080
15:15:16:WU03:FS01:Starting
15:15:16:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Bosses/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 03 -suffix 01 -version 704 -lifeline 5552 -checkpoint 5 -gpu 0 -gpu-vendor nvidia
15:15:16:WU03:FS01:Started FahCore on PID 5808
15:15:16:WU03:FS01:Core PID:4208
15:15:16:WU03:FS01:FahCore 0x21 started
15:15:17:WU03:FS01:0x21:*********************** Log Started 2016-08-03T15:15:17Z ***********************
15:15:17:WU03:FS01:0x21:Project: 10496 (Run 41, Clone 7, Gen 7)
15:15:17:WU03:FS01:0x21:Unit: 0x000000098ca304f556bba91320b495b9
15:15:17:WU03:FS01:0x21:CPU: 0x00000000000000000000000000000000
15:15:17:WU03:FS01:0x21:Machine: 1
15:15:17:WU03:FS01:0x21:Reading tar file core.xml
15:15:17:WU03:FS01:0x21:Reading tar file system.xml
15:15:18:WU03:FS01:0x21:Reading tar file integrator.xml
15:15:18:WU03:FS01:0x21:Reading tar file state.xml
15:15:20:WU03:FS01:0x21:Digital signatures verified
15:15:20:WU03:FS01:0x21:Folding@home GPU Core21 Folding@home Core
15:15:20:WU03:FS01:0x21:Version 0.0.17
15:15:22:WU00:FS01:Upload 50.59%
15:15:30:WU00:FS01:Upload complete
15:15:30:WU00:FS01:Server responded WORK_ACK (400)
15:15:30:WU00:FS01:Final credit estimate, 36130.00 points
15:15:30:WU00:FS01:Cleaning up
15:15:44:WU03:FS01:0x21:Completed 0 out of 2000000 steps (0%)
15:15:44:WU03:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
15:16:33:WU03:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
15:18:47:WU03:FS01:0x21:Completed 20000 out of 2000000 steps (1%)
15:21:01:WU03:FS01:0x21:Completed 40000 out of 2000000 steps (2%)
15:23:15:WU03:FS01:0x21:Completed 60000 out of 2000000 steps (3%)
15:23:21:WU02:FS02:0x21:Completed 3750000 out of 5000000 steps (75%)
15:25:24:WU03:FS01:0x21:Completed 80000 out of 2000000 steps (4%)
15:27:32:WU03:FS01:0x21:Completed 100000 out of 2000000 steps (5%)
15:29:39:WU03:FS01:0x21:Completed 120000 out of 2000000 steps (6%)
15:31:32:WU03:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
15:32:58:WU02:FS02:0x21:Completed 3800000 out of 5000000 steps (76%)
15:33:12:WU03:FS01:0x21:Completed 140000 out of 2000000 steps (7%)
******************************* Date: 2016-08-03 *******************************
15:35:27:WU03:FS01:0x21:Completed 160000 out of 2000000 steps (8%)
15:37:41:WU03:FS01:0x21:Completed 180000 out of 2000000 steps (9%)
15:39:55:WU03:FS01:0x21:Completed 200000 out of 2000000 steps (10%)
15:42:09:WU03:FS01:0x21:Completed 220000 out of 2000000 steps (11%)
15:43:59:WU02:FS02:0x21:Completed 3850000 out of 5000000 steps (77%)
15:44:24:WU03:FS01:0x21:Completed 240000 out of 2000000 steps (12%)
15:46:44:WU03:FS01:0x21:Completed 260000 out of 2000000 steps (13%)
15:47:53:WU03:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
15:47:53:WU03:FS01:0x21:ERROR:Max Retries Reached
15:47:53:WU03:FS01:0x21:Saving result file logfile_01.txt
15:47:53:WU03:FS01:0x21:Saving result file badstate-0.xml
15:47:59:WU03:FS01:0x21:Saving result file badstate-1.xml
15:48:06:WU03:FS01:0x21:Saving result file badstate-2.xml
15:48:13:WU03:FS01:0x21:Saving result file log.txt
15:48:13:WU03:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
15:48:14:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
15:48:14:WU03:FS01:Sending unit results: id:03 state:SEND error:FAULTY project:10496 run:41 clone:7 gen:7 core:0x21 unit:0x000000098ca304f556bba91320b495b9
15:48:14:WU03:FS01:Uploading 19.85KiB to 140.163.4.245
15:48:14:WU03:FS01:Connecting to 140.163.4.245:8080
15:48:14:WU03:FS01:Upload complete
15:48:14:WU00:FS01:Connecting to 171.67.108.45:80
15:48:14:WU03:FS01:Server responded WORK_ACK (400)
15:48:14:WU03:FS01:Cleaning up
15:48:15:WU00:FS01:Assigned to work server 171.64.65.92
15:48:15:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:GP104 [GeForce GTX 1070] from 171.64.65.92
15:48:15:WU00:FS01:Connecting to 171.64.65.92:8080
15:48:15:WU00:FS01:Downloading 2.73MiB
15:48:18:WU00:FS01:Download complete
15:48:18:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9160 run:189 clone:0 gen:307 core:0x18 unit:0x0000015dab40415c5674649f7219c984
15:48:18:WU00:FS01:Starting
15:48:18:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Bosses/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 00 -suffix 01 -version 704 -lifeline 5552 -checkpoint 5 -gpu 0 -gpu-vendor nvidia
15:48:18:WU00:FS01:Started FahCore on PID 16300
15:48:18:WU00:FS01:Core PID:13776
15:48:18:WU00:FS01:FahCore 0x18 started
15:48:19:WU00:FS01:0x18:*********************** Log Started 2016-08-03T15:48:18Z ***********************
15:48:19:WU00:FS01:0x18:Project: 9160 (Run 189, Clone 0, Gen 307)
15:48:19:WU00:FS01:0x18:Unit: 0x0000015dab40415c5674649f7219c984
15:48:19:WU00:FS01:0x18:CPU: 0x00000000000000000000000000000000
i7 5820k [4.4GHz] | MSI 1070 Gaming | MSI 980 Ti Sea Hawk | Corsair AXi 1500 | Custom 480mm CPU waterloop
User avatar
Bosses_Boss
 
Posts: 15
Joined: Tue Aug 02, 2016 12:54 pm
Location: Canada

Re: I am having trouble getting my two GPUs to both fold.

Postby foldy » Wed Aug 03, 2016 7:08 pm

Is your CPU overclocked and tested stable with e.g. intelburntest? The 4.5 Ghz looks like a psychological value and maybe 4.4 Ghz is really stable.
There is also a cpu stress tester tool for folding StressCPU v2.0
https://folding.stanford.edu/home/download-utilities/

In your screenshot your gtx 980 ti seems overclocked maybe this is not stable with folding -> remove oc.

In your logfile "Bad State detected" shows for gtx 1070 which seems overclocked in your screenshot -> remove oc.

Your gtx 1070 not going over 70°C is maybe because you need a newer msi afterburner for nvidia pascal gpus
http://www.guru3d.com/news-story/downlo ... eta-4.html
At 92°C the gpu hardware will shutdown so set max 90°C temp.
foldy
 
Posts: 1003
Joined: Sat Dec 01, 2012 3:43 pm

Re: I am having trouble getting my two GPUs to both fold.

Postby rwh202 » Wed Aug 03, 2016 9:06 pm

A GPU running at 2d clock speeds is usually due to a driver crash, which in turn is usually due to an unstable GPU over clock.
You should be able to find an entry in the Windows event viewer for a driver crash if that's what happened.
As I'm sure you're aware, there's a complicated interplay between the driver, temperature, power, voltage and boost clock speed of the GPU so even a small shift in operating conditions can move a previously stable over clock over the limit so I'd start by easing any clock offset back a bit and going from there.
rwh202
 
Posts: 308
Joined: Mon Nov 15, 2010 8:51 pm
Location: South Coast, UK

Re: I am having trouble getting my two GPUs to both fold.

Postby Bosses_Boss » Wed Aug 03, 2016 9:47 pm

Thank you both for replying.

foldy wrote:Is your CPU overclocked and tested stable with e.g. intelburntest? The 4.5 Ghz looks like a psychological value and maybe 4.4 Ghz is really stable.
There is also a cpu stress tester tool for folding StressCPU v2.0
https://folding.stanford.edu/home/download-utilities/

In your screenshot your gtx 980 ti seems overclocked maybe this is not stable with folding -> remove oc.

In your logfile "Bad State detected" shows for gtx 1070 which seems overclocked in your screenshot -> remove oc.

Your gtx 1070 not going over 70°C is maybe because you need a newer msi afterburner for nvidia pascal gpus
http://www.guru3d.com/news-story/downlo ... eta-4.html
At 92°C the gpu hardware will shutdown so set max 90°C temp.


CPU OC is being dialed in so I will go back to my known stable 4.4GHz OC. It's just sad if this is it because then all clocks about 4.4 fail :/ even up to 1.34v for 4.5GHz+ but 4.4GHz was 48 hours stable with IBT at 1.23v

Afterburner is up to date. Pascal has GPU Boost 3.0, it's a big change form 2.0. It thermal throttles at really low temps dispute the temp settings in afterburner, it's a vBIOS issue relating to GPU Boost 3.0 afaik.

1070 OC is stable, had it folding at 2GHz [or with GPU Boost 3.0 clocking it down to 1974MHz] for 3 weeks now. For science, I'll go back to stock.

980 Ti I tested at stock speeds too, no difference. Will return to stock for CPU reset.

I think it could also be the NV driver at play here, I haven't had it long and it could be more sensitive to overclocks.

rwh202 wrote:A GPU running at 2d clock speeds is usually due to a driver crash, which in turn is usually due to an unstable GPU over clock.
You should be able to find an entry in the Windows event viewer for a driver crash if that's what happened.
As I'm sure you're aware, there's a complicated interplay between the driver, temperature, power, voltage and boost clock speed of the GPU so even a small shift in operating conditions can move a previously stable over clock over the limit so I'd start by easing any clock offset back a bit and going from there.


Ah yes, NV drivers crashed around an hour after I left for bed. Recovered but not fully I guess.
User avatar
Bosses_Boss
 
Posts: 15
Joined: Tue Aug 02, 2016 12:54 pm
Location: Canada

Re: I am having trouble getting my two GPUs to both fold.

Postby Bosses_Boss » Wed Aug 03, 2016 10:38 pm

Not wanting to jinx it but a switch to 4,4GHz and clean install of a older NV driver seems to fix my issue.

1070 [@60c] is at 1835MHz and 980 Ti [@50c] is at 1400MHz without overclocking.

Just waiting for new projects to get an accurate PPD reading.
User avatar
Bosses_Boss
 
Posts: 15
Joined: Tue Aug 02, 2016 12:54 pm
Location: Canada

Re: I am having trouble getting my two GPUs to both fold.

Postby bruce » Thu Aug 04, 2016 12:16 am

rwh202 wrote:As I'm sure you're aware, there's a complicated interplay between the driver, temperature, power, voltage and boost clock speed of the GPU so even a small shift in operating conditions can move a previously stable over clock over the limit so I'd start by easing any clock offset back a bit and going from there.


That complex interplay is more or less what you do when you overclock. NVidia iis attempting to manage the clock rate based on temperature & power e.g.- they can boost the overclock when the GPU is doing less work than when FAH has it completely loaded. To some extent manual overclocking can get you into performance areas where the drivers are forced to reduce the clock rate, in spite of what you THINK you're doing. I'm not suggesting you can't find some sweet-spot that's better than the drivers can find, but it has become more challenging.

Just the simple act of installing a second GPU can increase the local temperatures enough to move the sweet-spot you used to have.

As has already been noted, "Bad State" is often a sign of not leaving enough overclock margin, and after retrying the WU 3x, it gives up.
bruce
 
Posts: 20940
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: I am having trouble getting my two GPUs to both fold.

Postby bruce » Thu Aug 04, 2016 12:45 am

I did some more trimming of the log you posted above. It was quite a process and if I missed something, say so.
Code: Select all
    06:55:34:WU00:FS01:0x21:Project: 10493 (Run 8, Clone 10, Gen 114)
    10:37:01:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
    11:53:48:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?

    12:53:13:WU03:FS01:0x18:Project: 9660 (Run 1, Clone 17, Gen 160)
    13:25:49:WU03:FS01:0x18:Bad State detected... attempting to resume from last good checkpoint

    15:15:17:WU03:FS01:0x21:Project: 10496 (Run 41, Clone 7, Gen 7)
    15:31:32:WU03:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
    15:47:53:WU03:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
    15:48:13:WU03:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
    15:48:14:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
    15:48:14:WU03:FS01:Sending unit results: id:03 state:SEND error:FAULTY project:10496 run:41 clone:7 gen:7 core:0x21


Observations:
All bad states in that log came from the GPU on slot 01.

Your bad states were mostly, but not exclusively, on Core_21.

Project: 10493 (Run 8, Clone 10, Gen 114) -- One report. Reported as completed by you in spite of bad states
Project: 9660 (Run 1, Clone 17, Gen 160) -- One report. Reported as completed by you in spite of bad states
Project: 10496 (Run 41, Clone 7, Gen 7) -- Three reports. Reported as an error by you and one other person. [The two errors were different.] Completed by a third person.
bruce
 
Posts: 20940
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: I am having trouble getting my two GPUs to both fold.

Postby Bosses_Boss » Thu Aug 04, 2016 1:19 am

bruce wrote:
rwh202 wrote:As I'm sure you're aware, there's a complicated interplay between the driver, temperature, power, voltage and boost clock speed of the GPU so even a small shift in operating conditions can move a previously stable over clock over the limit so I'd start by easing any clock offset back a bit and going from there.


That complex interplay is more or less what you do when you overclock. NVidia iis attempting to manage the clock rate based on temperature & power e.g.- they can boost the overclock when the GPU is doing less work than when FAH has it completely loaded. To some extent manual overclocking can get you into performance areas where the drivers are forced to reduce the clock rate, in spite of what you THINK you're doing. I'm not suggesting you can't find some sweet-spot that's better than the drivers can find, but it has become more challenging.

Just the simple act of installing a second GPU can increase the local temperatures enough to move the sweet-spot you used to have.

As has already been noted, "Bad State" is often a sign of not leaving enough overclock margin, and after retrying the WU 3x, it gives up.


bruce wrote:I did some more trimming of the log you posted above. It was quite a process and if I missed something, say so.
Code: Select all
    06:55:34:WU00:FS01:0x21:Project: 10493 (Run 8, Clone 10, Gen 114)
    10:37:01:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
    11:53:48:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?

    12:53:13:WU03:FS01:0x18:Project: 9660 (Run 1, Clone 17, Gen 160)
    13:25:49:WU03:FS01:0x18:Bad State detected... attempting to resume from last good checkpoint

    15:15:17:WU03:FS01:0x21:Project: 10496 (Run 41, Clone 7, Gen 7)
    15:31:32:WU03:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
    15:47:53:WU03:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
    15:48:13:WU03:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
    15:48:14:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
    15:48:14:WU03:FS01:Sending unit results: id:03 state:SEND error:FAULTY project:10496 run:41 clone:7 gen:7 core:0x21


Observations:
All bad states came from the GPU on slot 01.
You had BS reported mostly, but not exclusively, on Core_21.



Thank you very much for this insight, that is invaluable. Interestingly the 980 Ti is regularly just a little higher PPD. And yet a 1070 is higher FPS in games. I guess there is no replacement for displa--- I mean cores...

Currently at 1.37 Million PPD - https://imgur.com/a/fW2GL

Are bad states also bad for corrupting data or missing data?
User avatar
Bosses_Boss
 
Posts: 15
Joined: Tue Aug 02, 2016 12:54 pm
Location: Canada

Re: I am having trouble getting my two GPUs to both fold.

Postby bruce » Thu Aug 04, 2016 1:29 am

Bad States are generally indicative that a calculation error has been detected. It was discovered that a certain percentage of WUs that had that error could be restarted from the last checkpoint and completed. (say your GPU gives incorrect results about twice a day). Rather tan aborting immediately, a feature was added to automatically retry from the last checkpoint, avoiding the waste of whatever work had been completed. After a certain number of retries, the WU will still be aborted (say your GPU gives incorrect results every few minutes).

Remember consumer grade GPU memory has no parity check. Professional grade GPUs are more costly, but the parity on their VRAM does catch some memory errors. Of course there's no way to tell if the error was in the VRAM or somewhere else.

The theory is that if one pixel is the wrong color in a game, nobody really cares AND gamers won't spend the extra money to avoid it. FAH does it's best to work within the constraints of consumer grade GPUs.
bruce
 
Posts: 20940
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: I am having trouble getting my two GPUs to both fold.

Postby Bosses_Boss » Thu Aug 04, 2016 2:23 am

bruce wrote:Bad States are generally indicative that a calculation error has been detected. It was discovered that a certain percentage of WUs that had that error could be restarted from the last checkpoint and completed. (say your GPU gives incorrect results about twice a day). Rather tan aborting immediately, a feature was added to automatically retry from the last checkpoint, avoiding the waste of whatever work had been completed. After a certain number of retries, the WU will still be aborted (say your GPU gives incorrect results every few minutes).

Remember consumer grade GPU memory has no parity check. Professional grade GPUs are more costly, but the parity on their VRAM does catch some memory errors. Of course there's no way to tell if the error was in the VRAM or somewhere else.

The theory is that if one pixel is the wrong color in a game, nobody really cares AND gamers won't spend the extra money to avoid it. FAH does it's best to work within the constraints of consumer grade GPUs.


Very interesting, thank you. :)
User avatar
Bosses_Boss
 
Posts: 15
Joined: Tue Aug 02, 2016 12:54 pm
Location: Canada

Re: I am having trouble getting my two GPUs to both fold.

Postby rwh202 » Thu Aug 04, 2016 8:27 am

bruce wrote:
rwh202 wrote:As I'm sure you're aware, there's a complicated interplay between the driver, temperature, power, voltage and boost clock speed of the GPU so even a small shift in operating conditions can move a previously stable over clock over the limit so I'd start by easing any clock offset back a bit and going from there.


That complex interplay is more or less what you do when you overclock. NVidia iis attempting to manage the clock rate based on temperature & power e.g.- they can boost the overclock when the GPU is doing less work than when FAH has it completely loaded. To some extent manual overclocking can get you into performance areas where the drivers are forced to reduce the clock rate, in spite of what you THINK you're doing. I'm not suggesting you can't find some sweet-spot that's better than the drivers can find, but it has become more challenging.

Just the simple act of installing a second GPU can increase the local temperatures enough to move the sweet-spot you used to have.

As has already been noted, "Bad State" is often a sign of not leaving enough overclock margin, and after retrying the WU 3x, it gives up.


Yes, 'GPU Boost' and similar techniques have changed the approach (and even the need) to overclocking somewhat. A stock 980 with a nominal 1126 clock and 1216 boost can boost to 1400 MHz all by itself provided the configured power and temperature limits are respected - emphasis should be given on giving the card the power and thermal envelope needed to boost itself.
Manually adding an offset to get a higher speed in the 'usual' voltage/frequency bin doesn't guarantee stability in an other bin (A 200 MHz offset might work in one bin, but I doubt you'd get 1600 MHz stable out of the card if it found itself in the highest bin.
rwh202
 
Posts: 308
Joined: Mon Nov 15, 2010 8:51 pm
Location: South Coast, UK

Re: I am having trouble getting my two GPUs to both fold.

Postby toTOW » Thu Aug 04, 2016 2:24 pm

Bosses_Boss wrote:Ah yes, NV drivers crashed around an hour after I left for bed. Recovered but not fully I guess.

They never do ... you're always good for a reboot when it happens :(
Folding@Home beta tester since 2002. Folding Forum moderator since July 2008.

FAH-Addict : latest news, tests and reviews about Folding@Home project.

Image
User avatar
toTOW
Site Moderator
 
Posts: 8340
Joined: Sun Dec 02, 2007 10:38 am
Location: Bordeaux, France

Re: I am having trouble getting my two GPUs to both fold.

Postby Bosses_Boss » Thu Aug 04, 2016 9:34 pm

rwh202 wrote:
bruce wrote:
rwh202 wrote:As I'm sure you're aware, there's a complicated interplay between the driver, temperature, power, voltage and boost clock speed of the GPU so even a small shift in operating conditions can move a previously stable over clock over the limit so I'd start by easing any clock offset back a bit and going from there.


That complex interplay is more or less what you do when you overclock. NVidia iis attempting to manage the clock rate based on temperature & power e.g.- they can boost the overclock when the GPU is doing less work than when FAH has it completely loaded. To some extent manual overclocking can get you into performance areas where the drivers are forced to reduce the clock rate, in spite of what you THINK you're doing. I'm not suggesting you can't find some sweet-spot that's better than the drivers can find, but it has become more challenging.

Just the simple act of installing a second GPU can increase the local temperatures enough to move the sweet-spot you used to have.

As has already been noted, "Bad State" is often a sign of not leaving enough overclock margin, and after retrying the WU 3x, it gives up.


Yes, 'GPU Boost' and similar techniques have changed the approach (and even the need) to overclocking somewhat. A stock 980 with a nominal 1126 clock and 1216 boost can boost to 1400 MHz all by itself provided the configured power and temperature limits are respected - emphasis should be given on giving the card the power and thermal envelope needed to boost itself.
Manually adding an offset to get a higher speed in the 'usual' voltage/frequency bin doesn't guarantee stability in an other bin (A 200 MHz offset might work in one bin, but I doubt you'd get 1600 MHz stable out of the card if it found itself in the highest bin.


My 980 Ti Seahawk folding without errors/driver crashes folding for 24 hours now. http://i.imgur.com/qTojzvK.png

ASIC Quality 73.5%, better than 54% of all 980 Tis. Pretty much the perfect card to watercool. I can probably get even higher if I push the voltage up but for folding this'll do just swell.
User avatar
Bosses_Boss
 
Posts: 15
Joined: Tue Aug 02, 2016 12:54 pm
Location: Canada

Previous

Return to GPU Projects and FahCores

Who is online

Users browsing this forum: No registered users and 1 guest

cron