Core 21 failures on GTX970

[Kebast]

I've had a couple bad WUs in the past 4 days. Here's the logs:

Code: Select all

*********************** Log Started 2015-11-13T08:16:25Z ***********************
08:16:25:************************* Folding@home Client *************************
08:16:25:      Website: http://folding.stanford.edu/
08:16:25:    Copyright: (c) 2009-2014 Stanford University
08:16:25:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
08:16:25:         Args: 
08:16:25:       Config: C:/ProgramData/FAHClient/config.xml
08:16:25:******************************** Build ********************************
08:16:25:      Version: 7.4.4
08:16:25:         Date: Mar 4 2014
08:16:25:         Time: 20:26:54
08:16:25:      SVN Rev: 4130
08:16:25:       Branch: fah/trunk/client
08:16:25:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
08:16:25:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
08:16:25:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
08:16:25:     Platform: win32 XP
08:16:25:         Bits: 32
08:16:25:         Mode: Release
08:16:25:******************************* System ********************************
08:16:25:          CPU: AMD FX(tm)-6300 Six-Core Processor
08:16:25:       CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
08:16:25:         CPUs: 6
08:16:25:       Memory: 7.97GiB
08:16:25:  Free Memory: 6.91GiB
08:16:25:      Threads: WINDOWS_THREADS
08:16:25:   OS Version: 6.1
08:16:25:  Has Battery: false
08:16:25:   On Battery: false
08:16:25:   UTC Offset: -5
08:16:25:          PID: 3116
08:16:25:          CWD: C:/ProgramData/FAHClient
08:16:25:           OS: Windows 7 Home Premium
08:16:25:      OS Arch: AMD64
08:16:25:         GPUs: 1
08:16:25:        GPU 0: NVIDIA:5 GM204 [GeForce GTX 970]
08:16:25:         CUDA: 5.2
08:16:25:  CUDA Driver: 7050
08:16:25:Win32 Service: false
08:16:25:***********************************************************************
08:16:25:<config>
08:16:25:  <!-- HTTP Server -->
08:16:25:  <allow v='127.0.0.1,192.168.66.95-192.168.66.151'/>
08:16:25:
08:16:25:  <!-- Network -->
08:16:25:  <proxy v=':8080'/>
08:16:25:
08:16:25:  <!-- Remote Command Server -->
08:16:25:  <command-allow-no-pass v='127.0.0.1,192.168.66.95-192.168.66.151'/>
08:16:25:
08:16:25:  <!-- Slot Control -->
08:16:25:  <power v='full'/>
08:16:25:
08:16:25:  <!-- User Information -->
08:16:25:  <passkey v='********************************'/>
08:16:25:  <team v='229226'/>
08:16:25:  <user v='Kebast'/>
08:16:25:
08:16:25:  <!-- Work Unit Control -->
08:16:25:  <next-unit-percentage v='100'/>
08:16:25:
08:16:25:  <!-- Folding Slots -->
08:16:25:  <slot id='1' type='GPU'/>
08:16:25:</config>
08:16:25:Trying to access database...
08:16:26:Successfully acquired database lock
08:16:26:Enabled folding slot 01: READY gpu:0:GM204 [GeForce GTX 970]

18:33:48:WU01:FS01:Download complete
18:33:48:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9638 run:0 clone:9 gen:54 core:0x21 unit:0x00000041ab436c9b5609bee318cb4e30
18:33:48:WU01:FS01:Starting
18:33:48:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 704 -lifeline 3116 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
18:33:48:WU01:FS01:Started FahCore on PID 4760
18:33:48:WU01:FS01:Core PID:4392
18:33:48:WU01:FS01:FahCore 0x21 started
18:33:49:WU01:FS01:0x21:*********************** Log Started 2015-11-13T18:33:48Z ***********************
18:33:49:WU01:FS01:0x21:Project: 9638 (Run 0, Clone 9, Gen 54)
18:33:49:WU01:FS01:0x21:Unit: 0x00000041ab436c9b5609bee318cb4e30
18:33:49:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
18:33:49:WU01:FS01:0x21:Machine: 1
18:33:49:WU01:FS01:0x21:Reading tar file core.xml
18:33:49:WU01:FS01:0x21:Reading tar file integrator.xml
18:33:49:WU01:FS01:0x21:Reading tar file state.xml
18:33:49:WU01:FS01:0x21:Reading tar file system.xml
18:33:49:WU01:FS01:0x21:Digital signatures verified
18:33:49:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
18:33:49:WU01:FS01:0x21:Version 0.0.12

18:34:30:WU01:FS01:0x21:Completed 0 out of 2000000 steps (0%)
18:34:30:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900

18:36:33:WU01:FS01:0x21:Completed 20000 out of 2000000 steps (1%)
18:38:27:WU01:FS01:0x21:Completed 40000 out of 2000000 steps (2%)
18:40:21:WU01:FS01:0x21:Completed 60000 out of 2000000 steps (3%)
18:42:16:WU01:FS01:0x21:Completed 80000 out of 2000000 steps (4%)
18:44:10:WU01:FS01:0x21:Completed 100000 out of 2000000 steps (5%)
18:46:16:WU01:FS01:0x21:Completed 120000 out of 2000000 steps (6%)
18:48:10:WU01:FS01:0x21:Completed 140000 out of 2000000 steps (7%)
18:50:05:WU01:FS01:0x21:Completed 160000 out of 2000000 steps (8%)
18:52:00:WU01:FS01:0x21:Completed 180000 out of 2000000 steps (9%)
18:53:55:WU01:FS01:0x21:Completed 200000 out of 2000000 steps (10%)
18:56:01:WU01:FS01:0x21:Completed 220000 out of 2000000 steps (11%)
18:57:56:WU01:FS01:0x21:Completed 240000 out of 2000000 steps (12%)
18:59:51:WU01:FS01:0x21:Completed 260000 out of 2000000 steps (13%)
19:01:46:WU01:FS01:0x21:Completed 280000 out of 2000000 steps (14%)
19:03:41:WU01:FS01:0x21:Completed 300000 out of 2000000 steps (15%)
19:05:47:WU01:FS01:0x21:Completed 320000 out of 2000000 steps (16%)
19:07:42:WU01:FS01:0x21:Completed 340000 out of 2000000 steps (17%)
19:09:37:WU01:FS01:0x21:Completed 360000 out of 2000000 steps (18%)
19:11:33:WU01:FS01:0x21:Completed 380000 out of 2000000 steps (19%)
19:13:28:WU01:FS01:0x21:Completed 400000 out of 2000000 steps (20%)
19:13:37:WU01:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
19:15:32:WU01:FS01:0x21:Completed 320000 out of 2000000 steps (16%)
19:17:27:WU01:FS01:0x21:Completed 340000 out of 2000000 steps (17%)
19:19:22:WU01:FS01:0x21:Completed 360000 out of 2000000 steps (18%)
19:21:17:WU01:FS01:0x21:Completed 380000 out of 2000000 steps (19%)
19:23:12:WU01:FS01:0x21:Completed 400000 out of 2000000 steps (20%)
19:25:18:WU01:FS01:0x21:Completed 420000 out of 2000000 steps (21%)
19:27:13:WU01:FS01:0x21:Completed 440000 out of 2000000 steps (22%)
19:29:08:WU01:FS01:0x21:Completed 460000 out of 2000000 steps (23%)
19:31:03:WU01:FS01:0x21:Completed 480000 out of 2000000 steps (24%)
19:32:59:WU01:FS01:0x21:Completed 500000 out of 2000000 steps (25%)
19:35:05:WU01:FS01:0x21:Completed 520000 out of 2000000 steps (26%)
19:37:00:WU01:FS01:0x21:Completed 540000 out of 2000000 steps (27%)
19:38:55:WU01:FS01:0x21:Completed 560000 out of 2000000 steps (28%)
19:40:50:WU01:FS01:0x21:Completed 580000 out of 2000000 steps (29%)
19:42:45:WU01:FS01:0x21:Completed 600000 out of 2000000 steps (30%)
19:42:55:WU01:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
19:44:49:WU01:FS01:0x21:Completed 520000 out of 2000000 steps (26%)
19:46:45:WU01:FS01:0x21:Completed 540000 out of 2000000 steps (27%)
19:48:39:WU01:FS01:0x21:Completed 560000 out of 2000000 steps (28%)
19:50:34:WU01:FS01:0x21:Completed 580000 out of 2000000 steps (29%)
19:52:29:WU01:FS01:0x21:Completed 600000 out of 2000000 steps (30%)
19:54:36:WU01:FS01:0x21:Completed 620000 out of 2000000 steps (31%)
19:56:31:WU01:FS01:0x21:Completed 640000 out of 2000000 steps (32%)
19:58:26:WU01:FS01:0x21:Completed 660000 out of 2000000 steps (33%)
20:00:21:WU01:FS01:0x21:Completed 680000 out of 2000000 steps (34%)
20:02:16:WU01:FS01:0x21:Completed 700000 out of 2000000 steps (35%)
20:02:25:WU01:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
20:02:25:WU01:FS01:0x21:Max number of retries reached. Aborting.
20:02:25:WU01:FS01:0x21:ERROR:Max Retries Reached
20:02:25:WU01:FS01:0x21:Saving result file logfile_01.txt
20:02:25:WU01:FS01:0x21:Saving result file log.txt
20:02:25:WU01:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
20:02:26:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
20:02:26:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9638 run:0 clone:9 gen:54 core:0x21 unit:0x00000041ab436c9b5609bee318cb4e30
20:02:26:WU01:FS01:Uploading 12.00KiB to 171.67.108.155
20:02:26:WU01:FS01:Connecting to 171.67.108.155:8080
20:02:27:WU01:FS01:Upload complete
20:02:27:WU01:FS01:Server responded WORK_ACK (400)
20:02:27:WU01:FS01:Cleaning up

18:32:41:WU01:FS01:Download complete
18:32:41:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9205 run:14 clone:39 gen:23 core:0x21 unit:0x0000007b664f2dd055d4cf23527aa8b6
18:32:41:WU01:FS01:Starting
18:32:41:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 704 -lifeline 3116 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
18:32:41:WU01:FS01:Started FahCore on PID 1304
18:32:41:WU01:FS01:Core PID:968
18:32:41:WU01:FS01:FahCore 0x21 started
18:32:41:WU01:FS01:0x21:*********************** Log Started 2015-11-15T18:32:41Z ***********************
18:32:41:WU01:FS01:0x21:Project: 9205 (Run 14, Clone 39, Gen 23)
18:32:41:WU01:FS01:0x21:Unit: 0x0000007b664f2dd055d4cf23527aa8b6
18:32:41:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
18:32:41:WU01:FS01:0x21:Machine: 1
18:32:41:WU01:FS01:0x21:Reading tar file core.xml
18:32:41:WU01:FS01:0x21:Reading tar file system.xml
18:32:42:WU01:FS01:0x21:Reading tar file integrator.xml
18:32:42:WU01:FS01:0x21:Reading tar file state.xml
18:32:44:WU01:FS01:0x21:Digital signatures verified
18:32:44:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
18:32:44:WU01:FS01:0x21:Version 0.0.12
18:33:52:WU01:FS01:0x21:Completed 0 out of 2500000 steps (0%)
18:33:52:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
18:39:56:WU01:FS01:0x21:Completed 25000 out of 2500000 steps (1%)
18:45:43:WU01:FS01:0x21:Completed 50000 out of 2500000 steps (2%)
18:51:30:WU01:FS01:0x21:Completed 75000 out of 2500000 steps (3%)
18:57:17:WU01:FS01:0x21:Completed 100000 out of 2500000 steps (4%)
19:03:26:WU01:FS01:0x21:Completed 125000 out of 2500000 steps (5%)
19:09:12:WU01:FS01:0x21:Completed 150000 out of 2500000 steps (6%)
19:14:59:WU01:FS01:0x21:Completed 175000 out of 2500000 steps (7%)
19:20:46:WU01:FS01:0x21:Completed 200000 out of 2500000 steps (8%)
19:26:55:WU01:FS01:0x21:Completed 225000 out of 2500000 steps (9%)
19:32:42:WU01:FS01:0x21:Completed 250000 out of 2500000 steps (10%)
19:38:29:WU01:FS01:0x21:Completed 275000 out of 2500000 steps (11%)
19:44:16:WU01:FS01:0x21:Completed 300000 out of 2500000 steps (12%)
19:50:25:WU01:FS01:0x21:Completed 325000 out of 2500000 steps (13%)
19:56:13:WU01:FS01:0x21:Completed 350000 out of 2500000 steps (14%)
20:02:01:WU01:FS01:0x21:Completed 375000 out of 2500000 steps (15%)
20:07:48:WU01:FS01:0x21:Completed 400000 out of 2500000 steps (16%)
20:08:06:WU01:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
20:13:53:WU01:FS01:0x21:Completed 325000 out of 2500000 steps (13%)
20:19:40:WU01:FS01:0x21:Completed 350000 out of 2500000 steps (14%)
20:25:28:WU01:FS01:0x21:Completed 375000 out of 2500000 steps (15%)
20:31:16:WU01:FS01:0x21:Completed 400000 out of 2500000 steps (16%)
20:31:34:WU01:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
******************************* Date: 2015-11-15 *******************************
20:37:21:WU01:FS01:0x21:Completed 325000 out of 2500000 steps (13%)
20:43:08:WU01:FS01:0x21:Completed 350000 out of 2500000 steps (14%)
20:48:56:WU01:FS01:0x21:Completed 375000 out of 2500000 steps (15%)
20:54:44:WU01:FS01:0x21:Completed 400000 out of 2500000 steps (16%)
21:00:53:WU01:FS01:0x21:Completed 425000 out of 2500000 steps (17%)
21:06:41:WU01:FS01:0x21:Completed 450000 out of 2500000 steps (18%)
21:12:29:WU01:FS01:0x21:Completed 475000 out of 2500000 steps (19%)
21:18:17:WU01:FS01:0x21:Completed 500000 out of 2500000 steps (20%)
21:24:26:WU01:FS01:0x21:Completed 525000 out of 2500000 steps (21%)
21:30:14:WU01:FS01:0x21:Completed 550000 out of 2500000 steps (22%)
21:36:02:WU01:FS01:0x21:Completed 575000 out of 2500000 steps (23%)
21:41:50:WU01:FS01:0x21:Completed 600000 out of 2500000 steps (24%)
21:47:58:WU01:FS01:0x21:Completed 625000 out of 2500000 steps (25%)
21:53:46:WU01:FS01:0x21:Completed 650000 out of 2500000 steps (26%)
21:59:33:WU01:FS01:0x21:Completed 675000 out of 2500000 steps (27%)
22:05:21:WU01:FS01:0x21:Completed 700000 out of 2500000 steps (28%)
22:11:30:WU01:FS01:0x21:Completed 725000 out of 2500000 steps (29%)
22:17:18:WU01:FS01:0x21:Completed 750000 out of 2500000 steps (30%)
22:23:06:WU01:FS01:0x21:Completed 775000 out of 2500000 steps (31%)
22:28:54:WU01:FS01:0x21:Completed 800000 out of 2500000 steps (32%)
22:35:03:WU01:FS01:0x21:Completed 825000 out of 2500000 steps (33%)
22:40:51:WU01:FS01:0x21:Completed 850000 out of 2500000 steps (34%)
22:46:39:WU01:FS01:0x21:Completed 875000 out of 2500000 steps (35%)
22:52:27:WU01:FS01:0x21:Completed 900000 out of 2500000 steps (36%)
22:58:35:WU01:FS01:0x21:Completed 925000 out of 2500000 steps (37%)
23:04:23:WU01:FS01:0x21:Completed 950000 out of 2500000 steps (38%)
23:10:11:WU01:FS01:0x21:Completed 975000 out of 2500000 steps (39%)
23:15:59:WU01:FS01:0x21:Completed 1000000 out of 2500000 steps (40%)
23:22:08:WU01:FS01:0x21:Completed 1025000 out of 2500000 steps (41%)
23:27:55:WU01:FS01:0x21:Completed 1050000 out of 2500000 steps (42%)
23:33:42:WU01:FS01:0x21:Completed 1075000 out of 2500000 steps (43%)
23:39:30:WU01:FS01:0x21:Completed 1100000 out of 2500000 steps (44%)
23:45:39:WU01:FS01:0x21:Completed 1125000 out of 2500000 steps (45%)
23:51:27:WU01:FS01:0x21:Completed 1150000 out of 2500000 steps (46%)
23:57:15:WU01:FS01:0x21:Completed 1175000 out of 2500000 steps (47%)
00:03:03:WU01:FS01:0x21:Completed 1200000 out of 2500000 steps (48%)
00:09:12:WU01:FS01:0x21:Completed 1225000 out of 2500000 steps (49%)
00:15:00:WU01:FS01:0x21:Completed 1250000 out of 2500000 steps (50%)
00:20:47:WU01:FS01:0x21:Completed 1275000 out of 2500000 steps (51%)
00:26:36:WU01:FS01:0x21:Completed 1300000 out of 2500000 steps (52%)
00:32:44:WU01:FS01:0x21:Completed 1325000 out of 2500000 steps (53%)
00:38:32:WU01:FS01:0x21:Completed 1350000 out of 2500000 steps (54%)
00:44:20:WU01:FS01:0x21:Completed 1375000 out of 2500000 steps (55%)
00:50:08:WU01:FS01:0x21:Completed 1400000 out of 2500000 steps (56%)
00:56:18:WU01:FS01:0x21:Completed 1425000 out of 2500000 steps (57%)
01:02:06:WU01:FS01:0x21:Completed 1450000 out of 2500000 steps (58%)
01:07:54:WU01:FS01:0x21:Completed 1475000 out of 2500000 steps (59%)
01:13:41:WU01:FS01:0x21:Completed 1500000 out of 2500000 steps (60%)
01:19:50:WU01:FS01:0x21:Completed 1525000 out of 2500000 steps (61%)
01:25:38:WU01:FS01:0x21:Completed 1550000 out of 2500000 steps (62%)
01:31:26:WU01:FS01:0x21:Completed 1575000 out of 2500000 steps (63%)
01:37:14:WU01:FS01:0x21:Completed 1600000 out of 2500000 steps (64%)
01:37:31:WU01:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
01:37:31:WU01:FS01:0x21:Max number of retries reached. Aborting.
01:37:31:WU01:FS01:0x21:ERROR:Max Retries Reached
01:37:31:WU01:FS01:0x21:Saving result file logfile_01.txt
01:37:31:WU01:FS01:0x21:Saving result file log.txt
01:37:31:WU01:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
01:37:32:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
01:37:32:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9205 run:14 clone:39 gen:23 core:0x21 unit:0x0000007b664f2dd055d4cf23527aa8b6
01:37:32:WU01:FS01:Uploading 3.13KiB to 171.64.65.104
01:37:32:WU01:FS01:Connecting to 171.64.65.104:8080
01:37:33:WU01:FS01:Upload complete
01:37:33:WU01:FS01:Server responded WORK_ACK (400)
01:37:33:WU01:FS01:Cleaning up
01:37:34:WU00:FS01:Assigned to work server 171.67.108.155
01:37:34:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GM204 [GeForce GTX 970] from 171.67.108.155

[Duce H_K_]

There are plenty of such topics. Maybe programmers upgrade 21 core to more stable version somewhen?.. The only thing left for contributors at today is to downclock & overvltage our working horses

Code: Select all

13:23:51:************************* Folding@home Client *************************
13:23:51:      Website: http://folding.stanford.edu/
13:23:51:    Copyright: (c) 2009-2014 Stanford University
13:23:51:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
13:23:51:         Args: 
13:23:51:       Config: E:/Docbase/FaH-workdir/config.xml
13:23:51:******************************** Build ********************************
13:23:51:      Version: 7.4.4
13:23:51:         Date: Mar 4 2014
13:23:51:         Time: 20:26:54
13:23:51:      SVN Rev: 4130
13:23:51:       Branch: fah/trunk/client
13:23:51:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
13:23:51:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
13:23:51:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
13:23:51:     Platform: Win 8.1
13:23:51:         Bits: 64
13:23:51:         Mode: Release
13:23:51:******************************* System ********************************
13:23:51:          CPU: AMD A8-5600K APU with Radeon(tm) HD Graphics
13:23:51:       CPU ID: AuthenticAMD Family 21 Model 16 Stepping 1
13:23:51:         CPUs: 4
13:23:51:       Memory: 3.96GiB
13:23:51:  Free Memory: 2.69GiB
13:23:51:      Threads: WINDOWS_THREADS
13:23:51:   OS Version: 6.2
13:23:51:  Has Battery: false
13:23:51:   On Battery: false
13:23:51:   UTC Offset: 3
13:23:51:          PID: 3848
13:23:51:          CWD: E:/Docbase/FaH-workdir
13:23:51:           OS: Windows 8.1 Pro
13:23:51:      OS Arch: AMD64
13:23:51:         GPUs: 1
13:23:51:        GPU 0: ATI:5 Hawaii [Radeon R9 200/300X Series]
13:23:51:         CUDA: Not detected
13:23:51:Win32 Service: false
13:23:51:***********************************************************************
13:23:51:<config>
13:23:51:  <service-description v='Folding@home Client'/>
13:23:51:  <service-restart v='true'/>
13:23:51:  <service-restart-delay v='5000'/>
13:23:51:
13:23:51:  <!-- Client Control -->
13:23:51:  <client-threads v='6'/>
13:23:51:  <cycle-rate v='4'/>
13:23:51:  <cycles v='-1'/>
13:23:51:  <data-directory v='.'/>
13:23:51:  <disable-sleep-when-active v='true'/>
13:23:51:  <exec-directory v='C:\NO_UAC\FAHClient7'/>
13:23:51:  <exit-when-done v='false'/>
13:23:51:  <fold-anon v='false'/>
13:23:51:  <open-web-control v='false'/>
13:23:51:
13:23:51:  <!-- Configuration -->
13:23:51:  <config-rotate v='true'/>
13:23:51:  <config-rotate-dir v='configs'/>
13:23:51:  <config-rotate-max v='16'/>
13:23:51:
13:23:51:  <!-- Debugging -->
13:23:51:  <assignment-servers>
13:23:51:    assign3.stanford.edu:8080 assign4.stanford.edu:80
13:23:51:  </assignment-servers>
13:23:51:  <auth-as v='true'/>
13:23:51:  <capture-directory v='capture'/>
13:23:51:  <capture-on-error v='false'/>
13:23:51:  <capture-packets v='false'/>
13:23:51:  <capture-requests v='false'/>
13:23:51:  <capture-responses v='false'/>
13:23:51:  <capture-sockets v='false'/>
13:23:51:  <core-exec v='FahCore_$type'/>
13:23:51:  <core-wrapper-exec v='FAHCoreWrapper'/>
13:23:51:  <debug-sockets v='false'/>
13:23:51:  <exception-locations v='true'/>
13:23:51:  <gpu-assignment-servers>
13:23:51:    assign-GPU.stanford.edu:80 assign-GPU2.stanford.edu:80
13:23:51:  </gpu-assignment-servers>
13:23:51:  <stack-traces v='false'/>
13:23:51:
13:23:51:  <!-- Error Handling -->
13:23:51:  <max-slot-errors v='10'/>
13:23:51:  <max-unit-errors v='5'/>
13:23:51:
13:23:51:  <!-- Folding Core -->
13:23:51:  <checkpoint v='6'/>
13:23:51:  <core-dir v='cores'/>
13:23:51:  <core-priority v='low'/>
13:23:51:  <cpu-affinity v='false'/>
13:23:51:  <cpu-usage v='100'/>
13:23:51:  <gpu-usage v='100'/>
13:23:51:  <no-assembly v='false'/>
13:23:51:
13:23:51:  <!-- Folding Slot Configuration -->
13:23:51:  <cause v='ANY'/>
13:23:51:  <client-subtype v='STDCLI'/>
13:23:51:  <client-type v='normal'/>
13:23:51:  <cpu-species v='X86_AMD'/>
13:23:51:  <cpu-type v='AMD64'/>
13:23:51:  <cpus v='-1'/>
13:23:51:  <extra-core-args v='-tmax=80 -twait=800'/>
13:23:51:  <gpu v='true'/>
13:23:51:  <max-packet-size v='normal'/>
13:23:51:  <os-species v='UNKNOWN'/>
13:23:51:  <os-type v='WIN32'/>
13:23:51:  <project-key v='0'/>
13:23:51:  <smp v='true'/>
13:23:51:
13:23:51:  <!-- GUI -->
13:23:51:  <gui-enabled v='true'/>
13:23:51:
13:23:51:  <!-- HTTP Server -->
13:23:51:  <allow v='0.0.0.0/0'/>
13:23:51:  <connection-timeout v='60'/>
13:23:51:  <deny v='255.255.255.255/255.255.255.255'/>
13:23:51:  <http-addresses v='0:7396'/>
13:23:51:  <https-addresses v=''/>
13:23:51:  <max-connect-time v='900'/>
13:23:51:  <max-connections v='800'/>
13:23:51:  <max-request-length v='52428800'/>
13:23:51:  <min-connect-time v='300'/>
13:23:51:  <threads v='4'/>
13:23:51:
13:23:51:  <!-- Logging -->
13:23:51:  <log v='log.txt'/>
13:23:51:  <log-color v='false'/>
13:23:51:  <log-crlf v='true'/>
13:23:51:  <log-date v='false'/>
13:23:51:  <log-date-periodically v='21600'/>
13:23:51:  <log-debug v='true'/>
13:23:51:  <log-domain v='false'/>
13:23:51:  <log-header v='true'/>
13:23:51:  <log-level v='true'/>
13:23:51:  <log-no-info-header v='true'/>
13:23:51:  <log-redirect v='false'/>
13:23:51:  <log-rotate v='true'/>
13:23:51:  <log-rotate-dir v='logs'/>
13:23:51:  <log-rotate-max v='16'/>
13:23:51:  <log-short-level v='false'/>
13:23:51:  <log-simple-domains v='true'/>
13:23:51:  <log-thread-id v='false'/>
13:23:51:  <log-thread-prefix v='true'/>
13:23:51:  <log-time v='true'/>
13:23:51:  <log-to-screen v='true'/>
13:23:51:  <log-truncate v='false'/>
13:23:51:  <verbosity v='5'/>
13:23:51:
13:23:51:  <!-- Network -->
13:23:51:  <proxy v=':8080'/>
13:23:51:  <proxy-enable v='false'/>
13:23:51:  <proxy-pass v=''/>
13:23:51:  <proxy-user v=''/>
13:23:51:
13:23:51:  <!-- Process Control -->
13:23:51:  <child v='false'/>
13:23:51:  <daemon v='false'/>
13:23:51:  <pid v='false'/>
13:23:51:  <pid-file v='Folding@home Client.pid'/>
13:23:51:  <respawn v='false'/>
13:23:51:  <service v='false'/>
13:23:51:
13:23:51:  <!-- Remote Command Server -->
13:23:51:  <command-address v='0.0.0.0'/>
13:23:51:  <command-allow-no-pass v='192.168.161.0/25'/>
13:23:51:  <command-deny-no-pass v=''/>
13:23:51:  <command-enable v='true'/>
13:23:51:  <command-port v='49474'/>
13:23:51:  <password v='***********'/>
13:23:51:
13:23:51:  <!-- Slot Control -->
13:23:51:  <idle v='false'/>
13:23:51:  <max-shutdown-wait v='60'/>
13:23:51:  <pause-on-battery v='false'/>
13:23:51:  <pause-on-start v='true'/>
13:23:51:  <paused v='false'/>
13:23:51:  <power v='full'/>
13:23:51:
13:23:51:  <!-- User Information -->
13:23:51:  <machine-id v='0'/>
13:23:51:  <passkey v='********************************'/>
13:23:51:  <team v='47191'/>
13:23:51:  <user v='Duce-HK_FLDC_1CbwZo443yjerS5JY7QQ9AiavrSyT5NpXt'/>
13:23:51:
13:23:51:  <!-- Web Server -->
13:23:51:  <web-allow v='127.0.0.1'/>
13:23:51:  <web-deny v='0/0'/>
13:23:51:  <web-enable v='true'/>
13:23:51:
13:23:51:  <!-- Web Server Sessions -->
13:23:51:  <session-cookie v='sid'/>
13:23:51:  <session-lifetime v='86400'/>
13:23:51:  <session-timeout v='3600'/>
13:23:51:
13:23:51:  <!-- Work Unit Control -->
13:23:51:  <dump-after-deadline v='true'/>
13:23:51:  <max-queue v='16'/>
13:23:51:  <max-units v='0'/>
13:23:51:  <next-unit-percentage v='99'/>
13:23:51:  <stall-detection-enabled v='false'/>
13:23:51:  <stall-percent v='5'/>
13:23:51:  <stall-timeout v='1800'/>
13:23:51:
13:23:51:  <!-- Folding Slots -->
13:23:51:  <slot id='1' type='GPU'>
13:23:51:    <client-type v='beta'/>
13:23:51:    <gpu-index v='0'/>
13:23:51:    <opencl-index v='0'/>
13:23:51:    <paused v='true'/>
13:23:51:  </slot>
13:23:51:  <slot id='0' type='GPU'>
13:23:51:    <client-type v='advanced'/>
13:23:51:    <gpu-index v='0'/>
13:23:51:    <opencl-index v='0'/>
13:23:51:    <paused v='true'/>
13:23:51:  </slot>
13:23:51:</config>
13:23:52:Trying to access database...
13:23:52:Successfully acquired database lock
13:23:52:ERROR:Exception: GPU index 0 already in use
13:23:53:WU02:FS00:Starting

13:27:57:WU00:FS01:0x21:*********************** Log Started 2015-11-17T13:27:56Z ***********************
13:27:57:WU00:FS01:0x21:Project: 9704 (Run 39, Clone 5, Gen 102)
* * * it goes well so i ommited it's progress lines * * *
13:42:53:Removing old file 'configs/config-20151115-180911.xml'
13:42:53:Saving configuration to config.xml
13:42:53:<config>
13:42:53:  <!-- Folding Core -->
13:42:53:  <checkpoint v='6'/>
13:42:53:  <core-priority v='low'/>
13:42:53:
13:42:53:  <!-- Folding Slot Configuration -->
13:42:53:  <extra-core-args v='-tmax=80 -twait=800'/>
13:42:53:
13:42:53:  <!-- HTTP Server -->
13:42:53:  <allow v='0.0.0.0/0'/>
13:42:53:  <deny v='255.255.255.255/255.255.255.255'/>
13:42:53:
13:42:53:  <!-- Logging -->
13:42:53:  <verbosity v='5'/>
13:42:53:
13:42:53:  <!-- Network -->
13:42:53:  <proxy v=':8080'/>
13:42:53:
13:42:53:  <!-- Remote Command Server -->
13:42:53:  <command-allow-no-pass v='192.168.161.0/25'/>
13:42:53:  <command-deny-no-pass v=''/>
13:42:53:  <command-port v='49474'/>
13:42:53:  <password v='***********'/>
13:42:53:
13:42:53:  <!-- Slot Control -->
13:42:53:  <pause-on-battery v='false'/>
13:42:53:  <pause-on-start v='true'/>
13:42:53:  <power v='full'/>
13:42:53:
13:42:53:  <!-- User Information -->
13:42:53:  <passkey v='********************************'/>
13:42:53:  <team v='47191'/>
13:42:53:  <user v='Duce-HK_FLDC_1CbwZo443yjerS5JY7QQ9AiavrSyT5NpXt'/>
13:42:53:
13:42:53:  <!-- Folding Slots -->
13:42:53:  <slot id='1' type='GPU'>
13:42:53:    <client-type v='beta'/>
13:42:53:    <gpu-index v='0'/>
13:42:53:    <opencl-index v='0'/>
13:42:53:  </slot>
13:42:53:</config>
13:42:53:WARNING:WU02:Slot ID 0 no longer exists, migrating to FS01
13:43:11:Removing old file 'configs/config-20151115-181311.xml'
13:43:11:Saving configuration to config.xml
13:43:11:<config>
13:43:11:  <!-- Folding Core -->
13:43:11:  <checkpoint v='6'/>
13:43:11:  <core-priority v='low'/>
13:43:11:
13:43:11:  <!-- Folding Slot Configuration -->
13:43:11:  <extra-core-args v='-tmax=80 -twait=800'/>
13:43:11:
13:43:11:  <!-- HTTP Server -->
13:43:11:  <allow v='0.0.0.0/0'/>
13:43:11:  <deny v='255.255.255.255/255.255.255.255'/>
13:43:11:
13:43:11:  <!-- Logging -->
13:43:11:  <verbosity v='5'/>
13:43:11:
13:43:11:  <!-- Network -->
13:43:11:  <proxy v=':8080'/>
13:43:11:
13:43:11:  <!-- Remote Command Server -->
13:43:11:  <command-allow-no-pass v='192.168.161.0/25'/>
13:43:11:  <command-deny-no-pass v=''/>
13:43:11:  <command-port v='49474'/>
13:43:11:  <password v='***********'/>
13:43:11:
13:43:11:  <!-- Slot Control -->
13:43:11:  <pause-on-battery v='false'/>
13:43:11:  <pause-on-start v='true'/>
13:43:11:  <power v='full'/>
13:43:11:
13:43:11:  <!-- User Information -->
13:43:11:  <passkey v='********************************'/>
13:43:11:  <team v='47191'/>
13:43:11:  <user v='Duce-HK_FLDC_1CbwZo443yjerS5JY7QQ9AiavrSyT5NpXt'/>
13:43:11:
13:43:11:  <!-- Folding Slots -->
13:43:11:  <slot id='1' type='GPU'>
13:43:11:    <client-type v='beta'/>
13:43:11:    <gpu-index v='0'/>
13:43:11:    <opencl-index v='0'/>
13:43:11:  </slot>
13:43:11:</config>
14:52:10:WU02:FS01:Starting
14:52:10:WU02:FS01:Running FahCore: C:\NO_UAC\FAHClient7/FAHCoreWrapper.exe E:/Docbase/FaH-workdir/cores/web.stanford.edu/~pande/Win32/AMD64/ATI/R600/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 704 -lifeline 3848 -checkpoint 6 -gpu 0 -gpu-vendor ati -tmax=80 -twait=800
14:52:10:WU02:FS01:Started FahCore on PID 3140
14:52:10:Started thread 14 on PID 3848
14:52:11:WU02:FS01:Core PID:2388
14:52:11:WU02:FS01:FahCore 0x21 started
14:52:11:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
14:52:11:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:9704 run:39 clone:5 gen:102 core:0x21 unit:0x00000093ab404162553ec1b29935fab2
14:52:11:WU00:FS01:Uploading 14.99MiB to 171.64.65.98
14:52:11:WU00:FS01:Connecting to 171.64.65.98:8080
14:52:12:WU02:FS01:0x21:*********************** Log Started 2015-11-17T14:52:11Z ***********************
14:52:12:WU02:FS01:0x21:Project: 9637 (Run 1, Clone 36, Gen 21)
14:52:12:WU02:FS01:0x21:Unit: 0x0000001dab436c9b5609bee310853551
14:52:12:WU02:FS01:0x21:CPU: 0x00000000000000000000000000000000
14:52:12:WU02:FS01:0x21:Machine: 0
14:52:12:WU02:FS01:0x21:Digital signatures verified
14:52:12:WU02:FS01:0x21:Folding@home GPU Core21 Folding@home Core
14:52:12:WU02:FS01:0x21:Version 0.0.12
14:52:12:WU02:FS01:0x21:  Found a checkpoint file
14:52:19:WU00:FS01:Upload 60.88%
14:52:53:WU00:FS01:Upload complete
14:52:53:WU00:FS01:Server responded WORK_ACK (400)
14:52:53:WU00:FS01:Final credit estimate, 40144.00 points
14:52:54:WU00:FS01:Cleaning up

17:32:34:WU00:FS01:Connecting to 171.67.108.45:80
17:32:35:WU00:FS01:Assigned to work server 140.163.4.245
17:32:35:WU00:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:Hawaii [Radeon R9 200/300X Series] from 140.163.4.245
17:32:35:WU00:FS01:Connecting to 140.163.4.245:8080
17:32:36:WU00:FS01:Downloading 15.52MiB
17:32:42:WU00:FS01:Download 43.49%
17:32:47:WU00:FS01:Download complete
17:32:47:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:10496 run:85 clone:1 gen:0 core:0x21 unit:0x000000008ca304f555d121437eb57f18
17:34:43:WU00:FS01:Starting
17:34:43:WU00:FS01:Running FahCore: C:\NO_UAC\FAHClient7/FAHCoreWrapper.exe E:/Docbase/FaH-workdir/cores/web.stanford.edu/~pande/Win32/AMD64/ATI/R600/beta/Core_21.fah/FahCore_21.exe -dir 00 -suffix 01 -version 704 -lifeline 3848 -checkpoint 6 -gpu 0 -gpu-vendor ati -tmax=80 -twait=800
17:34:43:WU00:FS01:Started FahCore on PID 4020
17:34:43:Started thread 15 on PID 3848
17:34:43:WU00:FS01:Core PID:3824
17:34:43:WU00:FS01:FahCore 0x21 started
17:34:44:WU00:FS01:0x21:*********************** Log Started 2015-11-17T17:34:43Z ***********************
17:34:44:WU00:FS01:0x21:Project: 10496 (Run 85, Clone 1, Gen 0)
17:34:44:WU00:FS01:0x21:Unit: 0x000000008ca304f555d121437eb57f18
17:34:44:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
17:34:44:WU00:FS01:0x21:Machine: 0
17:34:44:WU00:FS01:0x21:Reading tar file core.xml
17:34:44:WU00:FS01:0x21:Reading tar file system.xml
17:34:45:WU00:FS01:0x21:Reading tar file integrator.xml
17:34:45:WU00:FS01:0x21:Reading tar file state.xml
17:34:47:WU00:FS01:0x21:Digital signatures verified
17:34:47:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
17:34:47:WU00:FS01:0x21:Version 0.0.12
17:34:49:WU02:FS01:Upload 42.95%
17:34:55:WU02:FS01:Upload complete
17:34:56:WU02:FS01:Server responded WORK_ACK (400)
17:34:56:WU02:FS01:Final credit estimate, 27274.00 points
17:34:56:WU02:FS01:Cleaning up
17:35:46:WU00:FS01:0x21:Completed 0 out of 2000000 steps (0%)
17:35:47:WU00:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
17:39:15:WU00:FS01:0x21:Completed 20000 out of 2000000 steps (1%)
17:42:26:WU00:FS01:0x21:Completed 40000 out of 2000000 steps (2%)
17:45:39:WU00:FS01:0x21:Completed 60000 out of 2000000 steps (3%)
17:48:49:WU00:FS01:0x21:Completed 80000 out of 2000000 steps (4%)
17:52:08:WU00:FS01:0x21:Completed 100000 out of 2000000 steps (5%)
17:55:26:WU00:FS01:0x21:Completed 120000 out of 2000000 steps (6%)
17:56:14:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
17:59:27:WU00:FS01:0x21:Completed 20000 out of 2000000 steps (1%)
18:02:39:WU00:FS01:0x21:Completed 40000 out of 2000000 steps (2%)
18:05:53:WU00:FS01:0x21:Completed 60000 out of 2000000 steps (3%)
18:09:09:WU00:FS01:0x21:Completed 80000 out of 2000000 steps (4%)
18:12:24:WU00:FS01:0x21:Completed 100000 out of 2000000 steps (5%)
18:15:37:WU00:FS01:0x21:Completed 120000 out of 2000000 steps (6%)
18:19:13:WU00:FS01:0x21:Completed 140000 out of 2000000 steps (7%)
18:22:29:WU00:FS01:0x21:Completed 160000 out of 2000000 steps (8%)
18:25:44:WU00:FS01:0x21:Completed 180000 out of 2000000 steps (9%)
18:29:01:WU00:FS01:0x21:Completed 200000 out of 2000000 steps (10%)
18:32:19:WU00:FS01:0x21:Completed 220000 out of 2000000 steps (11%)
18:35:36:WU00:FS01:0x21:Completed 240000 out of 2000000 steps (12%)
18:37:50:Started thread 16 on PID 3848
18:39:14:WU00:FS01:0x21:Completed 260000 out of 2000000 steps (13%)
18:42:30:WU00:FS01:0x21:Completed 280000 out of 2000000 steps (14%)
18:45:47:WU00:FS01:0x21:Completed 300000 out of 2000000 steps (15%)
18:49:03:WU00:FS01:0x21:Completed 320000 out of 2000000 steps (16%)
18:52:18:WU00:FS01:0x21:Completed 340000 out of 2000000 steps (17%)
18:55:34:WU00:FS01:0x21:Completed 360000 out of 2000000 steps (18%)
18:59:12:WU00:FS01:0x21:Completed 380000 out of 2000000 steps (19%)
19:02:27:WU00:FS01:0x21:Completed 400000 out of 2000000 steps (20%)
19:05:43:WU00:FS01:0x21:Completed 420000 out of 2000000 steps (21%)
19:08:59:WU00:FS01:0x21:Completed 440000 out of 2000000 steps (22%)
19:12:17:WU00:FS01:0x21:Completed 460000 out of 2000000 steps (23%)
19:15:34:WU00:FS01:0x21:Completed 480000 out of 2000000 steps (24%)
19:18:48:WU00:FS01:0x21:Completed 500000 out of 2000000 steps (25%)
19:18:48:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
19:19:37:WU00:FS01:0x21:Completed 380000 out of 2000000 steps (19%)
19:22:56:WU00:FS01:0x21:Completed 400000 out of 2000000 steps (20%)
******************************* Date: 2015-11-17 *******************************
19:26:11:WU00:FS01:0x21:Completed 420000 out of 2000000 steps (21%)
19:29:25:WU00:FS01:0x21:Completed 440000 out of 2000000 steps (22%)
19:32:42:WU00:FS01:0x21:Completed 460000 out of 2000000 steps (23%)
19:35:57:WU00:FS01:0x21:Completed 480000 out of 2000000 steps (24%)
19:39:15:WU00:FS01:0x21:Completed 500000 out of 2000000 steps (25%)
19:42:56:WU00:FS01:0x21:Completed 520000 out of 2000000 steps (26%)
19:46:10:WU00:FS01:0x21:Completed 540000 out of 2000000 steps (27%)
19:49:22:WU00:FS01:0x21:Completed 560000 out of 2000000 steps (28%)
19:52:32:WU00:FS01:0x21:Completed 580000 out of 2000000 steps (29%)
19:55:41:WU00:FS01:0x21:Completed 600000 out of 2000000 steps (30%)
19:58:53:WU00:FS01:0x21:Completed 620000 out of 2000000 steps (31%)
20:02:27:WU00:FS01:0x21:Completed 640000 out of 2000000 steps (32%)
20:05:42:WU00:FS01:0x21:Completed 660000 out of 2000000 steps (33%)
20:08:55:WU00:FS01:0x21:Completed 680000 out of 2000000 steps (34%)
20:12:09:WU00:FS01:0x21:Completed 700000 out of 2000000 steps (35%)
20:15:25:WU00:FS01:0x21:Completed 720000 out of 2000000 steps (36%)
20:18:38:WU00:FS01:0x21:Completed 740000 out of 2000000 steps (37%)
20:22:17:WU00:FS01:0x21:Completed 760000 out of 2000000 steps (38%)
20:25:37:WU00:FS01:0x21:Completed 780000 out of 2000000 steps (39%)
20:28:55:WU00:FS01:0x21:Completed 800000 out of 2000000 steps (40%)
20:32:15:WU00:FS01:0x21:Completed 820000 out of 2000000 steps (41%)
20:35:39:WU00:FS01:0x21:Completed 840000 out of 2000000 steps (42%)
20:36:11:Started thread 17 on PID 3848
20:38:58:WU00:FS01:0x21:Completed 860000 out of 2000000 steps (43%)
20:42:40:WU00:FS01:0x21:Completed 880000 out of 2000000 steps (44%)
20:45:01:Started thread 18 on PID 3848
20:46:03:WU00:FS01:0x21:Completed 900000 out of 2000000 steps (45%)
20:49:26:WU00:FS01:0x21:Completed 920000 out of 2000000 steps (46%)
20:52:45:WU00:FS01:0x21:Completed 940000 out of 2000000 steps (47%)
20:56:03:WU00:FS01:0x21:Completed 960000 out of 2000000 steps (48%)
20:59:22:WU00:FS01:0x21:Completed 980000 out of 2000000 steps (49%)
21:02:39:WU00:FS01:0x21:Completed 1000000 out of 2000000 steps (50%)
21:06:24:WU00:FS01:0x21:Completed 1020000 out of 2000000 steps (51%)
21:09:42:WU00:FS01:0x21:Completed 1040000 out of 2000000 steps (52%)
21:13:04:WU00:FS01:0x21:Completed 1060000 out of 2000000 steps (53%)
21:16:28:WU00:FS01:0x21:Completed 1080000 out of 2000000 steps (54%)
21:19:46:WU00:FS01:0x21:Completed 1100000 out of 2000000 steps (55%)
21:23:05:WU00:FS01:0x21:Completed 1120000 out of 2000000 steps (56%)
21:26:56:WU00:FS01:0x21:Completed 1140000 out of 2000000 steps (57%)
21:30:17:WU00:FS01:0x21:Completed 1160000 out of 2000000 steps (58%)
21:33:37:WU00:FS01:0x21:Completed 1180000 out of 2000000 steps (59%)
21:36:56:WU00:FS01:0x21:Completed 1200000 out of 2000000 steps (60%)
21:40:10:WU00:FS01:0x21:Completed 1220000 out of 2000000 steps (61%)
21:43:22:WU00:FS01:0x21:Completed 1240000 out of 2000000 steps (62%)
21:48:07:WU00:FS01:0x21:Completed 1260000 out of 2000000 steps (63%)
21:51:20:WU00:FS01:0x21:Completed 1280000 out of 2000000 steps (64%)
21:54:35:WU00:FS01:0x21:Completed 1300000 out of 2000000 steps (65%)
21:57:47:WU00:FS01:0x21:Completed 1320000 out of 2000000 steps (66%)
22:01:01:WU00:FS01:0x21:Completed 1340000 out of 2000000 steps (67%)
22:04:16:WU00:FS01:0x21:Completed 1360000 out of 2000000 steps (68%)
22:06:43:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
22:06:44:WU00:FS01:0x21:Max number of retries reached. Aborting.
22:06:45:WU00:FS01:0x21:ERROR:Max Retries Reached
22:06:45:WU00:FS01:0x21:Saving result file logfile_01.txt
22:06:45:WU00:FS01:0x21:Saving result file log.txt
22:06:45:WU00:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
22:07:00:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
22:07:00:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:10496 run:85 clone:1 gen:0 core:0x21 unit:0x000000008ca304f555d121437eb57f18

[bruce]

... Maybe programmers upgrade 21 core to more stable version somewhen?.. The only thing left for contributors at today is to downclock & overvltage our working horses

That's pretty much it. The programmers in Development are working very hard on a more stable version. The only prediction we can make is "soon™"

[Kebast]

Here's another:

Code: Select all

17:44:26:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
17:50:25:WU00:FS01:0x21:Completed 1725000 out of 2500000 steps (69%)
17:56:24:WU00:FS01:0x21:Completed 1750000 out of 2500000 steps (70%)
18:02:23:WU00:FS01:0x21:Completed 1775000 out of 2500000 steps (71%)
18:08:22:WU00:FS01:0x21:Completed 1800000 out of 2500000 steps (72%)
18:08:41:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
18:14:39:WU00:FS01:0x21:Completed 1725000 out of 2500000 steps (69%)
18:20:38:WU00:FS01:0x21:Completed 1750000 out of 2500000 steps (70%)
18:26:37:WU00:FS01:0x21:Completed 1775000 out of 2500000 steps (71%)
18:32:36:WU00:FS01:0x21:Completed 1800000 out of 2500000 steps (72%)
18:38:57:WU00:FS01:0x21:Completed 1825000 out of 2500000 steps (73%)
18:44:55:WU00:FS01:0x21:Completed 1850000 out of 2500000 steps (74%)
18:50:54:WU00:FS01:0x21:Completed 1875000 out of 2500000 steps (75%)
18:56:53:WU00:FS01:0x21:Completed 1900000 out of 2500000 steps (76%)
18:57:11:WU00:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
18:57:11:WU00:FS01:0x21:Max number of retries reached. Aborting.
18:57:11:WU00:FS01:0x21:ERROR:Max Retries Reached
18:57:11:WU00:FS01:0x21:Saving result file logfile_01.txt
18:57:11:WU00:FS01:0x21:Saving result file log.txt
18:57:11:WU00:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
18:57:12:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
18:57:12:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9206 run:0 clone:2081 gen:8 core:0x21 unit:0x00000028664f2dd0563820499bf1791b
18:57:12:WU00:FS01:Uploading 3.19KiB to 171.64.65.104
18:57:12:WU00:FS01:Connecting to 171.64.65.104:8080
18:57:13:WU00:FS01:Upload complete

[Duce H_K_]

bruce I didn't know what you told me in pm, thanks.
9206 seems like non-beta project and it still unstable? Last 3 units of 9704 went ok ran on overclocked r9 290X. I thought I could play around with overclock folding 9206 (as well as all projects 92##)

[bruce]

You're posting about an R9 290X in a topic about a GTX970.

[Duce H_K_]

I know but wus' crashes does not depend on card vendor. OK I will not post here any more. Sorry

[bruce]

Actually, it does ... especially right now.

The OpenCL stacks supplied by NVida and by AMD are different and their interface with the drivers is significantly different.

Moreover, the NV {Tesla,Fermi,Kepler,Maxwell1,Maxwell2} have driver differences which can lead to differences in support. The same is true for various subspecies of AMD GPUs.

Whenever possible, FAH attempts to provide the same support and identical projects for all vendors and all subspecies, but it isn't always possible.

(I can remember when CPUs didn't all get identical support, depending on SSE vs. 3DNow! but fortunately that eventually got worked out.)

[Duce H_K_]

Here's another:

Code: Select all

14:04:27:WU00:FS01:0x21:*********************** Log Started 2015-12-04T14:04:26Z ***********************
14:04:27:WU00:FS01:0x21:Project: 11411 (Run 2, Clone 12, Gen 2)
14:04:27:WU00:FS01:0x21:Unit: 0x000000028ca304f1565a241d0f6e295c
14:04:27:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
14:04:27:WU00:FS01:0x21:Machine: 8
14:04:27:WU00:FS01:0x21:Reading tar file core.xml
14:04:27:WU00:FS01:0x21:Reading tar file system.xml
14:04:27:WU00:FS01:0x21:Reading tar file integrator.xml
14:04:27:WU00:FS01:0x21:Reading tar file state.xml
14:04:27:WU00:FS01:0x21:Digital signatures verified
14:04:27:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
14:04:27:WU00:FS01:0x21:Version 0.0.14
14:04:50:WU00:FS01:0x21:Completed 0 out of 5000000 steps (0%)
14:04:50:WU00:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
14:08:09:WU00:FS01:0x21:Completed 50000 out of 5000000 steps (1%)
14:11:22:WU00:FS01:0x21:Completed 100000 out of 5000000 steps (2%)
14:14:43:WU00:FS01:0x21:Completed 150000 out of 5000000 steps (3%)
14:17:56:WU00:FS01:0x21:Completed 200000 out of 5000000 steps (4%)
14:21:12:WU00:FS01:0x21:Completed 250000 out of 5000000 steps (5%)
14:24:32:WU00:FS01:0x21:Completed 300000 out of 5000000 steps (6%)
14:27:46:WU00:FS01:0x21:Completed 350000 out of 5000000 steps (7%)
14:29:37:WU00:FS01:0x21:ERROR:exception: Error invoking kernel clearFourBuffers: clEnqueueNDRangeKernel (-5)
14:29:37:WU00:FS01:0x21:Saving result file logfile_01.txt
14:29:37:WU00:FS01:0x21:Saving result file log.txt
14:29:37:WU00:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
14:29:39:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
14:29:40:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:11411 run:2 clone:12 gen:2 core:0x21 unit:0x000000028ca304f1565a241d0f6e295c
14:29:40:WU00:FS01:Uploading 2.89KiB to 140.163.4.241
14:29:40:WU00:FS01:Connecting to 140.163.4.241:8080
14:29:40:WU00:FS01:Upload complete

Info

Code: Select all

20:52:23:************************* Folding@home Client *************************
20:52:23:      Website: http://folding.stanford.edu/
20:52:23:    Copyright: (c) 2009-2014 Stanford University
20:52:23:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
20:52:23:         Args: --pause-on-start=true
20:52:23:       Config: E:/Docbase/FaH-workdir/config.xml
20:52:23:******************************** Build ********************************
20:52:23:      Version: 7.4.4
20:52:23:         Date: Mar 4 2014
20:52:23:         Time: 20:26:54
20:52:23:      SVN Rev: 4130
20:52:23:       Branch: fah/trunk/client
20:52:23:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
20:52:23:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
20:52:23:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
20:52:23:     Platform: win32 XP
20:52:23:         Bits: 32
20:52:23:         Mode: Release
20:52:23:******************************* System ********************************
20:52:23:          CPU: AMD A8-5600K APU with Radeon(tm) HD Graphics
20:52:23:       CPU ID: AuthenticAMD Family 21 Model 16 Stepping 1
20:52:23:         CPUs: 4
20:52:23:       Memory: 3.71GiB
20:52:23:  Free Memory: 2.73GiB
20:52:23:      Threads: WINDOWS_THREADS
20:52:23:   OS Version: 6.2
20:52:23:  Has Battery: false
20:52:23:   On Battery: false
20:52:23:   UTC Offset: 3
20:52:23:          PID: 1648
20:52:23:          CWD: E:/Docbase/FaH-workdir
20:52:23:           OS: Windows 8.1 Pro
20:52:23:      OS Arch: AMD64
20:52:23:         GPUs: 2
20:52:23:        GPU 0: ATI:5 Trinity [Radeon HD 7560D]
20:52:23:        GPU 1: NVIDIA:5 GM204 [GeForce GTX 970]
20:52:23:         CUDA: 5.2
20:52:23:  CUDA Driver: 7050
20:52:23:Win32 Service: false

It's a pitty to lose a WU with such a big PPD. Everything seemed so good before I got VPU recover & crash

p.s Yep I bought a new GTX970 card

[bruce]

(...and the developers though they had this problem licked with the V0.0.14 version of FAHCore_21.)

What drivers are you running?

How much is the GPU overclocked?

[JohnChodera]

Thanks for the report!

Looking into these failures (since they are ours):

project:10496 run:85 clone:1 gen:0
This failure appeared to use 0.0.12, so we didn't get the extra debug info back.

project:11411 run:2 clone:12 gen:2
This one didn't return debug information either. This seems to be due to the fact that it died with a CL_OUT_OF_RESOURCES error:
Performance since last checkpoint: 44.53608247 ns/day
OpenCL internal error: CL_OUT_OF_RESOURCES error executing CL_COMMAND_NDRANGE_KERNEL on GeForce GTX 970 (Device 0).

Code: Select all

Error invoking kernel reciprocalConvolution: clEnqueueNDRangeKernel (-5)
OpenCL internal error: CL_OUT_OF_RESOURCES error executing CL_COMMAND_NDRANGE_KERNEL on GeForce GTX 970 (Device 0).

ERROR:exception: Error invoking kernel clearFourBuffers: clEnqueueNDRangeKernel (-5)^M

Will report this on the internal bug tracker.

Can you look at the core logs for the other failures? I can't get to those servers.

[toTOW]

This is the TDR issue ... now, is triggered by faulty hardware or by software ?

[Grandpa_01]

1443Mhz boost is pretty high for a 970 if it is running at that I would say that is what I would be looking at.

[JohnChodera]

> This is the TDR issue ... now, is triggered by faulty hardware or by software ?

Are you sure this is the same as the TDR? Project 11411 is not very large, so it would be surprising if one of the GPU kernels was exceeding the windows timeout on a GTX 970.

[Duce H_K_]

What drivers are you running?

How much is the GPU overclocked?

358.87
As Grandpa correct said 1443Mhz vcore. Actually I overclocked memory to it's afaik default 3505MHz in P2 state with NVidia Inspector.
Now 10495 goes pretty OK at the same core version and with even more GPU overclock.
I don't know why mem resets to 3005MHz but I'll leave as is because It must not affect to PPD. If other p114xx come I will look further situation