P9406 - Errors at the beginning of new WU's

Moderators: Site Moderators, FAHC Science Team

Post Reply
DocJonz
Posts: 243
Joined: Thu Dec 06, 2007 6:31 pm
Hardware configuration: Folding with: 4x RTX 4070Ti, 1x RTX 3070
Location: United Kingdom
Contact:
Contact DocJonz

P9406 - Errors at the beginning of new WU's

Post by DocJonz »

I have a number of GTX 780's, all running at stock speed, and all have come up with similar 'bad work unit' errors at the very beginning of new WU's - very similar to ChristianVirtual's issues P9406 thread. Example from today is below.
In terms of temps, the cards all control to 80DegC (+/- 1DegC) - this has not changed since I installed the cards a year ago.

Code: Select all

*********************** Log Started 2014-08-30T21:37:39Z ***********************
21:37:40:WU00:FS01:Starting
21:37:40:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 00 -suffix 01 -version 703 -lifeline 1207 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
21:37:40:WU00:FS01:Started FahCore on PID 1231
21:37:40:WU00:FS01:Core PID:1249
21:37:40:WU00:FS01:FahCore 0x17 started
21:37:42:WU00:FS01:0x17:*********************** Log Started 2014-08-30T21:37:42Z ***********************
21:37:42:WU00:FS01:0x17:Project: 9406 (Run 321, Clone 0, Gen 102)
21:37:42:WU00:FS01:0x17:Unit: 0x000000a90a3b1e5c533e03f670f14c3b
21:37:42:WU00:FS01:0x17:CPU: 0x00000000000000000000000000000000
21:37:42:WU00:FS01:0x17:Machine: 1
21:37:42:WU00:FS01:0x17:Digital signatures verified
21:37:43:WU00:FS01:0x17:  Found a checkpoint file
21:41:39:WU00:FS01:0x17:Completed 200000 out of 2000000 steps (10%)
21:45:23:WU00:FS01:0x17:Completed 220000 out of 2000000 steps (11%)
21:48:56:WU00:FS01:0x17:Completed 240000 out of 2000000 steps (12%)
21:52:36:WU00:FS01:0x17:Completed 260000 out of 2000000 steps (13%)
21:56:08:WU00:FS01:0x17:Completed 280000 out of 2000000 steps (14%)
21:59:41:WU00:FS01:0x17:Completed 300000 out of 2000000 steps (15%)
22:03:20:WU00:FS01:0x17:Completed 320000 out of 2000000 steps (16%)
22:06:52:WU00:FS01:0x17:Completed 340000 out of 2000000 steps (17%)
22:10:31:WU00:FS01:0x17:Completed 360000 out of 2000000 steps (18%)
22:14:04:WU00:FS01:0x17:Completed 380000 out of 2000000 steps (19%)
22:17:36:WU00:FS01:0x17:Completed 400000 out of 2000000 steps (20%)
22:21:15:WU00:FS01:0x17:Completed 420000 out of 2000000 steps (21%)
22:24:47:WU00:FS01:0x17:Completed 440000 out of 2000000 steps (22%)
22:28:27:WU00:FS01:0x17:Completed 460000 out of 2000000 steps (23%)
22:31:59:WU00:FS01:0x17:Completed 480000 out of 2000000 steps (24%)
22:35:32:WU00:FS01:0x17:Completed 500000 out of 2000000 steps (25%)
22:39:11:WU00:FS01:0x17:Completed 520000 out of 2000000 steps (26%)
22:42:44:WU00:FS01:0x17:Completed 540000 out of 2000000 steps (27%)
22:46:24:WU00:FS01:0x17:Completed 560000 out of 2000000 steps (28%)
22:49:56:WU00:FS01:0x17:Completed 580000 out of 2000000 steps (29%)
22:53:29:WU00:FS01:0x17:Completed 600000 out of 2000000 steps (30%)
22:57:08:WU00:FS01:0x17:Completed 620000 out of 2000000 steps (31%)
23:00:41:WU00:FS01:0x17:Completed 640000 out of 2000000 steps (32%)
23:04:20:WU00:FS01:0x17:Completed 660000 out of 2000000 steps (33%)
23:07:53:WU00:FS01:0x17:Completed 680000 out of 2000000 steps (34%)
23:11:25:WU00:FS01:0x17:Completed 700000 out of 2000000 steps (35%)
23:15:04:WU00:FS01:0x17:Completed 720000 out of 2000000 steps (36%)
23:18:37:WU00:FS01:0x17:Completed 740000 out of 2000000 steps (37%)
23:22:16:WU00:FS01:0x17:Completed 760000 out of 2000000 steps (38%)
23:25:48:WU00:FS01:0x17:Completed 780000 out of 2000000 steps (39%)
23:29:21:WU00:FS01:0x17:Completed 800000 out of 2000000 steps (40%)
23:33:00:WU00:FS01:0x17:Completed 820000 out of 2000000 steps (41%)
23:36:33:WU00:FS01:0x17:Completed 840000 out of 2000000 steps (42%)
23:40:12:WU00:FS01:0x17:Completed 860000 out of 2000000 steps (43%)
23:43:45:WU00:FS01:0x17:Completed 880000 out of 2000000 steps (44%)
23:47:17:WU00:FS01:0x17:Completed 900000 out of 2000000 steps (45%)
23:50:57:WU00:FS01:0x17:Completed 920000 out of 2000000 steps (46%)
23:54:29:WU00:FS01:0x17:Completed 940000 out of 2000000 steps (47%)
23:58:09:WU00:FS01:0x17:Completed 960000 out of 2000000 steps (48%)
00:01:41:WU00:FS01:0x17:Completed 980000 out of 2000000 steps (49%)
00:05:14:WU00:FS01:0x17:Completed 1000000 out of 2000000 steps (50%)
00:08:53:WU00:FS01:0x17:Completed 1020000 out of 2000000 steps (51%)
00:12:26:WU00:FS01:0x17:Completed 1040000 out of 2000000 steps (52%)
00:16:05:WU00:FS01:0x17:Completed 1060000 out of 2000000 steps (53%)
00:19:37:WU00:FS01:0x17:Completed 1080000 out of 2000000 steps (54%)
00:23:07:WU00:FS01:0x17:Completed 1100000 out of 2000000 steps (55%)
00:26:45:WU00:FS01:0x17:Completed 1120000 out of 2000000 steps (56%)
00:30:16:WU00:FS01:0x17:Completed 1140000 out of 2000000 steps (57%)
00:33:55:WU00:FS01:0x17:Completed 1160000 out of 2000000 steps (58%)
00:37:27:WU00:FS01:0x17:Completed 1180000 out of 2000000 steps (59%)
00:40:59:WU00:FS01:0x17:Completed 1200000 out of 2000000 steps (60%)
00:44:38:WU00:FS01:0x17:Completed 1220000 out of 2000000 steps (61%)
00:48:10:WU00:FS01:0x17:Completed 1240000 out of 2000000 steps (62%)
00:51:48:WU00:FS01:0x17:Completed 1260000 out of 2000000 steps (63%)
00:55:20:WU00:FS01:0x17:Completed 1280000 out of 2000000 steps (64%)
00:58:52:WU00:FS01:0x17:Completed 1300000 out of 2000000 steps (65%)
01:02:30:WU00:FS01:0x17:Completed 1320000 out of 2000000 steps (66%)
01:06:02:WU00:FS01:0x17:Completed 1340000 out of 2000000 steps (67%)
01:09:41:WU00:FS01:0x17:Completed 1360000 out of 2000000 steps (68%)
01:13:12:WU00:FS01:0x17:Completed 1380000 out of 2000000 steps (69%)
01:16:44:WU00:FS01:0x17:Completed 1400000 out of 2000000 steps (70%)
01:20:23:WU00:FS01:0x17:Completed 1420000 out of 2000000 steps (71%)
01:23:55:WU00:FS01:0x17:Completed 1440000 out of 2000000 steps (72%)
01:27:34:WU00:FS01:0x17:Completed 1460000 out of 2000000 steps (73%)
01:31:06:WU00:FS01:0x17:Completed 1480000 out of 2000000 steps (74%)
01:34:37:WU00:FS01:0x17:Completed 1500000 out of 2000000 steps (75%)
01:38:16:WU00:FS01:0x17:Completed 1520000 out of 2000000 steps (76%)
01:41:48:WU00:FS01:0x17:Completed 1540000 out of 2000000 steps (77%)
01:45:26:WU00:FS01:0x17:Completed 1560000 out of 2000000 steps (78%)
01:48:58:WU00:FS01:0x17:Completed 1580000 out of 2000000 steps (79%)
01:52:30:WU00:FS01:0x17:Completed 1600000 out of 2000000 steps (80%)
01:56:08:WU00:FS01:0x17:Completed 1620000 out of 2000000 steps (81%)
01:59:40:WU00:FS01:0x17:Completed 1640000 out of 2000000 steps (82%)
02:03:19:WU00:FS01:0x17:Completed 1660000 out of 2000000 steps (83%)
02:06:51:WU00:FS01:0x17:Completed 1680000 out of 2000000 steps (84%)
02:10:22:WU00:FS01:0x17:Completed 1700000 out of 2000000 steps (85%)
02:14:01:WU00:FS01:0x17:Completed 1720000 out of 2000000 steps (86%)
02:17:33:WU00:FS01:0x17:Completed 1740000 out of 2000000 steps (87%)
02:21:11:WU00:FS01:0x17:Completed 1760000 out of 2000000 steps (88%)
02:24:42:WU00:FS01:0x17:Completed 1780000 out of 2000000 steps (89%)
02:28:14:WU00:FS01:0x17:Completed 1800000 out of 2000000 steps (90%)
02:31:52:WU00:FS01:0x17:Completed 1820000 out of 2000000 steps (91%)
02:35:24:WU00:FS01:0x17:Completed 1840000 out of 2000000 steps (92%)
02:39:02:WU00:FS01:0x17:Completed 1860000 out of 2000000 steps (93%)
02:42:33:WU00:FS01:0x17:Completed 1880000 out of 2000000 steps (94%)
02:46:05:WU00:FS01:0x17:Completed 1900000 out of 2000000 steps (95%)
02:49:43:WU00:FS01:0x17:Completed 1920000 out of 2000000 steps (96%)
02:53:15:WU00:FS01:0x17:Completed 1940000 out of 2000000 steps (97%)
02:56:53:WU00:FS01:0x17:Completed 1960000 out of 2000000 steps (98%)
03:00:24:WU00:FS01:0x17:Completed 1980000 out of 2000000 steps (99%)
03:03:56:WU00:FS01:0x17:Completed 2000000 out of 2000000 steps (100%)
03:03:56:WU02:FS01:Connecting to assign-GPU.stanford.edu:80
03:03:57:WU02:FS01:News: Welcome to Folding@Home
03:03:57:WU02:FS01:Assigned to work server 171.64.65.56
03:03:57:WU02:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GK110 [GeForce GTX 780] from 171.64.65.56
03:03:57:WU02:FS01:Connecting to 171.64.65.56:8080
03:03:57:WU02:FS01:Downloading 4.47MiB
03:03:59:WU02:FS01:Download complete
03:03:59:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:9406 run:725 clone:0 gen:58 core:0x17 unit:0x0000006f0a3b1e5c533e4f94b05dbdc6
03:04:04:WU00:FS01:0x17:Saving result file logfile_01.txt
03:04:04:WU00:FS01:0x17:Saving result file checkpointState.xml
03:04:06:WU00:FS01:0x17:Saving result file checkpt.crc
03:04:06:WU00:FS01:0x17:Saving result file log.txt
03:04:06:WU00:FS01:0x17:Saving result file positions.xtc
03:04:09:WU00:FS01:0x17:Folding@home Core Shutdown: FINISHED_UNIT
03:04:09:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
03:04:09:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:9406 run:321 clone:0 gen:102 core:0x17 unit:0x000000a90a3b1e5c533e03f670f14c3b
03:04:09:WU00:FS01:Uploading 12.95MiB to 171.64.65.56
03:04:09:WU00:FS01:Connecting to 171.64.65.56:8080
03:04:09:WU02:FS01:Starting
03:04:09:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 02 -suffix 01 -version 703 -lifeline 1207 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
03:04:09:WU02:FS01:Started FahCore on PID 3347
03:04:09:WU02:FS01:Core PID:3351
03:04:09:WU02:FS01:FahCore 0x17 started
03:04:10:WU02:FS01:0x17:*********************** Log Started 2014-08-31T03:04:09Z ***********************
03:04:10:WU02:FS01:0x17:Project: 9406 (Run 725, Clone 0, Gen 58)
03:04:10:WU02:FS01:0x17:Unit: 0x0000006f0a3b1e5c533e4f94b05dbdc6
03:04:10:WU02:FS01:0x17:CPU: 0x00000000000000000000000000000000
03:04:10:WU02:FS01:0x17:Machine: 1
03:04:10:WU02:FS01:0x17:Reading tar file state.xml
03:04:10:WU02:FS01:0x17:Reading tar file system.xml
03:04:10:WU02:FS01:0x17:Reading tar file integrator.xml
03:04:10:WU02:FS01:0x17:Reading tar file core.xml
03:04:10:WU02:FS01:0x17:Digital signatures verified
03:04:15:WU00:FS01:Upload 24.14%
03:04:21:WU00:FS01:Upload 64.21%
03:04:36:WU00:FS01:Upload complete
03:04:36:WU00:FS01:Server responded WORK_ACK (400)
03:04:36:WU00:FS01:Final credit estimate, 38178.00 points
03:04:36:WU00:FS01:Cleaning up
03:06:55:WU02:FS01:0x17:ERROR:exception: Potential energy error of 11.2732, threshold of 10
03:06:55:WU02:FS01:0x17:ERROR:Reference Potential Energy: -802524 | Given Potential Energy: -802535
03:06:55:WU02:FS01:0x17:Saving result file logfile_01.txt
03:06:55:WU02:FS01:0x17:Saving result file badStateCheckpoint_583078111
03:06:56:WU02:FS01:0x17:Saving result file badStateForceGroup0_583078111Core.xml
03:06:58:WU02:FS01:0x17:Saving result file badStateForceGroup0_583078111Ref.xml
03:07:00:WU02:FS01:0x17:Saving result file badStateForceGroup1_583078111Core.xml
03:07:03:WU02:FS01:0x17:Saving result file badStateForceGroup1_583078111Ref.xml
03:07:05:WU02:FS01:0x17:Saving result file badStateForceGroup2_583078111Core.xml
03:07:07:WU02:FS01:0x17:Saving result file badStateForceGroup2_583078111Ref.xml
03:07:09:WU02:FS01:0x17:Saving result file log.txt
03:07:09:WU02:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
03:07:09:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:07:09:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:9406 run:725 clone:0 gen:58 core:0x17 unit:0x0000006f0a3b1e5c533e4f94b05dbdc6
03:07:09:WU02:FS01:Uploading 23.02MiB to 171.64.65.56
03:07:09:WU02:FS01:Connecting to 171.64.65.56:8080
03:07:10:WU00:FS01:Connecting to assign-GPU.stanford.edu:80
03:07:10:WU00:FS01:News: Welcome to Folding@Home
03:07:10:WU00:FS01:Assigned to work server 171.64.65.56
03:07:10:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780] from 171.64.65.56
03:07:10:WU00:FS01:Connecting to 171.64.65.56:8080
03:07:11:WU00:FS01:Downloading 5.44MiB
03:07:13:WU00:FS01:Download complete
03:07:13:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9406 run:12 clone:0 gen:127 core:0x17 unit:0x000000b90a3b1e5c533dce00ba72d9f0
03:07:14:WU00:FS01:Starting
03:07:14:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 00 -suffix 01 -version 703 -lifeline 1207 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
03:07:14:WU00:FS01:Started FahCore on PID 3365
03:07:14:WU00:FS01:Core PID:3369
03:07:14:WU00:FS01:FahCore 0x17 started
03:07:14:WU00:FS01:0x17:*********************** Log Started 2014-08-31T03:07:14Z ***********************
03:07:14:WU00:FS01:0x17:Project: 9406 (Run 12, Clone 0, Gen 127)
03:07:14:WU00:FS01:0x17:Unit: 0x000000b90a3b1e5c533dce00ba72d9f0
03:07:14:WU00:FS01:0x17:CPU: 0x00000000000000000000000000000000
03:07:14:WU00:FS01:0x17:Machine: 1
03:07:14:WU00:FS01:0x17:Reading tar file state.xml
03:07:15:WU00:FS01:0x17:Reading tar file system.xml
03:07:15:WU02:FS01:Upload 14.12%
03:07:15:WU00:FS01:0x17:Reading tar file integrator.xml
03:07:15:WU00:FS01:0x17:Reading tar file core.xml
03:07:15:WU00:FS01:0x17:Digital signatures verified
03:07:21:WU02:FS01:Upload 29.87%
03:07:27:WU02:FS01:Upload 72.50%
03:07:32:WU02:FS01:Upload complete
03:07:32:WU02:FS01:Server responded WORK_ACK (400)
03:07:32:WU02:FS01:Cleaning up
03:10:46:WU00:FS01:0x17:Completed 0 out of 2000000 steps (0%)
03:14:57:WU00:FS01:0x17:Completed 20000 out of 2000000 steps (1%)
03:18:57:WU00:FS01:0x17:Completed 40000 out of 2000000 steps (2%)
03:23:04:WU00:FS01:0x17:Completed 60000 out of 2000000 steps (3%)
03:27:03:WU00:FS01:0x17:Completed 80000 out of 2000000 steps (4%)
03:31:01:WU00:FS01:0x17:Completed 100000 out of 2000000 steps (5%)
03:35:06:WU00:FS01:0x17:Completed 120000 out of 2000000 steps (6%)
******************************* Date: 2014-08-31 *******************************
03:39:05:WU00:FS01:0x17:Completed 140000 out of 2000000 steps (7%)
03:43:10:WU00:FS01:0x17:Completed 160000 out of 2000000 steps (8%)
03:47:08:WU00:FS01:0x17:Completed 180000 out of 2000000 steps (9%)
03:51:06:WU00:FS01:0x17:Completed 200000 out of 2000000 steps (10%)
03:55:12:WU00:FS01:0x17:Completed 220000 out of 2000000 steps (11%)
03:59:10:WU00:FS01:0x17:Completed 240000 out of 2000000 steps (12%)
04:03:15:WU00:FS01:0x17:Completed 260000 out of 2000000 steps (13%)
04:07:12:WU00:FS01:0x17:Completed 280000 out of 2000000 steps (14%)
04:11:10:WU00:FS01:0x17:Completed 300000 out of 2000000 steps (15%)
04:15:16:WU00:FS01:0x17:Completed 320000 out of 2000000 steps (16%)
04:19:13:WU00:FS01:0x17:Completed 340000 out of 2000000 steps (17%)
04:23:19:WU00:FS01:0x17:Completed 360000 out of 2000000 steps (18%)
04:27:16:WU00:FS01:0x17:Completed 380000 out of 2000000 steps (19%)
04:31:15:WU00:FS01:0x17:Completed 400000 out of 2000000 steps (20%)
04:35:20:WU00:FS01:0x17:Completed 420000 out of 2000000 steps (21%)
04:39:19:WU00:FS01:0x17:Completed 440000 out of 2000000 steps (22%)
04:43:24:WU00:FS01:0x17:Completed 460000 out of 2000000 steps (23%)
04:47:22:WU00:FS01:0x17:Completed 480000 out of 2000000 steps (24%)
04:51:20:WU00:FS01:0x17:Completed 500000 out of 2000000 steps (25%)
04:55:26:WU00:FS01:0x17:Completed 520000 out of 2000000 steps (26%)
04:59:24:WU00:FS01:0x17:Completed 540000 out of 2000000 steps (27%)
05:03:29:WU00:FS01:0x17:Completed 560000 out of 2000000 steps (28%)
05:07:27:WU00:FS01:0x17:Completed 580000 out of 2000000 steps (29%)
05:11:25:WU00:FS01:0x17:Completed 600000 out of 2000000 steps (30%)
05:15:30:WU00:FS01:0x17:Completed 620000 out of 2000000 steps (31%)
05:19:28:WU00:FS01:0x17:Completed 640000 out of 2000000 steps (32%)
05:23:33:WU00:FS01:0x17:Completed 660000 out of 2000000 steps (33%)
05:27:31:WU00:FS01:0x17:Completed 680000 out of 2000000 steps (34%)
05:31:29:WU00:FS01:0x17:Completed 700000 out of 2000000 steps (35%)
05:35:34:WU00:FS01:0x17:Completed 720000 out of 2000000 steps (36%)
05:39:32:WU00:FS01:0x17:Completed 740000 out of 2000000 steps (37%)
05:43:38:WU00:FS01:0x17:Completed 760000 out of 2000000 steps (38%)
05:47:35:WU00:FS01:0x17:Completed 780000 out of 2000000 steps (39%)
05:51:34:WU00:FS01:0x17:Completed 800000 out of 2000000 steps (40%)
05:55:39:WU00:FS01:0x17:Completed 820000 out of 2000000 steps (41%)
05:59:37:WU00:FS01:0x17:Completed 840000 out of 2000000 steps (42%)
06:03:42:WU00:FS01:0x17:Completed 860000 out of 2000000 steps (43%)
06:07:41:WU00:FS01:0x17:Completed 880000 out of 2000000 steps (44%)
06:11:38:WU00:FS01:0x17:Completed 900000 out of 2000000 steps (45%)
06:15:44:WU00:FS01:0x17:Completed 920000 out of 2000000 steps (46%)
06:19:42:WU00:FS01:0x17:Completed 940000 out of 2000000 steps (47%)
06:23:47:WU00:FS01:0x17:Completed 960000 out of 2000000 steps (48%)
06:27:44:WU00:FS01:0x17:Completed 980000 out of 2000000 steps (49%)
06:31:41:WU00:FS01:0x17:Completed 1000000 out of 2000000 steps (50%)
06:35:45:WU00:FS01:0x17:Completed 1020000 out of 2000000 steps (51%)
06:39:43:WU00:FS01:0x17:Completed 1040000 out of 2000000 steps (52%)
06:43:53:WU00:FS01:0x17:Completed 1060000 out of 2000000 steps (53%)
06:47:54:WU00:FS01:0x17:Completed 1080000 out of 2000000 steps (54%)
06:51:53:WU00:FS01:0x17:Completed 1100000 out of 2000000 steps (55%)
06:56:05:WU00:FS01:0x17:Completed 1120000 out of 2000000 steps (56%)
07:00:04:WU00:FS01:0x17:Completed 1140000 out of 2000000 steps (57%)
07:04:15:WU00:FS01:0x17:Completed 1160000 out of 2000000 steps (58%)
07:08:15:WU00:FS01:0x17:Completed 1180000 out of 2000000 steps (59%)
07:12:13:WU00:FS01:0x17:Completed 1200000 out of 2000000 steps (60%)
07:16:25:WU00:FS01:0x17:Completed 1220000 out of 2000000 steps (61%)
07:20:24:WU00:FS01:0x17:Completed 1240000 out of 2000000 steps (62%)
07:24:35:WU00:FS01:0x17:Completed 1260000 out of 2000000 steps (63%)
07:28:36:WU00:FS01:0x17:Completed 1280000 out of 2000000 steps (64%)
07:32:35:WU00:FS01:0x17:Completed 1300000 out of 2000000 steps (65%)
07:36:47:WU00:FS01:0x17:Completed 1320000 out of 2000000 steps (66%)
07:40:46:WU00:FS01:0x17:Completed 1340000 out of 2000000 steps (67%)
07:44:57:WU00:FS01:0x17:Completed 1360000 out of 2000000 steps (68%)
07:48:58:WU00:FS01:0x17:Completed 1380000 out of 2000000 steps (69%)
07:52:57:WU00:FS01:0x17:Completed 1400000 out of 2000000 steps (70%)
07:57:10:WU00:FS01:0x17:Completed 1420000 out of 2000000 steps (71%)
08:01:10:WU00:FS01:0x17:Completed 1440000 out of 2000000 steps (72%)
08:05:21:WU00:FS01:0x17:Completed 1460000 out of 2000000 steps (73%)
08:09:22:WU00:FS01:0x17:Completed 1480000 out of 2000000 steps (74%)
08:13:21:WU00:FS01:0x17:Completed 1500000 out of 2000000 steps (75%)
08:17:28:WU00:FS01:0x17:Completed 1520000 out of 2000000 steps (76%)
08:21:27:WU00:FS01:0x17:Completed 1540000 out of 2000000 steps (77%)
08:25:35:WU00:FS01:0x17:Completed 1560000 out of 2000000 steps (78%)
08:29:34:WU00:FS01:0x17:Completed 1580000 out of 2000000 steps (79%)
08:33:34:WU00:FS01:0x17:Completed 1600000 out of 2000000 steps (80%)
08:37:41:WU00:FS01:0x17:Completed 1620000 out of 2000000 steps (81%)
08:41:41:WU00:FS01:0x17:Completed 1640000 out of 2000000 steps (82%)
08:45:48:WU00:FS01:0x17:Completed 1660000 out of 2000000 steps (83%)
08:49:48:WU00:FS01:0x17:Completed 1680000 out of 2000000 steps (84%)
08:53:47:WU00:FS01:0x17:Completed 1700000 out of 2000000 steps (85%)
08:57:54:WU00:FS01:0x17:Completed 1720000 out of 2000000 steps (86%)
09:01:54:WU00:FS01:0x17:Completed 1740000 out of 2000000 steps (87%)
09:06:01:WU00:FS01:0x17:Completed 1760000 out of 2000000 steps (88%)
09:10:01:WU00:FS01:0x17:Completed 1780000 out of 2000000 steps (89%)
09:14:00:WU00:FS01:0x17:Completed 1800000 out of 2000000 steps (90%)
09:18:07:WU00:FS01:0x17:Completed 1820000 out of 2000000 steps (91%)
09:22:06:WU00:FS01:0x17:Completed 1840000 out of 2000000 steps (92%)
09:26:13:WU00:FS01:0x17:Completed 1860000 out of 2000000 steps (93%)
09:30:12:WU00:FS01:0x17:Completed 1880000 out of 2000000 steps (94%)
09:34:11:WU00:FS01:0x17:Completed 1900000 out of 2000000 steps (95%)
09:38:18:WU00:FS01:0x17:Completed 1920000 out of 2000000 steps (96%)
******************************* Date: 2014-08-31 *******************************
09:42:18:WU00:FS01:0x17:Completed 1940000 out of 2000000 steps (97%)
09:46:25:WU00:FS01:0x17:Completed 1960000 out of 2000000 steps (98%)
09:50:25:WU00:FS01:0x17:Completed 1980000 out of 2000000 steps (99%)
09:54:25:WU00:FS01:0x17:Completed 2000000 out of 2000000 steps (100%)
09:54:25:WU01:FS01:Connecting to assign-GPU.stanford.edu:80
09:54:26:WU01:FS01:News: Welcome to Folding@Home
09:54:26:WU01:FS01:Assigned to work server 171.64.65.56
09:54:26:WU01:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GK110 [GeForce GTX 780] from 171.64.65.56
09:54:26:WU01:FS01:Connecting to 171.64.65.56:8080
09:54:26:WU01:FS01:Downloading 5.26MiB
09:54:28:WU01:FS01:Download complete
09:54:28:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9406 run:598 clone:0 gen:50 core:0x17 unit:0x0000005d0a3b1e5c533e37ba71ae9db5
09:54:34:WU00:FS01:0x17:Saving result file logfile_01.txt
09:54:34:WU00:FS01:0x17:Saving result file checkpointState.xml
09:54:36:WU00:FS01:0x17:Saving result file checkpt.crc
09:54:36:WU00:FS01:0x17:Saving result file log.txt
09:54:36:WU00:FS01:0x17:Saving result file positions.xtc
09:54:39:WU00:FS01:0x17:Folding@home Core Shutdown: FINISHED_UNIT
09:54:39:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
09:54:40:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:9406 run:12 clone:0 gen:127 core:0x17 unit:0x000000b90a3b1e5c533dce00ba72d9f0
09:54:40:WU00:FS01:Uploading 14.56MiB to 171.64.65.56
09:54:40:WU00:FS01:Connecting to 171.64.65.56:8080
09:54:40:WU01:FS01:Starting
09:54:40:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 01 -suffix 01 -version 703 -lifeline 1207 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
09:54:40:WU01:FS01:Started FahCore on PID 4381
09:54:40:WU01:FS01:Core PID:4385
09:54:40:WU01:FS01:FahCore 0x17 started
09:54:40:WU01:FS01:0x17:*********************** Log Started 2014-08-31T09:54:40Z ***********************
09:54:40:WU01:FS01:0x17:Project: 9406 (Run 598, Clone 0, Gen 50)
09:54:40:WU01:FS01:0x17:Unit: 0x0000005d0a3b1e5c533e37ba71ae9db5
09:54:40:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
09:54:40:WU01:FS01:0x17:Machine: 1
09:54:40:WU01:FS01:0x17:Reading tar file state.xml
09:54:41:WU01:FS01:0x17:Reading tar file system.xml
09:54:41:WU01:FS01:0x17:Reading tar file integrator.xml
09:54:41:WU01:FS01:0x17:Reading tar file core.xml
09:54:41:WU01:FS01:0x17:Digital signatures verified
09:54:46:WU00:FS01:Upload 44.65%
09:54:52:WU00:FS01:Upload 95.73%
09:55:06:WU00:FS01:Upload complete
09:55:06:WU00:FS01:Server responded WORK_ACK (400)
09:55:06:WU00:FS01:Final credit estimate, 36242.00 points
09:55:06:WU00:FS01:Cleaning up
09:57:59:WU01:FS01:0x17:Completed 0 out of 2000000 steps (0%)
10:02:04:WU01:FS01:0x17:Completed 20000 out of 2000000 steps (1%)
10:05:56:WU01:FS01:0x17:Completed 40000 out of 2000000 steps (2%)
10:09:54:WU01:FS01:0x17:Completed 60000 out of 2000000 steps (3%)
10:13:45:WU01:FS01:0x17:Completed 80000 out of 2000000 steps (4%)
10:17:36:WU01:FS01:0x17:Completed 100000 out of 2000000 steps (5%)
10:21:34:WU01:FS01:0x17:Completed 120000 out of 2000000 steps (6%)
10:25:26:WU01:FS01:0x17:Completed 140000 out of 2000000 steps (7%)
10:29:24:WU01:FS01:0x17:Completed 160000 out of 2000000 steps (8%)
10:33:15:WU01:FS01:0x17:Completed 180000 out of 2000000 steps (9%)
10:37:06:WU01:FS01:0x17:Completed 200000 out of 2000000 steps (10%)
10:41:04:WU01:FS01:0x17:Completed 220000 out of 2000000 steps (11%)
10:44:55:WU01:FS01:0x17:Completed 240000 out of 2000000 steps (12%)
10:48:53:WU01:FS01:0x17:Completed 260000 out of 2000000 steps (13%)
10:52:44:WU01:FS01:0x17:Completed 280000 out of 2000000 steps (14%)
10:56:34:WU01:FS01:0x17:Completed 300000 out of 2000000 steps (15%)
11:00:32:WU01:FS01:0x17:Completed 320000 out of 2000000 steps (16%)
11:04:23:WU01:FS01:0x17:Completed 340000 out of 2000000 steps (17%)
11:08:21:WU01:FS01:0x17:Completed 360000 out of 2000000 steps (18%)
11:12:12:WU01:FS01:0x17:Completed 380000 out of 2000000 steps (19%)
11:16:03:WU01:FS01:0x17:Completed 400000 out of 2000000 steps (20%)
11:20:00:WU01:FS01:0x17:Completed 420000 out of 2000000 steps (21%)
11:23:51:WU01:FS01:0x17:Completed 440000 out of 2000000 steps (22%)
11:27:47:WU01:FS01:0x17:Completed 460000 out of 2000000 steps (23%)
11:31:38:WU01:FS01:0x17:Completed 480000 out of 2000000 steps (24%)
11:35:28:WU01:FS01:0x17:Completed 500000 out of 2000000 steps (25%)
11:39:25:WU01:FS01:0x17:Completed 520000 out of 2000000 steps (26%)
11:43:15:WU01:FS01:0x17:Completed 540000 out of 2000000 steps (27%)
11:47:12:WU01:FS01:0x17:Completed 560000 out of 2000000 steps (28%)
11:51:02:WU01:FS01:0x17:Completed 580000 out of 2000000 steps (29%)
11:54:51:WU01:FS01:0x17:Completed 600000 out of 2000000 steps (30%)
11:58:48:WU01:FS01:0x17:Completed 620000 out of 2000000 steps (31%)
12:02:38:WU01:FS01:0x17:Completed 640000 out of 2000000 steps (32%)
12:06:34:WU01:FS01:0x17:Completed 660000 out of 2000000 steps (33%)
12:10:25:WU01:FS01:0x17:Completed 680000 out of 2000000 steps (34%)
12:14:14:WU01:FS01:0x17:Completed 700000 out of 2000000 steps (35%)
12:18:11:WU01:FS01:0x17:Completed 720000 out of 2000000 steps (36%)
12:22:01:WU01:FS01:0x17:Completed 740000 out of 2000000 steps (37%)
12:25:57:WU01:FS01:0x17:Completed 760000 out of 2000000 steps (38%)
12:29:48:WU01:FS01:0x17:Completed 780000 out of 2000000 steps (39%)
12:33:38:WU01:FS01:0x17:Completed 800000 out of 2000000 steps (40%)
12:37:36:WU01:FS01:0x17:Completed 820000 out of 2000000 steps (41%)
12:41:26:WU01:FS01:0x17:Completed 840000 out of 2000000 steps (42%)
12:45:24:WU01:FS01:0x17:Completed 860000 out of 2000000 steps (43%)
12:49:12:WU01:FS01:0x17:Completed 880000 out of 2000000 steps (44%)
12:53:02:WU01:FS01:0x17:Completed 900000 out of 2000000 steps (45%)
12:56:59:WU01:FS01:0x17:Completed 920000 out of 2000000 steps (46%)
13:00:50:WU01:FS01:0x17:Completed 940000 out of 2000000 steps (47%)
13:04:48:WU01:FS01:0x17:Completed 960000 out of 2000000 steps (48%)
13:08:39:WU01:FS01:0x17:Completed 980000 out of 2000000 steps (49%)
13:12:30:WU01:FS01:0x17:Completed 1000000 out of 2000000 steps (50%)
13:16:28:WU01:FS01:0x17:Completed 1020000 out of 2000000 steps (51%)
13:20:19:WU01:FS01:0x17:Completed 1040000 out of 2000000 steps (52%)
13:24:17:WU01:FS01:0x17:Completed 1060000 out of 2000000 steps (53%)
13:28:08:WU01:FS01:0x17:Completed 1080000 out of 2000000 steps (54%)
13:32:00:WU01:FS01:0x17:Completed 1100000 out of 2000000 steps (55%)
13:35:58:WU01:FS01:0x17:Completed 1120000 out of 2000000 steps (56%)
13:39:48:WU01:FS01:0x17:Completed 1140000 out of 2000000 steps (57%)
13:43:44:WU01:FS01:0x17:Completed 1160000 out of 2000000 steps (58%)
13:47:34:WU01:FS01:0x17:Completed 1180000 out of 2000000 steps (59%)
13:51:22:WU01:FS01:0x17:Completed 1200000 out of 2000000 steps (60%)
13:55:17:WU01:FS01:0x17:Completed 1220000 out of 2000000 steps (61%)
13:59:05:WU01:FS01:0x17:Completed 1240000 out of 2000000 steps (62%)
14:03:00:WU01:FS01:0x17:Completed 1260000 out of 2000000 steps (63%)
14:06:49:WU01:FS01:0x17:Completed 1280000 out of 2000000 steps (64%)
14:10:36:WU01:FS01:0x17:Completed 1300000 out of 2000000 steps (65%)
14:14:32:WU01:FS01:0x17:Completed 1320000 out of 2000000 steps (66%)
14:18:19:WU01:FS01:0x17:Completed 1340000 out of 2000000 steps (67%)
14:22:14:WU01:FS01:0x17:Completed 1360000 out of 2000000 steps (68%)
14:26:01:WU01:FS01:0x17:Completed 1380000 out of 2000000 steps (69%)
14:29:49:WU01:FS01:0x17:Completed 1400000 out of 2000000 steps (70%)
14:33:44:WU01:FS01:0x17:Completed 1420000 out of 2000000 steps (71%)
14:37:31:WU01:FS01:0x17:Completed 1440000 out of 2000000 steps (72%)
14:41:25:WU01:FS01:0x17:Completed 1460000 out of 2000000 steps (73%)
14:45:13:WU01:FS01:0x17:Completed 1480000 out of 2000000 steps (74%)
14:49:01:WU01:FS01:0x17:Completed 1500000 out of 2000000 steps (75%)
14:52:57:WU01:FS01:0x17:Completed 1520000 out of 2000000 steps (76%)
14:56:46:WU01:FS01:0x17:Completed 1540000 out of 2000000 steps (77%)
15:00:43:WU01:FS01:0x17:Completed 1560000 out of 2000000 steps (78%)
15:04:32:WU01:FS01:0x17:Completed 1580000 out of 2000000 steps (79%)
15:08:22:WU01:FS01:0x17:Completed 1600000 out of 2000000 steps (80%)
15:12:18:WU01:FS01:0x17:Completed 1620000 out of 2000000 steps (81%)
15:16:08:WU01:FS01:0x17:Completed 1640000 out of 2000000 steps (82%)
15:20:05:WU01:FS01:0x17:Completed 1660000 out of 2000000 steps (83%)
15:23:55:WU01:FS01:0x17:Completed 1680000 out of 2000000 steps (84%)
15:27:45:WU01:FS01:0x17:Completed 1700000 out of 2000000 steps (85%)
15:31:41:WU01:FS01:0x17:Completed 1720000 out of 2000000 steps (86%)
15:35:29:WU01:FS01:0x17:Completed 1740000 out of 2000000 steps (87%)
******************************* Date: 2014-08-31 *******************************
15:39:24:WU01:FS01:0x17:Completed 1760000 out of 2000000 steps (88%)
15:43:11:WU01:FS01:0x17:Completed 1780000 out of 2000000 steps (89%)
15:46:59:WU01:FS01:0x17:Completed 1800000 out of 2000000 steps (90%)
15:50:53:WU01:FS01:0x17:Completed 1820000 out of 2000000 steps (91%)
15:54:41:WU01:FS01:0x17:Completed 1840000 out of 2000000 steps (92%)
15:58:36:WU01:FS01:0x17:Completed 1860000 out of 2000000 steps (93%)
16:02:25:WU01:FS01:0x17:Completed 1880000 out of 2000000 steps (94%)
16:06:14:WU01:FS01:0x17:Completed 1900000 out of 2000000 steps (95%)
16:10:11:WU01:FS01:0x17:Completed 1920000 out of 2000000 steps (96%)
16:14:01:WU01:FS01:0x17:Completed 1940000 out of 2000000 steps (97%)
16:17:57:WU01:FS01:0x17:Completed 1960000 out of 2000000 steps (98%)
16:21:47:WU01:FS01:0x17:Completed 1980000 out of 2000000 steps (99%)
16:25:37:WU01:FS01:0x17:Completed 2000000 out of 2000000 steps (100%)
16:25:38:WU02:FS01:Connecting to assign-GPU.stanford.edu:80
16:25:39:WU02:FS01:News: Welcome to Folding@Home
16:25:39:WU02:FS01:Assigned to work server 171.64.65.56
16:25:39:WU02:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GK110 [GeForce GTX 780] from 171.64.65.56
16:25:39:WU02:FS01:Connecting to 171.64.65.56:8080
16:25:39:WU02:FS01:Downloading 4.64MiB
16:25:42:WU02:FS01:Download complete
16:25:42:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:9406 run:613 clone:0 gen:27 core:0x17 unit:0x0000002b0a3b1e5c533e3a9fb5172e6a
16:25:46:WU01:FS01:0x17:Saving result file logfile_01.txt
16:25:46:WU01:FS01:0x17:Saving result file checkpointState.xml
16:25:48:WU01:FS01:0x17:Saving result file checkpt.crc
16:25:48:WU01:FS01:0x17:Saving result file log.txt
16:25:48:WU01:FS01:0x17:Saving result file positions.xtc
16:25:51:WU01:FS01:0x17:Folding@home Core Shutdown: FINISHED_UNIT
16:25:51:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
16:25:51:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:9406 run:598 clone:0 gen:50 core:0x17 unit:0x0000005d0a3b1e5c533e37ba71ae9db5
16:25:51:WU01:FS01:Uploading 14.05MiB to 171.64.65.56
16:25:51:WU01:FS01:Connecting to 171.64.65.56:8080
16:25:51:WU02:FS01:Starting
16:25:51:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 02 -suffix 01 -version 703 -lifeline 1207 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
16:25:51:WU02:FS01:Started FahCore on PID 5013
16:25:51:WU02:FS01:Core PID:5017
16:25:51:WU02:FS01:FahCore 0x17 started
16:25:52:WU02:FS01:0x17:*********************** Log Started 2014-08-31T16:25:51Z ***********************
16:25:52:WU02:FS01:0x17:Project: 9406 (Run 613, Clone 0, Gen 27)
16:25:52:WU02:FS01:0x17:Unit: 0x0000002b0a3b1e5c533e3a9fb5172e6a
16:25:52:WU02:FS01:0x17:CPU: 0x00000000000000000000000000000000
16:25:52:WU02:FS01:0x17:Machine: 1
16:25:52:WU02:FS01:0x17:Reading tar file state.xml
16:25:52:WU02:FS01:0x17:Reading tar file system.xml
16:25:53:WU02:FS01:0x17:Reading tar file integrator.xml
16:25:53:WU02:FS01:0x17:Reading tar file core.xml
16:25:53:WU02:FS01:0x17:Digital signatures verified
16:25:57:WU01:FS01:Upload 30.69%
16:26:03:WU01:FS01:Upload 70.72%
16:26:19:WU01:FS01:Upload complete
16:26:19:WU01:FS01:Server responded WORK_ACK (400)
16:26:19:WU01:FS01:Final credit estimate, 36977.00 points
16:26:19:WU01:FS01:Cleaning up
16:28:50:WU02:FS01:0x17:ERROR:exception: Potential energy error of 20.4059, threshold of 10
16:28:50:WU02:FS01:0x17:ERROR:Reference Potential Energy: -833974 | Given Potential Energy: -833953
16:28:50:WU02:FS01:0x17:Saving result file logfile_01.txt
16:28:50:WU02:FS01:0x17:Saving result file badStateCheckpoint_1162962435
16:28:52:WU02:FS01:0x17:Saving result file badStateForceGroup0_1162962435Core.xml
16:28:54:WU02:FS01:0x17:Saving result file badStateForceGroup0_1162962435Ref.xml
16:28:56:WU02:FS01:0x17:Saving result file badStateForceGroup1_1162962435Core.xml
16:28:59:WU02:FS01:0x17:Saving result file badStateForceGroup1_1162962435Ref.xml
16:29:01:WU02:FS01:0x17:Saving result file badStateForceGroup2_1162962435Core.xml
16:29:03:WU02:FS01:0x17:Saving result file badStateForceGroup2_1162962435Ref.xml
16:29:05:WU02:FS01:0x17:Saving result file log.txt
16:29:05:WU02:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
16:29:05:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:29:06:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:9406 run:613 clone:0 gen:27 core:0x17 unit:0x0000002b0a3b1e5c533e3a9fb5172e6a
16:29:06:WU02:FS01:Uploading 26.39MiB to 171.64.65.56
16:29:06:WU02:FS01:Connecting to 171.64.65.56:8080
16:29:06:WU01:FS01:Connecting to assign-GPU.stanford.edu:80
16:29:06:WU01:FS01:News: Welcome to Folding@Home
16:29:06:WU01:FS01:Assigned to work server 171.64.65.56
16:29:06:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780] from 171.64.65.56
16:29:06:WU01:FS01:Connecting to 171.64.65.56:8080
16:29:07:WU01:FS01:Downloading 5.15MiB
16:29:10:WU01:FS01:Download complete
16:29:10:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9406 run:222 clone:0 gen:108 core:0x17 unit:0x000000b10a3b1e5c533df1ebbe742ce6
16:29:10:WU01:FS01:Starting
16:29:10:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 01 -suffix 01 -version 703 -lifeline 1207 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
16:29:10:WU01:FS01:Started FahCore on PID 5032
16:29:10:WU01:FS01:Core PID:5036
16:29:10:WU01:FS01:FahCore 0x17 started
16:29:10:WU01:FS01:0x17:*********************** Log Started 2014-08-31T16:29:10Z ***********************
16:29:10:WU01:FS01:0x17:Project: 9406 (Run 222, Clone 0, Gen 108)
16:29:10:WU01:FS01:0x17:Unit: 0x000000b10a3b1e5c533df1ebbe742ce6
16:29:10:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
16:29:10:WU01:FS01:0x17:Machine: 1
16:29:10:WU01:FS01:0x17:Reading tar file state.xml
16:29:11:WU01:FS01:0x17:Reading tar file system.xml
16:29:11:WU01:FS01:0x17:Reading tar file integrator.xml
16:29:11:WU01:FS01:0x17:Reading tar file core.xml
16:29:11:WU01:FS01:0x17:Digital signatures verified
16:29:12:WU02:FS01:Upload 15.16%
16:29:18:WU02:FS01:Upload 32.21%
16:29:24:WU02:FS01:Upload 69.64%
16:29:30:WU02:FS01:Upload complete
16:29:30:WU02:FS01:Server responded WORK_ACK (400)
16:29:30:WU02:FS01:Cleaning up
16:32:49:WU01:FS01:0x17:ERROR:exception: Potential energy error of 33.3511, threshold of 10
16:32:49:WU01:FS01:0x17:ERROR:Reference Potential Energy: -931291 | Given Potential Energy: -931324
16:32:49:WU01:FS01:0x17:Saving result file logfile_01.txt
16:32:49:WU01:FS01:0x17:Saving result file badStateCheckpoint_1587761961
16:32:50:WU01:FS01:0x17:Saving result file badStateForceGroup0_1587761961Core.xml
16:32:52:WU01:FS01:0x17:Saving result file badStateForceGroup0_1587761961Ref.xml
16:32:55:WU01:FS01:0x17:Saving result file badStateForceGroup1_1587761961Core.xml
16:32:58:WU01:FS01:0x17:Saving result file badStateForceGroup1_1587761961Ref.xml
16:33:00:WU01:FS01:0x17:Saving result file badStateForceGroup2_1587761961Core.xml
16:33:02:WU01:FS01:0x17:Saving result file badStateForceGroup2_1587761961Ref.xml
16:33:05:WU01:FS01:0x17:Saving result file log.txt
16:33:05:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
16:33:05:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:33:05:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9406 run:222 clone:0 gen:108 core:0x17 unit:0x000000b10a3b1e5c533df1ebbe742ce6
16:33:05:WU01:FS01:Uploading 26.80MiB to 171.64.65.56
16:33:05:WU01:FS01:Connecting to 171.64.65.56:8080
16:33:06:WU02:FS01:Connecting to assign-GPU.stanford.edu:80
16:33:06:WU02:FS01:News: Welcome to Folding@Home
16:33:06:WU02:FS01:Assigned to work server 171.64.65.56
16:33:06:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780] from 171.64.65.56
16:33:06:WU02:FS01:Connecting to 171.64.65.56:8080
16:33:07:WU02:FS01:Downloading 5.48MiB
16:33:09:WU02:FS01:Download complete
16:33:09:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:9406 run:190 clone:0 gen:67 core:0x17 unit:0x000000730a3b1e5c533debf2d57e061c
16:33:09:WU02:FS01:Starting
16:33:09:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 02 -suffix 01 -version 703 -lifeline 1207 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
16:33:09:WU02:FS01:Started FahCore on PID 5052
16:33:09:WU02:FS01:Core PID:5056
16:33:09:WU02:FS01:FahCore 0x17 started
16:33:10:WU02:FS01:0x17:*********************** Log Started 2014-08-31T16:33:09Z ***********************
16:33:10:WU02:FS01:0x17:Project: 9406 (Run 190, Clone 0, Gen 67)
16:33:10:WU02:FS01:0x17:Unit: 0x000000730a3b1e5c533debf2d57e061c
16:33:10:WU02:FS01:0x17:CPU: 0x00000000000000000000000000000000
16:33:10:WU02:FS01:0x17:Machine: 1
16:33:10:WU02:FS01:0x17:Reading tar file state.xml
16:33:10:WU02:FS01:0x17:Reading tar file system.xml
16:33:11:WU02:FS01:0x17:Reading tar file integrator.xml
16:33:11:WU02:FS01:0x17:Reading tar file core.xml
16:33:11:WU02:FS01:0x17:Digital signatures verified
16:33:11:WU01:FS01:Upload 14.69%
16:33:17:WU01:FS01:Upload 26.82%
16:33:23:WU01:FS01:Upload 61.11%
16:33:29:WU01:FS01:Upload 98.90%
16:33:33:WU01:FS01:Upload complete
16:33:33:WU01:FS01:Server responded WORK_ACK (400)
16:33:33:WU01:FS01:Cleaning up
16:36:44:WU02:FS01:0x17:Completed 0 out of 2000000 steps (0%)
16:40:53:WU02:FS01:0x17:Completed 20000 out of 2000000 steps (1%)
16:44:53:WU02:FS01:0x17:Completed 40000 out of 2000000 steps (2%)
16:48:59:WU02:FS01:0x17:Completed 60000 out of 2000000 steps (3%)
16:52:58:WU02:FS01:0x17:Completed 80000 out of 2000000 steps (4%)
16:56:57:WU02:FS01:0x17:Completed 100000 out of 2000000 steps (5%)
17:01:04:WU02:FS01:0x17:Completed 120000 out of 2000000 steps (6%)
17:05:03:WU02:FS01:0x17:Completed 140000 out of 2000000 steps (7%)
17:09:10:WU02:FS01:0x17:Completed 160000 out of 2000000 steps (8%)
17:13:09:WU02:FS01:0x17:Completed 180000 out of 2000000 steps (9%)
17:17:08:WU02:FS01:0x17:Completed 200000 out of 2000000 steps (10%)
17:21:15:WU02:FS01:0x17:Completed 220000 out of 2000000 steps (11%)
17:25:14:WU02:FS01:0x17:Completed 240000 out of 2000000 steps (12%)
17:29:20:WU02:FS01:0x17:Completed 260000 out of 2000000 steps (13%)
17:33:19:WU02:FS01:0x17:Completed 280000 out of 2000000 steps (14%)
17:37:19:WU02:FS01:0x17:Completed 300000 out of 2000000 steps (15%)
17:41:25:WU02:FS01:0x17:Completed 320000 out of 2000000 steps (16%)
17:45:24:WU02:FS01:0x17:Completed 340000 out of 2000000 steps (17%)
17:49:31:WU02:FS01:0x17:Completed 360000 out of 2000000 steps (18%)
17:53:30:WU02:FS01:0x17:Completed 380000 out of 2000000 steps (19%)
17:57:29:WU02:FS01:0x17:Completed 400000 out of 2000000 steps (20%)
18:01:35:WU02:FS01:0x17:Completed 420000 out of 2000000 steps (21%)
18:05:35:WU02:FS01:0x17:Completed 440000 out of 2000000 steps (22%)
18:09:41:WU02:FS01:0x17:Completed 460000 out of 2000000 steps (23%)
18:13:41:WU02:FS01:0x17:Completed 480000 out of 2000000 steps (24%)
18:17:40:WU02:FS01:0x17:Completed 500000 out of 2000000 steps (25%)
18:21:46:WU02:FS01:0x17:Completed 520000 out of 2000000 steps (26%)
18:25:46:WU02:FS01:0x17:Completed 540000 out of 2000000 steps (27%)
18:29:52:WU02:FS01:0x17:Completed 560000 out of 2000000 steps (28%)
18:33:52:WU02:FS01:0x17:Completed 580000 out of 2000000 steps (29%)
18:37:51:WU02:FS01:0x17:Completed 600000 out of 2000000 steps (30%)
18:41:57:WU02:FS01:0x17:Completed 620000 out of 2000000 steps (31%)
18:45:56:WU02:FS01:0x17:Completed 640000 out of 2000000 steps (32%)
18:50:02:WU02:FS01:0x17:Completed 660000 out of 2000000 steps (33%)
18:53:58:WU02:FS01:0x17:Completed 680000 out of 2000000 steps (34%)
18:57:56:WU02:FS01:0x17:Completed 700000 out of 2000000 steps (35%)
19:02:02:WU02:FS01:0x17:Completed 720000 out of 2000000 steps (36%)
19:06:01:WU02:FS01:0x17:Completed 740000 out of 2000000 steps (37%)
19:10:07:WU02:FS01:0x17:Completed 760000 out of 2000000 steps (38%)
19:14:06:WU02:FS01:0x17:Completed 780000 out of 2000000 steps (39%)
19:18:05:WU02:FS01:0x17:Completed 800000 out of 2000000 steps (40%)
19:22:12:WU02:FS01:0x17:Completed 820000 out of 2000000 steps (41%)
19:26:11:WU02:FS01:0x17:Completed 840000 out of 2000000 steps (42%)
19:30:17:WU02:FS01:0x17:Completed 860000 out of 2000000 steps (43%)
19:34:17:WU02:FS01:0x17:Completed 880000 out of 2000000 steps (44%)
19:38:16:WU02:FS01:0x17:Completed 900000 out of 2000000 steps (45%)
19:42:23:WU02:FS01:0x17:Completed 920000 out of 2000000 steps (46%)
19:46:22:WU02:FS01:0x17:Completed 940000 out of 2000000 steps (47%)
19:50:28:WU02:FS01:0x17:Completed 960000 out of 2000000 steps (48%)
19:54:28:WU02:FS01:0x17:Completed 980000 out of 2000000 steps (49%)
19:58:26:WU02:FS01:0x17:Completed 1000000 out of 2000000 steps (50%)
20:02:33:WU02:FS01:0x17:Completed 1020000 out of 2000000 steps (51%)
20:06:32:WU02:FS01:0x17:Completed 1040000 out of 2000000 steps (52%)
20:10:38:WU02:FS01:0x17:Completed 1060000 out of 2000000 steps (53%)
20:14:36:WU02:FS01:0x17:Completed 1080000 out of 2000000 steps (54%)
20:18:35:WU02:FS01:0x17:Completed 1100000 out of 2000000 steps (55%)
Folding Stats (HFM.NET): DocJonz Folding Farm Stats
Top
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: P9406 - Errors at the beginning of new WU's

Post by bruce »

You and a couple other people returned Project: 9406 (Run 725, Clone 0, Gen 58) quickly with an error. Another person completed it successfully. I have no way to determine how your system differs from the one that completed it successfully other than to start guessing.

A reasonable assumption is that your method of determining GPU stability does not load your GPU as heavily as that particular WU or maybe you happened to run it on an especially hot day.
Top
DocJonz
Posts: 243
Joined: Thu Dec 06, 2007 6:31 pm
Hardware configuration: Folding with: 4x RTX 4070Ti, 1x RTX 3070
Location: United Kingdom
Contact:
Contact DocJonz

Re: P9406 - Errors at the beginning of new WU's

Post by DocJonz »

In regard to stability, I have had systems in the past where there have been issues, and I've also had systems where the GPU has developed a fault - in these cases, they tend to bomb out at varying points in different WU's.
What is unusual in this case, however, is that all the 9406 WU's that fail do so immediately on starting new WU's. Does that not seem strange? I'm not sure what parameters you can track at your end (GPU type, OS, clock speed ....), but it might be worth someone checking if there are some commonalities to the drop-outs a number of people are obviously seeing for this particular WU?

For reference: the machines are running Ubuntu64 14.04 with nVidia 331.38. Since the GTX 780 is designed to self-control to 80DegC, I have not done any mods to this. Room temps are normal for a UK summer, i.e. cool and wet :lol:
Folding Stats (HFM.NET): DocJonz Folding Farm Stats
Top
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: P9406 - Errors at the beginning of new WU's

Post by bruce »

Stanford does not collect clock speed. You're on your own there as their recommendation is that you don't overclock.

I can check OS (which, if different, probably means be a driver difference) but I'm not sure how anybody would use that information. You can't run Linux drivers or Windows or vice versa. We'd just have to wait until the next version of the drivers is distributed.

Note the message:
03:06:55:WU02:FS01:0x17:ERROR:exception: Potential energy error of 11.2732, threshold of 10
03:06:55:WU02:FS01:0x17:ERROR:Reference Potential Energy: -802524 | Given Potential Energy: -802535

You're calculating a PE that's 12% beyond the threshold. That's not a lot, but it's still an error, and that's checked carefully, especially at the beginning of a new WU.
Top
ChristianVirtual
Posts: 1596
Joined: Tue May 28, 2013 12:14 pm
Location: Tokyo

Re: P9406 - Errors at the beginning of new WU's

Post by ChristianVirtual »

But aren't those PE values part of the parameter given with the WU assigned ? Given by the investigator or calculated by earlier generation inside the same run/clone ?

How could a individual GPU influence this simulation parameter ? Is there a simple way to explain/understand ? I don't think we have a problem with the GPUs as all other 9406 before and after got done within the same environmental condition with not OC applied ?
Last edited by ChristianVirtual on Mon Sep 01, 2014 9:54 pm, edited 1 time in total.
ImageImage
Please contribute your logs to http://ppd.fahmm.net
Top
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: P9406 - Errors at the beginning of new WU's

Post by bruce »

Two values are being compared. One is probably obtained from the completion of previous WU, as you suggest, and it's compared to one you calculate, similar to verifying a checksum. (That's independent of verifying the digital signature of the files.)

If your hardware is perfect and the energy value supplied for the previous Gen is inconsistent with the final velocities, the WU will be aborted as "corrupt" but that doesn't explain how one person was able to complete the same WU.

Hi DocJonz (team 35947), (Linux)
Your WU (P9406 R222 C0 G108) was added to the stats database on 2014-08-31 10:04:58 for 0 points of credit.
Days taken to complete WU: 0.02

Hi xxx (team xxxx), (Linux)
Your WU (P9406 R222 C0 G108) was added to the stats database on 2014-08-31 10:04:58 for 0 points of credit.
Days taken to complete WU: 0.02

Hi yyy (team yyyy), (Windows)
Your WU (P9406 R222 C0 G108) was added to the stats database on 2014-09-01 07:03:54 for 20674.5 points of credit.
Days taken to complete WU: 0.89
Top
Post Reply

Return to “Issues with a specific WU”