Project 9412 (74,0,33) Bad State - resumed

[Ricky]

I had a couple Bad States in this unit. Is this common? What is the likely cause of this?

Code: Select all

05:54:47:WU01:FS02:0x21:Completed 588800 out of 640000 steps (92%)
05:58:11:WU01:FS02:0x21:Completed 595200 out of 640000 steps (93%)
06:01:34:WU01:FS02:0x21:Completed 601600 out of 640000 steps (94%)
06:04:57:WU01:FS02:0x21:Completed 608000 out of 640000 steps (95%)
06:08:21:WU01:FS02:0x21:Completed 614400 out of 640000 steps (96%)
06:11:43:WU01:FS02:0x21:Completed 620800 out of 640000 steps (97%)
06:15:06:WU01:FS02:0x21:Completed 627200 out of 640000 steps (98%)
06:18:29:WU01:FS02:0x21:Completed 633600 out of 640000 steps (99%)
06:18:30:WU00:FS02:Connecting to 171.67.108.200:80
06:18:30:WU00:FS02:Assigned to work server 171.64.65.58
06:18:30:WU00:FS02:Requesting new work unit for slot 02: RUNNING gpu:1:GM206 [GeForce GTX 960] from 171.64.65.58
06:18:30:WU00:FS02:Connecting to 171.64.65.58:8080
06:18:31:WU00:FS02:Downloading 884.19KiB
06:18:32:WU00:FS02:Download complete
06:18:32:WU00:FS02:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9412 run:74 clone:0 gen:33 core:0x18 unit:0x00000028ab40413a5535e211c5e3996a
06:21:51:WU01:FS02:0x21:Completed 640000 out of 640000 steps (100%)
06:22:46:WU01:FS02:0x21:Saving result file logfile_01.txt
06:22:46:WU01:FS02:0x21:Saving result file checkpointState.xml
06:22:53:WU01:FS02:0x21:Saving result file checkpt.crc
06:22:53:WU01:FS02:0x21:Saving result file log.txt
06:22:53:WU01:FS02:0x21:Saving result file positions.xtc
06:22:55:WU01:FS02:0x21:Folding@home Core Shutdown: FINISHED_UNIT
06:22:56:WU01:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
06:22:56:WU01:FS02:Sending unit results: id:01 state:SEND error:NO_ERROR project:9704 run:54 clone:9 gen:1 core:0x21 unit:0x00000002ab404162553ec4274dfe2bce
06:22:56:WU01:FS02:Uploading 14.98MiB to 171.64.65.98
06:22:56:WU00:FS02:Starting
06:22:56:WU01:FS02:Connecting to 171.64.65.98:8080
06:22:56:WU00:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 00 -suffix 01 -version 704 -lifeline 5656 -checkpoint 15 -gpu 1 -gpu-vendor nvidia
06:22:56:WU00:FS02:Started FahCore on PID 4548
06:22:56:WU00:FS02:Core PID:3064
06:22:56:WU00:FS02:FahCore 0x18 started
06:22:57:WU00:FS02:0x18:*********************** Log Started 2015-08-23T06:22:57Z ***********************
06:22:57:WU00:FS02:0x18:Project: 9412 (Run 74, Clone 0, Gen 33)
06:22:57:WU00:FS02:0x18:Unit: 0x00000028ab40413a5535e211c5e3996a
06:22:57:WU00:FS02:0x18:CPU: 0x00000000000000000000000000000000
06:22:57:WU00:FS02:0x18:Machine: 2
06:22:57:WU00:FS02:0x18:Reading tar file state.xml
06:22:57:WU00:FS02:0x18:Reading tar file system.xml
06:22:57:WU00:FS02:0x18:Reading tar file integrator.xml
06:22:57:WU00:FS02:0x18:Reading tar file core.xml
06:22:57:WU00:FS02:0x18:Digital signatures verified
06:22:57:WU00:FS02:0x18:Folding@home GPU core18
06:22:57:WU00:FS02:0x18:Version 0.0.4
06:23:02:WU01:FS02:Upload 37.54%
06:23:08:WU01:FS02:Upload 92.61%
06:23:09:WU00:FS02:0x18:Completed 0 out of 16000000 steps (0%)
06:23:09:WU00:FS02:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
06:23:40:WU01:FS02:Upload complete
06:23:41:WU01:FS02:Server responded WORK_ACK (400)
06:23:41:WU01:FS02:Final credit estimate, 35283.00 points
06:23:41:WU01:FS02:Cleaning up
06:29:40:WU00:FS02:0x18:Completed 160000 out of 16000000 steps (1%)
06:36:08:WU00:FS02:0x18:Completed 320000 out of 16000000 steps (2%)
06:42:35:WU00:FS02:0x18:Completed 480000 out of 16000000 steps (3%)
06:49:02:WU00:FS02:0x18:Completed 640000 out of 16000000 steps (4%)
06:55:31:WU00:FS02:0x18:Completed 800000 out of 16000000 steps (5%)
07:02:04:WU00:FS02:0x18:Completed 960000 out of 16000000 steps (6%)
07:08:30:WU00:FS02:0x18:Completed 1120000 out of 16000000 steps (7%)
07:14:59:WU00:FS02:0x18:Completed 1280000 out of 16000000 steps (8%)
07:21:24:WU00:FS02:0x18:Completed 1440000 out of 16000000 steps (9%)
07:27:50:WU00:FS02:0x18:Completed 1600000 out of 16000000 steps (10%)
07:34:20:WU00:FS02:0x18:Completed 1760000 out of 16000000 steps (11%)
07:40:45:WU00:FS02:0x18:Completed 1920000 out of 16000000 steps (12%)
07:47:11:WU00:FS02:0x18:Completed 2080000 out of 16000000 steps (13%)
07:53:37:WU00:FS02:0x18:Completed 2240000 out of 16000000 steps (14%)
08:00:02:WU00:FS02:0x18:Completed 2400000 out of 16000000 steps (15%)
08:06:32:WU00:FS02:0x18:Completed 2560000 out of 16000000 steps (16%)
08:12:58:WU00:FS02:0x18:Completed 2720000 out of 16000000 steps (17%)
08:19:24:WU00:FS02:0x18:Completed 2880000 out of 16000000 steps (18%)
08:25:50:WU00:FS02:0x18:Completed 3040000 out of 16000000 steps (19%)
08:32:17:WU00:FS02:0x18:Completed 3200000 out of 16000000 steps (20%)
08:38:47:WU00:FS02:0x18:Completed 3360000 out of 16000000 steps (21%)
08:45:13:WU00:FS02:0x18:Completed 3520000 out of 16000000 steps (22%)
08:51:39:WU00:FS02:0x18:Completed 3680000 out of 16000000 steps (23%)
08:58:04:WU00:FS02:0x18:Completed 3840000 out of 16000000 steps (24%)
09:04:32:WU00:FS02:0x18:Completed 4000000 out of 16000000 steps (25%)
09:11:04:WU00:FS02:0x18:Completed 4160000 out of 16000000 steps (26%)
09:17:30:WU00:FS02:0x18:Completed 4320000 out of 16000000 steps (27%)
09:23:54:WU00:FS02:0x18:Completed 4480000 out of 16000000 steps (28%)
09:30:19:WU00:FS02:0x18:Completed 4640000 out of 16000000 steps (29%)
09:36:44:WU00:FS02:0x18:Completed 4800000 out of 16000000 steps (30%)
09:43:16:WU00:FS02:0x18:Completed 4960000 out of 16000000 steps (31%)
09:49:43:WU00:FS02:0x18:Completed 5120000 out of 16000000 steps (32%)
09:56:12:WU00:FS02:0x18:Completed 5280000 out of 16000000 steps (33%)
10:02:41:WU00:FS02:0x18:Completed 5440000 out of 16000000 steps (34%)
10:09:09:WU00:FS02:0x18:Completed 5600000 out of 16000000 steps (35%)
10:15:42:WU00:FS02:0x18:Completed 5760000 out of 16000000 steps (36%)
10:22:10:WU00:FS02:0x18:Completed 5920000 out of 16000000 steps (37%)
******************************* Date: 2015-08-23 *******************************
10:28:39:WU00:FS02:0x18:Completed 6080000 out of 16000000 steps (38%)
10:35:07:WU00:FS02:0x18:Completed 6240000 out of 16000000 steps (39%)
10:41:35:WU00:FS02:0x18:Completed 6400000 out of 16000000 steps (40%)
10:48:08:WU00:FS02:0x18:Completed 6560000 out of 16000000 steps (41%)
10:54:36:WU00:FS02:0x18:Completed 6720000 out of 16000000 steps (42%)
11:01:04:WU00:FS02:0x18:Completed 6880000 out of 16000000 steps (43%)
11:07:30:WU00:FS02:0x18:Completed 7040000 out of 16000000 steps (44%)
11:13:58:WU00:FS02:0x18:Completed 7200000 out of 16000000 steps (45%)
11:20:32:WU00:FS02:0x18:Completed 7360000 out of 16000000 steps (46%)
11:27:00:WU00:FS02:0x18:Completed 7520000 out of 16000000 steps (47%)
11:33:29:WU00:FS02:0x18:Completed 7680000 out of 16000000 steps (48%)
11:39:58:WU00:FS02:0x18:Completed 7840000 out of 16000000 steps (49%)
11:46:26:WU00:FS02:0x18:Completed 8000000 out of 16000000 steps (50%)
11:52:59:WU00:FS02:0x18:Completed 8160000 out of 16000000 steps (51%)
11:59:27:WU00:FS02:0x18:Completed 8320000 out of 16000000 steps (52%)
12:05:55:WU00:FS02:0x18:Completed 8480000 out of 16000000 steps (53%)
12:12:25:WU00:FS02:0x18:Completed 8640000 out of 16000000 steps (54%)
12:18:56:WU00:FS02:0x18:Completed 8800000 out of 16000000 steps (55%)
12:25:31:WU00:FS02:0x18:Completed 8960000 out of 16000000 steps (56%)
12:32:07:WU00:FS02:0x18:Completed 9120000 out of 16000000 steps (57%)
12:38:42:WU00:FS02:0x18:Completed 9280000 out of 16000000 steps (58%)
12:45:19:WU00:FS02:0x18:Completed 9440000 out of 16000000 steps (59%)
12:51:57:WU00:FS02:0x18:Completed 9600000 out of 16000000 steps (60%)
12:58:40:WU00:FS02:0x18:Completed 9760000 out of 16000000 steps (61%)
13:02:24:WU00:FS02:0x18:Bad State detected... attempting to resume from last good checkpoint
13:06:32:WU00:FS02:0x18:Bad State detected... attempting to resume from last good checkpoint
13:09:24:WU00:FS02:0x18:Completed 9920000 out of 16000000 steps (62%)
13:16:07:WU00:FS02:0x18:Completed 10080000 out of 16000000 steps (63%)
13:22:49:WU00:FS02:0x18:Completed 10240000 out of 16000000 steps (64%)
13:27:23:WU00:FS02:0x18:Bad State detected... attempting to resume from last good checkpoint
13:27:23:WU00:FS02:0x18:Max number of retries reached. Aborting.
13:27:23:WU00:FS02:0x18:ERROR:exception: Max Retries Reached
13:27:23:WU00:FS02:0x18:Saving result file logfile_01.txt
13:27:23:WU00:FS02:0x18:Saving result file log.txt
13:27:23:WU00:FS02:0x18:Folding@home Core Shutdown: BAD_WORK_UNIT
13:27:23:WARNING:WU00:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
13:27:23:WU00:FS02:Sending unit results: id:00 state:SEND error:FAULTY project:9412 run:74 clone:0 gen:33 core:0x18 unit:0x00000028ab40413a5535e211c5e3996a
13:27:23:WU00:FS02:Uploading 3.14KiB to 171.64.65.58
13:27:23:WU00:FS02:Connecting to 171.64.65.58:8080
13:27:24:WU00:FS02:Upload complete
13:27:24:WU00:FS02:Server responded WORK_ACK (400)
13:27:24:WU00:FS02:Cleaning up
13:27:24:WU01:FS02:Connecting to 171.67.108.200:80
13:27:24:WU01:FS02:Assigned to work server 171.64.65.84
13:27:24:WU01:FS02:Requesting new work unit for slot 02: READY gpu:1:GM206 [GeForce GTX 960] from 171.64.65.84
13:27:24:WU01:FS02:Connecting to 171.64.65.84:8080
13:27:25:WU01:FS02:Downloading 3.73MiB
13:27:27:WU01:FS02:Download complete
13:27:27:WU01:FS02:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9119 run:9 clone:1 gen:92 core:0x18 unit:0x0000006d0a3b1e78553e7ef58fa960f6
13:27:27:WU01:FS02:Starting
13:27:27:WU01:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 01 -suffix 01 -version 704 -lifeline 5656 -checkpoint 15 -gpu 1 -gpu-vendor nvidia
13:27:27:WU01:FS02:Started FahCore on PID 3192
13:27:27:WU01:FS02:Core PID:2244
13:27:27:WU01:FS02:FahCore 0x18 started
13:27:28:WU01:FS02:0x18:*********************** Log Started 2015-08-23T13:27:28Z ***********************
13:27:28:WU01:FS02:0x18:Project: 9119 (Run 9, Clone 1, Gen 92)
13:27:28:WU01:FS02:0x18:Unit: 0x0000006d0a3b1e78553e7ef58fa960f6
13:27:28:WU01:FS02:0x18:CPU: 0x00000000000000000000000000000000
13:27:28:WU01:FS02:0x18:Machine: 2
13:27:28:WU01:FS02:0x18:Reading tar file state.xml
13:27:29:WU01:FS02:0x18:Reading tar file system.xml
13:27:30:WU01:FS02:0x18:Reading tar file integrator.xml
13:27:30:WU01:FS02:0x18:Reading tar file core.xml
13:27:30:WU01:FS02:0x18:Digital signatures verified
13:27:30:WU01:FS02:0x18:Folding@home GPU core18
13:27:30:WU01:FS02:0x18:Version 0.0.4
13:28:13:WU01:FS02:0x18:Completed 0 out of 2500000 steps (0%)
13:28:13:WU01:FS02:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
13:31:35:WU01:FS02:0x18:Completed 25000 out of 2500000 steps (1%)

I am just curious if there is a problem that I can remedy. I am running win 8.1 and the GPU is at 55C. The driver is 347.52 from NVIDIA.

Thanks

[bruce]

You can't fix it, but it's good to report it.

Development is working toward a fix but it's proving to be a challenging problem.

In many cases (but not all, obviously) the WU runs to a successful completion after a restart.

[toTOW]

Just to help debug :

How many RAM (main memory, not video one) do you have in the system ?

Is the GPU overclocked ?

Were you using the system when bad states occurred ?

[Ricky]

I have 64G of RAM, this is a server board with 2 processors that each have 32G of RAM. I was only watching the log file when the first bad state fault occurred. There were no other activity running on by me on the system. I have no idea what windows may have been doing in the background. I have updates set to download, but wait for my approval to install. I was writing the message when it failed. I edited the message after I realized that it failed. And I have not set any overclocking, but I have noticed that task manager reports my memory running at 2133 MHz, the processors are suppose to be rated to run 1600/1866.

[Ricky]

I just got another 9412 run (35,0,54). It is on the GTX780. The one that failed was running on a GTX760. I will post if it completes without errors. The 780 runs hotter at 80C.

EDIT: This WU completed with no issues.

[Ricky]

Project: 9412 (Run 133, Clone 3, Gen 34) Failed with bad states. Factory overclocked GTX980FTW, Windows 8.1 OS, NVdia driver 359.00, on new machine that has been running a week. The check point must have been bad somehow. This machine is not running beta yet.

Code: Select all

05:45:18:WU02:FS01:Connecting to 140.163.4.235:8080
05:45:18:WU00:FS01:Starting
05:45:18:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win2/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 00 -suffix 01 -version 704 -lifeline 1000 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
05:45:18:WU00:FS01:Started FahCore on PID 380
05:45:18:WU00:FS01:Core PID:736
05:45:18:WU00:FS01:FahCore 0x18 started
05:45:19:WU00:FS01:0x18:*********************** Log Started 2016-01-02T05:45:18Z ***********************
05:45:19:WU00:FS01:0x18:Project: 9412 (Run 133, Clone 3, Gen 34)
05:45:19:WU00:FS01:0x18:Unit: 0x00000031ab40413a5535e276a8825ab8
05:45:19:WU00:FS01:0x18:CPU: 0x00000000000000000000000000000000
05:45:19:WU00:FS01:0x18:Machine: 1
05:45:19:WU00:FS01:0x18:Reading tar file state.xml
05:45:19:WU00:FS01:0x18:Reading tar file system.xml
05:45:19:WU00:FS01:0x18:Reading tar file integrator.xml
05:45:19:WU00:FS01:0x18:Reading tar file core.xml
05:45:19:WU00:FS01:0x18:Digital signatures verified
05:45:19:WU00:FS01:0x18:Folding@home GPU core18
05:45:19:WU00:FS01:0x18:Version 0.0.4
05:45:24:WU02:FS01:Upload 6.30%
05:45:28:WU00:FS01:0x18:Completed 0 out of 16000000 steps (0%)
05:45:28:WU00:FS01:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
05:45:30:WU02:FS01:Upload 11.97%
05:45:36:WU02:FS01:Upload 17.64%
05:45:43:WU02:FS01:Upload 22.05%
05:45:49:WU02:FS01:Upload 27.09%
05:45:55:WU02:FS01:Upload 32.13%
05:46:01:WU02:FS01:Upload 38.43%
05:46:07:WU02:FS01:Upload 44.73%
05:46:13:WU02:FS01:Upload 49.14%
05:46:19:WU02:FS01:Upload 54.18%
05:46:25:WU02:FS01:Upload 59.85%
05:46:31:WU02:FS01:Upload 65.52%
05:46:37:WU02:FS01:Upload 69.93%
05:46:43:WU02:FS01:Upload 74.34%
05:46:50:WU02:FS01:Upload 80.02%
05:46:56:WU02:FS01:Upload 86.95%
05:47:02:WU02:FS01:Upload 91.36%
05:47:08:WU02:FS01:Upload 95.14%
05:47:21:WU02:FS01:Upload complete
05:47:22:WU02:FS01:Server responded WORK_ACK (400)
05:47:22:WU02:FS01:Final credit estimate, 80122.00 points
05:47:22:WU02:FS01:Cleaning up
05:49:05:WU00:FS01:0x18:Completed 160000 out of 16000000 steps (1%)
05:52:40:WU00:FS01:0x18:Completed 320000 out of 16000000 steps (2%)
05:56:15:WU00:FS01:0x18:Completed 480000 out of 16000000 steps (3%)
05:59:51:WU00:FS01:0x18:Completed 640000 out of 16000000 steps (4%)
06:03:27:WU00:FS01:0x18:Completed 800000 out of 16000000 steps (5%)
06:07:07:WU00:FS01:0x18:Completed 960000 out of 16000000 steps (6%)
06:10:44:WU00:FS01:0x18:Completed 1120000 out of 16000000 steps (7%)
06:14:20:WU00:FS01:0x18:Completed 1280000 out of 16000000 steps (8%)
06:17:49:WU00:FS01:0x18:Completed 1440000 out of 16000000 steps (9%)
06:21:19:WU00:FS01:0x18:Completed 1600000 out of 16000000 steps (10%)
06:24:52:WU00:FS01:0x18:Completed 1760000 out of 16000000 steps (11%)
06:28:24:WU00:FS01:0x18:Completed 1920000 out of 16000000 steps (12%)
06:32:00:WU00:FS01:0x18:Completed 2080000 out of 16000000 steps (13%)
06:35:37:WU00:FS01:0x18:Completed 2240000 out of 16000000 steps (14%)
06:39:14:WU00:FS01:0x18:Completed 2400000 out of 16000000 steps (15%)
06:42:54:WU00:FS01:0x18:Completed 2560000 out of 16000000 steps (16%)
06:46:31:WU00:FS01:0x18:Completed 2720000 out of 16000000 steps (17%)
06:50:07:WU00:FS01:0x18:Completed 2880000 out of 16000000 steps (18%)
06:53:44:WU00:FS01:0x18:Completed 3040000 out of 16000000 steps (19%)
06:57:21:WU00:FS01:0x18:Completed 3200000 out of 16000000 steps (20%)
07:01:01:WU00:FS01:0x18:Completed 3360000 out of 16000000 steps (21%)
07:04:37:WU00:FS01:0x18:Completed 3520000 out of 16000000 steps (22%)
07:08:12:WU00:FS01:0x18:Completed 3680000 out of 16000000 steps (23%)
07:11:47:WU00:FS01:0x18:Completed 3840000 out of 16000000 steps (24%)
07:15:23:WU00:FS01:0x18:Completed 4000000 out of 16000000 steps (25%)
07:19:03:WU00:FS01:0x18:Completed 4160000 out of 16000000 steps (26%)
07:22:39:WU00:FS01:0x18:Completed 4320000 out of 16000000 steps (27%)
07:26:16:WU00:FS01:0x18:Completed 4480000 out of 16000000 steps (28%)
07:29:53:WU00:FS01:0x18:Completed 4640000 out of 16000000 steps (29%)
07:33:30:WU00:FS01:0x18:Completed 4800000 out of 16000000 steps (30%)
07:34:41:WU00:FS01:0x18:Bad State detected... attempting to resume from last good checkpoint
07:35:49:WU00:FS01:0x18:Bad State detected... attempting to resume from last good checkpoint
07:36:56:WU00:FS01:0x18:Bad State detected... attempting to resume from last good checkpoint
07:36:56:WU00:FS01:0x18:Max number of retries reached. Aborting.
07:36:56:WU00:FS01:0x18:ERROR:exception: Max Retries Reached
07:36:56:WU00:FS01:0x18:Saving result file logfile_01.txt
07:36:56:WU00:FS01:0x18:Saving result file log.txt
07:36:56:WU00:FS01:0x18:Folding@home Core Shutdown: BAD_WORK_UNIT
07:36:57:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:36:57:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9412 run:133 clone:3 gen:34 core:0x18 unit:0x00000031ab40413a5535e276a8825ab8
07:36:57:WU00:FS01:Uploading 2.88KiB to 171.64.65.58
07:36:57:WU00:FS01:Connecting to 171.64.65.58:8080
07:36:57:WU00:FS01:Upload complete
07:36:57:WU00:FS01:Server responded WORK_ACK (400)
07:36:57:WU00:FS01:Cleaning up

[toTOW]

Someone else completed it ...