7643: (Run 490, Clone 0, Gen 19) Unstable Machine etc

[rexrzer]

I've been having a terrible time with one specific WU in my PC No.3 desktop FAHome
dedicated PC, running Win7 64-bit SP1 build 7601, and i cannot seem to get rid of the
darn thing, and it has disabled my nVidia GTX 560Ti video card for folding presently.
I have tried all the traditional manners of stopping it, without anything good to report...
it just keeps downloading over and over again, even after repeatedly dismissing the
Work file, Core file, and both Info files within the FAHGPU-1a file itself.

I am without words to describe the frustration of having a perfectly good Fermi GPU
sitting idle because of a bad WU like this. This machine has been folding GPU WU's
for more than one solid year, without any glitches or anomalies up to now, and given
the chance would complete any other WU but things like this quickly, without incidents
of any type, and frankly I am at a loss at to what to do next.

Here is what it looks like in the aggregate from the FAH Log file for almost 2 days worth
of attempts at folding this GPU WU, but now it sits idle because it seems the machine will
only continue downloading this specific WU over and over again, given the chance...

Code: Select all

Launch directory: C:\Users\poweruser\FAHGPU-1
Executable: C:\Users\poweruser\FAHGPU-1\FAHGPU-1a.exe
Arguments: -gpu 0 -verbosity 9 

[05:05:25] - Ask before connecting: No
[05:05:25] - User name: rexrzer (Team 111065)
[05:05:25] - User ID: 71038D4567CAF5F8
[05:05:25] - Machine ID: 11
[05:05:25] 
[05:05:25] Gpu type=3 species=21.
[05:05:25] Loaded queue successfully.
[05:05:25] - Preparing to get new work unit...
[05:05:25] Cleaning up work directory
[05:05:25] - Autosending finished units... [May 10 05:05:25 UTC]
[05:05:25] Trying to send all finished work units
[05:05:25] + No unsent completed units remaining.
[05:05:25] - Autosend completed
[05:05:25] + Attempting to get work packet
[05:05:25] Passkey found
[05:05:25] - Will indicate memory of 4192 MB
[05:05:25] Gpu type=3 species=21.
[05:05:25] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 10, Stepping: 5
[05:05:25] - Connecting to assignment server
[05:05:25] Connecting to http://assign-GPU.stanford.edu:8080/
[05:05:26] Posted data.
[05:05:26] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[05:05:26] + News From Folding@Home: Welcome to Folding@Home
[05:05:26] Loaded queue successfully.
[05:05:26] Gpu type=3 species=21.
[05:05:26] Sent data
[05:05:26] Connecting to http://171.64.65.93:8080/
[05:05:26] Posted data.
[05:05:26] Initial: 0000; - Receiving payload (expected size: 551953)
[05:05:27] - Downloaded at ~539 kB/s
[05:05:27] - Averaged speed for that direction ~503 kB/s
[05:05:27] + Received work.
[05:05:27] + Closed connections
[05:05:27] 
[05:05:27] + Processing work unit
[05:05:27] Core required: FahCore_15.exe
[05:05:27] Core found.
[05:05:27] Working on queue slot 05 [May 10 05:05:27 UTC]
[05:05:27] + Working ...
[05:05:27] - Calling '.\FahCore_15.exe -dir work/ -suffix 05 -nice 19 -checkpoint 15 -verbose -lifeline 4400 -version 641'

[05:05:27] 
[05:05:27] *------------------------------*
[05:05:27] Folding@Home GPU Core
[05:05:27] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[05:05:27] Build host             SimbiosNvdWin7
[05:05:27] Board Type             NVIDIA/CUDA
[05:05:27] Core                   15
[05:05:27] 
[05:05:27] Window's signal control handler registered.
[05:05:27] Preparing to commence simulation
[05:05:27] - Looking at optimizations...
[05:05:27] DeleteFrameFiles: successfully deleted file=work/wudata_05.ckp
[05:05:27] - Created dyn
[05:05:27] - Files status OK
[05:05:27] sizeof(CORE_PACKET_HDR) = 512 file=<>
[05:05:27] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[05:05:27] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[05:05:27] - Digital signature verified
[05:05:27] 
[05:05:27] Project: 7643 (Run 490, Clone 0, Gen 19)
[05:05:27] 
[05:05:27] Assembly optimizations on if available.
[05:05:27] Entering M.D.
[05:05:29] Tpr hash work/wudata_05.tpr:  2722680634 1728328070 1454610611 2632785485 3344963210
[05:05:29] GPU device info: vendor=0 device=0 name=<NA> match=0
[05:05:29] Working on Protein in water
[05:05:29] Client config found, loading data.
[05:05:29] Starting GUI Server
[05:07:09] Setting checkpoint frequency: 25000
[05:07:09] Completed         3 out of 2500000 steps (0%).
[05:16:08] Completed     25000 out of 2500000 steps (1%).
[05:25:06] Completed     50000 out of 2500000 steps (2%).
[05:34:02] Completed     75000 out of 2500000 steps (3%).
[05:52:22] Completed    100000 out of 2500000 steps (4%).
[05:52:23] mdrun_gpu returned 52
[05:52:23] NANs detected on GPU
[05:52:23] 
[05:52:23] Folding@home Core Shutdown: UNSTABLE_MACHINE
[05:52:26] CoreStatus = 7A (122)
[05:52:26] Sending work to server
[05:52:26] Project: 7643 (Run 490, Clone 0, Gen 19)
[05:52:26] - Read packet limit of 540015616... Set to 524286976.
[05:52:26] - Error: Could not get length of results file work/wuresults_05.dat
[05:52:26] - Error: Could not read unit 05 file. Removing from queue.
[05:52:26] Trying to send all finished work units
[05:52:26] + No unsent completed units remaining.
[05:52:26] - Preparing to get new work unit...
[05:52:26] Cleaning up work directory
[05:52:26] + Attempting to get work packet
[05:52:26] Passkey found
[05:52:26] - Will indicate memory of 4192 MB
[05:52:26] Gpu type=3 species=21.
[05:52:26] - Connecting to assignment server
[05:52:26] Connecting to http://assign-GPU.stanford.edu:8080/
[05:52:26] Posted data.
[05:52:26] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[05:52:26] + News From Folding@Home: Welcome to Folding@Home
[05:52:26] Loaded queue successfully.
[05:52:26] Gpu type=3 species=21.
[05:52:26] Sent data
[05:52:26] Connecting to http://171.64.65.93:8080/
[05:52:26] Posted data.
[05:52:26] Initial: 0000; - Receiving payload (expected size: 551953)
[05:52:27] - Downloaded at ~539 kB/s
[05:52:27] - Averaged speed for that direction ~510 kB/s
[05:52:27] + Received work.
[05:52:27] Trying to send all finished work units
[05:52:27] + No unsent completed units remaining.
[05:52:27] + Closed connections
[05:52:32] 
[05:52:32] + Processing work unit
[05:52:32] Core required: FahCore_15.exe
[05:52:32] Core found.
[05:52:32] Working on queue slot 06 [May 10 05:52:32 UTC]
[05:52:32] + Working ...
[05:52:32] - Calling '.\FahCore_15.exe -dir work/ -suffix 06 -nice 19 -checkpoint 15 -verbose -lifeline 4400 -version 641'

[05:52:32] 
[05:52:32] *------------------------------*
[05:52:32] Folding@Home GPU Core
[05:52:32] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[05:52:32] Build host             SimbiosNvdWin7
[05:52:32] Board Type             NVIDIA/CUDA
[05:52:32] Core                   15
[05:52:32] 
[05:52:32] Window's signal control handler registered.
[05:52:32] Preparing to commence simulation
[05:52:32] - Looking at optimizations...
[05:52:32] DeleteFrameFiles: successfully deleted file=work/wudata_06.ckp
[05:52:32] - Created dyn
[05:52:32] - Files status OK
[05:52:32] sizeof(CORE_PACKET_HDR) = 512 file=<>
[05:52:32] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[05:52:32] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[05:52:32] - Digital signature verified
[05:52:32] 
[05:52:32] Project: 7643 (Run 490, Clone 0, Gen 19)
[05:52:32] 
[05:52:32] Assembly optimizations on if available.
[05:52:32] Entering M.D.
[05:52:34] Tpr hash work/wudata_06.tpr:  2722680634 1728328070 1454610611 2632785485 3344963210
[05:52:34] GPU device info: vendor=0 device=0 name=<NA> match=0
[05:52:34] Working on Protein in water
[05:52:34] Client config found, loading data.
[05:52:34] Starting GUI Server
[05:54:14] Setting checkpoint frequency: 25000
[05:54:14] Completed         3 out of 2500000 steps (0%).
[06:21:52] Completed     25000 out of 2500000 steps (1%).
[06:21:52] mdrun_gpu returned 52
[06:21:52] NANs detected on GPU
[06:21:52] 
[06:21:52] Folding@home Core Shutdown: UNSTABLE_MACHINE
[06:21:56] CoreStatus = 7A (122)
[06:21:56] Sending work to server
[06:21:56] Project: 7643 (Run 490, Clone 0, Gen 19)
[06:21:56] - Read packet limit of 540015616... Set to 524286976.
[06:21:56] - Error: Could not get length of results file work/wuresults_06.dat
[06:21:56] - Error: Could not read unit 06 file. Removing from queue.
[06:21:56] Trying to send all finished work units
[06:21:56] + No unsent completed units remaining.
[06:21:56] - Preparing to get new work unit...
[06:21:56] Cleaning up work directory
[06:21:56] + Attempting to get work packet
[06:21:56] Passkey found
[06:21:56] - Will indicate memory of 4192 MB
[06:21:56] Gpu type=3 species=21.
[06:21:56] - Connecting to assignment server
[06:21:56] Connecting to http://assign-GPU.stanford.edu:8080/
[06:21:57] Posted data.
[06:21:57] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[06:21:57] + News From Folding@Home: Welcome to Folding@Home
[06:21:57] Loaded queue successfully.
[06:21:57] Gpu type=3 species=21.
[06:21:57] Sent data
[06:21:57] Connecting to http://171.64.65.93:8080/
[06:21:57] Posted data.
[06:21:57] Initial: 0000; - Receiving payload (expected size: 551953)
[06:21:58] - Downloaded at ~539 kB/s
[06:21:58] - Averaged speed for that direction ~516 kB/s
[06:21:58] + Received work.
[06:21:58] Trying to send all finished work units
[06:21:58] + No unsent completed units remaining.
[06:21:58] + Closed connections
[06:22:03] 
[06:22:03] + Processing work unit
[06:22:03] Core required: FahCore_15.exe
[06:22:03] Core found.
[06:22:03] Working on queue slot 07 [May 10 06:22:03 UTC]
[06:22:03] + Working ...
[06:22:03] - Calling '.\FahCore_15.exe -dir work/ -suffix 07 -nice 19 -checkpoint 15 -verbose -lifeline 4400 -version 641'

[06:22:03] 
[06:22:03] *------------------------------*
[06:22:03] Folding@Home GPU Core
[06:22:03] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[06:22:03] Build host             SimbiosNvdWin7
[06:22:03] Board Type             NVIDIA/CUDA
[06:22:03] Core                   15
[06:22:03] 
[06:22:03] Window's signal control handler registered.
[06:22:03] Preparing to commence simulation
[06:22:03] - Looking at optimizations...
[06:22:03] DeleteFrameFiles: successfully deleted file=work/wudata_07.ckp
[06:22:03] - Created dyn
[06:22:03] - Files status OK
[06:22:03] sizeof(CORE_PACKET_HDR) = 512 file=<>
[06:22:03] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[06:22:03] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[06:22:03] - Digital signature verified
[06:22:03] 
[06:22:03] Project: 7643 (Run 490, Clone 0, Gen 19)
[06:22:03] 
[06:22:03] Assembly optimizations on if available.
[06:22:03] Entering M.D.
[06:22:05] Tpr hash work/wudata_07.tpr:  2722680634 1728328070 1454610611 2632785485 3344963210
[06:22:05] GPU device info: vendor=0 device=0 name=<NA> match=0
[06:22:05] Working on Protein in water
[06:22:05] Client config found, loading data.
[06:22:05] Starting GUI Server
[06:23:45] Setting checkpoint frequency: 25000
[06:23:45] Completed         3 out of 2500000 steps (0%).
[06:32:44] Completed     25000 out of 2500000 steps (1%).
[06:58:14] Completed     50000 out of 2500000 steps (2%).
[06:58:14] mdrun_gpu returned 52
[06:58:14] NANs detected on GPU
[06:58:14] 
[06:58:14] Folding@home Core Shutdown: UNSTABLE_MACHINE
[06:58:17] CoreStatus = 7A (122)
[06:58:17] Sending work to server
[06:58:17] Project: 7643 (Run 490, Clone 0, Gen 19)
[06:58:17] - Read packet limit of 540015616... Set to 524286976.
[06:58:17] - Error: Could not get length of results file work/wuresults_07.dat
[06:58:17] - Error: Could not read unit 07 file. Removing from queue.
[06:58:17] Trying to send all finished work units
[06:58:17] + No unsent completed units remaining.
[06:58:17] - Preparing to get new work unit...
[06:58:17] Cleaning up work directory
[06:58:17] + Attempting to get work packet
[06:58:17] Passkey found
[06:58:17] - Will indicate memory of 4192 MB
[06:58:17] Gpu type=3 species=21.
[06:58:17] - Connecting to assignment server
[06:58:17] Connecting to http://assign-GPU.stanford.edu:8080/
[06:58:18] Posted data.
[06:58:18] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[06:58:18] + News From Folding@Home: Welcome to Folding@Home
[06:58:18] Loaded queue successfully.
[06:58:18] Gpu type=3 species=21.
[06:58:18] Sent data
[06:58:18] Connecting to http://171.64.65.93:8080/
[06:58:18] Posted data.
[06:58:18] Initial: 0000; - Receiving payload (expected size: 551953)
[06:58:18] Conversation time very short, giving reduced weight in bandwidth avg
[06:58:18] - Downloaded at ~1078 kB/s
[06:58:18] - Averaged speed for that direction ~578 kB/s
[06:58:18] + Received work.
[06:58:18] Trying to send all finished work units
[06:58:18] + No unsent completed units remaining.
[06:58:18] + Closed connections
[06:58:23] 
[06:58:23] + Processing work unit
[06:58:23] Core required: FahCore_15.exe
[06:58:23] Core found.
[06:58:23] Working on queue slot 08 [May 10 06:58:23 UTC]
[06:58:23] + Working ...
[06:58:23] - Calling '.\FahCore_15.exe -dir work/ -suffix 08 -nice 19 -checkpoint 15 -verbose -lifeline 4400 -version 641'

[06:58:23] 
[06:58:23] *------------------------------*
[06:58:23] Folding@Home GPU Core
[06:58:23] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[06:58:23] Build host             SimbiosNvdWin7
[06:58:23] Board Type             NVIDIA/CUDA
[06:58:23] Core                   15
[06:58:23] 
[06:58:23] Window's signal control handler registered.
[06:58:23] Preparing to commence simulation
[06:58:23] - Looking at optimizations...
[06:58:23] DeleteFrameFiles: successfully deleted file=work/wudata_08.ckp
[06:58:23] - Created dyn
[06:58:23] - Files status OK
[06:58:23] sizeof(CORE_PACKET_HDR) = 512 file=<>
[06:58:23] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[06:58:23] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[06:58:23] - Digital signature verified
[06:58:23] 
[06:58:23] Project: 7643 (Run 490, Clone 0, Gen 19)
[06:58:23] 
[06:58:23] Assembly optimizations on if available.
[06:58:23] Entering M.D.
[06:58:25] Tpr hash work/wudata_08.tpr:  2722680634 1728328070 1454610611 2632785485 3344963210
[06:58:25] GPU device info: vendor=0 device=0 name=<NA> match=0
[06:58:26] Working on Protein in water
[06:58:26] Client config found, loading data.
[06:58:26] Starting GUI Server
[07:00:04] Setting checkpoint frequency: 25000
[07:00:04] Completed         3 out of 2500000 steps (0%).
[07:08:58] Completed     25000 out of 2500000 steps (1%).
[07:17:54] Completed     50000 out of 2500000 steps (2%).
[07:45:04] Completed     75000 out of 2500000 steps (3%).
[07:45:05] mdrun_gpu returned 52
[07:45:05] NANs detected on GPU
[07:45:05] 
[07:45:05] Folding@home Core Shutdown: UNSTABLE_MACHINE
[07:45:08] CoreStatus = 7A (122)
[07:45:08] Sending work to server
[07:45:08] Project: 7643 (Run 490, Clone 0, Gen 19)
[07:45:08] - Read packet limit of 540015616... Set to 524286976.
[07:45:08] - Error: Could not get length of results file work/wuresults_08.dat
[07:45:08] - Error: Could not read unit 08 file. Removing from queue.
[07:45:08] Trying to send all finished work units
[07:45:08] + No unsent completed units remaining.
[07:45:08] - Preparing to get new work unit...
[07:45:08] Cleaning up work directory
[07:45:08] + Attempting to get work packet
[07:45:08] Passkey found
[07:45:08] - Will indicate memory of 4192 MB
[07:45:08] Gpu type=3 species=21.
[07:45:08] - Connecting to assignment server
[07:45:08] Connecting to http://assign-GPU.stanford.edu:8080/
[07:45:08] Posted data.
[07:45:08] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[07:45:08] + News From Folding@Home: Welcome to Folding@Home
[07:45:09] Loaded queue successfully.
[07:45:09] Gpu type=3 species=21.
[07:45:09] Sent data
[07:45:09] Connecting to http://171.64.65.93:8080/
[07:45:09] Posted data.
[07:45:09] Initial: 0000; - Receiving payload (expected size: 551953)
[07:45:09] Conversation time very short, giving reduced weight in bandwidth avg
[07:45:09] - Downloaded at ~1078 kB/s
[07:45:09] - Averaged speed for that direction ~634 kB/s
[07:45:09] + Received work.
[07:45:09] Trying to send all finished work units
[07:45:09] + No unsent completed units remaining.
[07:45:09] + Closed connections
[07:45:14] 
[07:45:14] + Processing work unit
[07:45:14] Core required: FahCore_15.exe
[07:45:14] Core found.
[07:45:14] Working on queue slot 09 [May 10 07:45:14 UTC]
[07:45:14] + Working ...
[07:45:14] - Calling '.\FahCore_15.exe -dir work/ -suffix 09 -nice 19 -checkpoint 15 -verbose -lifeline 4400 -version 641'

[07:45:14] 
[07:45:14] *------------------------------*
[07:45:14] Folding@Home GPU Core
[07:45:14] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[07:45:14] Build host             SimbiosNvdWin7
[07:45:14] Board Type             NVIDIA/CUDA
[07:45:14] Core                   15
[07:45:14] 
[07:45:14] Window's signal control handler registered.
[07:45:14] Preparing to commence simulation
[07:45:14] - Looking at optimizations...
[07:45:14] DeleteFrameFiles: successfully deleted file=work/wudata_09.ckp
[07:45:14] - Created dyn
[07:45:14] - Files status OK
[07:45:14] sizeof(CORE_PACKET_HDR) = 512 file=<>
[07:45:14] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[07:45:14] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[07:45:14] - Digital signature verified
[07:45:14] 
[07:45:14] Project: 7643 (Run 490, Clone 0, Gen 19)
[07:45:14] 
[07:45:14] Assembly optimizations on if available.
[07:45:14] Entering M.D.
[07:45:16] Tpr hash work/wudata_09.tpr:  2722680634 1728328070 1454610611 2632785485 3344963210
[07:45:16] GPU device info: vendor=0 device=0 name=<NA> match=0
[07:45:17] Working on Protein in water
[07:45:17] Client config found, loading data.
[07:45:17] Starting GUI Server
[07:46:57] Setting checkpoint frequency: 25000
[07:46:57] Completed         3 out of 2500000 steps (0%).
[08:10:21] Completed     25000 out of 2500000 steps (1%).
[08:10:22] mdrun_gpu returned 52
[08:10:22] NANs detected on GPU
[08:10:22] 
[08:10:22] Folding@home Core Shutdown: UNSTABLE_MACHINE
[08:10:25] CoreStatus = 7A (122)
[08:10:25] Sending work to server
[08:10:25] Project: 7643 (Run 490, Clone 0, Gen 19)
[08:10:25] - Read packet limit of 540015616... Set to 524286976.
[08:10:25] - Error: Could not get length of results file work/wuresults_09.dat
[08:10:25] - Error: Could not read unit 09 file. Removing from queue.
[08:10:25] EUE limit exceeded. Pausing 24 hours.
[11:05:25] - Autosending finished units... [May 10 11:05:25 UTC]
[11:05:25] Trying to send all finished work units
[11:05:25] + No unsent completed units remaining.
[11:05:25] - Autosend completed
[13:12:11] ***** Got a SIGTERM signal (2)
[13:12:11] Killing all core threads

Folding@Home Client Shutdown.


--- Opening Log file [May 10 13:14:34 UTC] 


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.41r2

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Users\poweruser\FAHGPU-1
Executable: C:\Users\poweruser\FAHGPU-1\FAHGPU-1a.exe
Arguments: -gpu 0 -verbosity 9 

[13:14:34] - Ask before connecting: No
[13:14:34] - User name: rexrzer (Team 111065)
[13:14:34] - User ID: 71038D4567CAF5F8
[13:14:34] - Machine ID: 11
[13:14:34] 
[13:14:34] Gpu type=3 species=21.
[13:14:34] Work directory not found. Creating...
[13:14:34] Could not open work queue, generating new queue...
[13:14:34] - Preparing to get new work unit...
[13:14:34] - Autosending finished units... [May 10 13:14:34 UTC]
[13:14:34] Cleaning up work directory
[13:14:34] Trying to send all finished work units
[13:14:34] + Attempting to get work packet
[13:14:34] + No unsent completed units remaining.
[13:14:34] Passkey found
[13:14:34] - Autosend completed
[13:14:34] - Will indicate memory of 4192 MB
[13:14:34] Gpu type=3 species=21.
[13:14:34] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 10, Stepping: 5
[13:14:34] - Connecting to assignment server
[13:14:34] Connecting to http://assign-GPU.stanford.edu:8080/
[13:14:34] Posted data.
[13:14:34] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[13:14:34] + News From Folding@Home: Welcome to Folding@Home
[13:14:34] Loaded queue successfully.
[13:14:34] Gpu type=3 species=21.
[13:14:34] Sent data
[13:14:34] Connecting to http://171.64.65.93:8080/
[13:14:34] Posted data.
[13:14:34] Initial: 0000; - Receiving payload (expected size: 551953)
[13:14:35] - Downloaded at ~539 kB/s
[13:14:35] - Averaged speed for that direction ~539 kB/s
[13:14:35] + Received work.
[13:14:35] + Closed connections
[13:14:35] 
[13:14:35] + Processing work unit
[13:14:35] Core required: FahCore_15.exe
[13:14:35] Core not found.
[13:14:35] - Core is not present or corrupted.
[13:14:35] - Attempting to download new core...
[13:14:35] + Downloading new core: FahCore_15.exe
[13:14:35] Downloading core (/~pande/Win32/x86/NVIDIA/Fermi/Core_15.fah from www.stanford.edu)
[13:14:35] Initial: AFDE; + 10240 bytes downloaded
[13:14:35] Initial: B149; + 20480 bytes downloaded
[13:14:35] Initial: F258; + 30720 bytes downloaded
[13:14:35] Initial: 3445; + 40960 bytes downloaded
[13:14:35] Initial: D51F; + 51200 bytes downloaded
[13:14:35] Initial: 8320; + 61440 bytes downloaded
[13:14:35] Initial: 6857; + 71680 bytes downloaded
[13:14:35] Initial: 8B0D; + 81920 bytes downloaded
[13:14:35] Initial: 5CC2; + 92160 bytes downloaded
[13:14:35] Initial: 49D2; + 102400 bytes downloaded
[13:14:35] Initial: 7422; + 112640 bytes downloaded
[13:14:35] Initial: 1089; + 122880 bytes downloaded
[13:14:35] Initial: 432F; + 133120 bytes downloaded
[13:14:35] Initial: 269E; + 143360 bytes downloaded
[13:14:35] Initial: 6958; + 153600 bytes downloaded
[13:14:35] Initial: A0EA; + 163840 bytes downloaded
[13:14:35] Initial: 28C4; + 174080 bytes downloaded
[13:14:35] Initial: 4000; + 184320 bytes downloaded
[13:14:35] Initial: 6390; + 194560 bytes downloaded
[13:14:35] Initial: A0A9; + 204800 bytes downloaded
[13:14:35] Initial: 8BB6; + 215040 bytes downloaded
[13:14:35] Initial: EF7E; + 225280 bytes downloaded
[13:14:35] Initial: B00E; + 235520 bytes downloaded
[13:14:35] Initial: 21E9; + 245760 bytes downloaded
[13:14:35] Initial: CBE4; + 256000 bytes downloaded
[13:14:35] Initial: 8E95; + 266240 bytes downloaded
[13:14:35] Initial: 4680; + 276480 bytes downloaded
[13:14:35] Initial: AD7E; + 286720 bytes downloaded
[13:14:35] Initial: 286B; + 296960 bytes downloaded
[13:14:35] Initial: CF0F; + 307200 bytes downloaded
[13:14:35] Initial: 9232; + 317440 bytes downloaded
[13:14:35] Initial: 1560; + 327680 bytes downloaded
[13:14:35] Initial: 1EEA; + 337920 bytes downloaded
[13:14:35] Initial: 3405; + 348160 bytes downloaded
[13:14:35] Initial: DC5B; + 358400 bytes downloaded
[13:14:35] Initial: F98E; + 368640 bytes downloaded
[13:14:35] Initial: 586D; + 378880 bytes downloaded
[13:14:35] Initial: EBD3; + 389120 bytes downloaded
[13:14:35] Initial: 55CE; + 399360 bytes downloaded
[13:14:35] Initial: 9783; + 409600 bytes downloaded
[13:14:35] Initial: 354C; + 419840 bytes downloaded
[13:14:35] Initial: 9ED3; + 430080 bytes downloaded
[13:14:35] Initial: 4724; + 440320 bytes downloaded
[13:14:35] Initial: 595F; + 450560 bytes downloaded
[13:14:35] Initial: 3C30; + 460800 bytes downloaded
[13:14:35] Initial: 6DCC; + 471040 bytes downloaded
[13:14:35] Initial: 4C51; + 481280 bytes downloaded
[13:14:35] Initial: 0AC2; + 491520 bytes downloaded
[13:14:35] Initial: BAF8; + 501760 bytes downloaded
[13:14:35] Initial: ECEA; + 512000 bytes downloaded
[13:14:35] Initial: 9F17; + 522240 bytes downloaded
[13:14:35] Initial: 9FDA; + 532480 bytes downloaded
[13:14:35] Initial: 9C9D; + 542720 bytes downloaded
[13:14:35] Initial: E006; + 552960 bytes downloaded
[13:14:35] Initial: 29C4; + 563200 bytes downloaded
[13:14:35] Initial: 7460; + 573440 bytes downloaded
[13:14:35] Initial: 2157; + 583680 bytes downloaded
[13:14:35] Initial: 93F1; + 593920 bytes downloaded
[13:14:35] Initial: 8EFC; + 604160 bytes downloaded
[13:14:35] Initial: 7329; + 614400 bytes downloaded
[13:14:35] Initial: 80F2; + 624640 bytes downloaded
[13:14:35] Initial: 9A1F; + 634880 bytes downloaded
[13:14:35] Initial: 4C46; + 645120 bytes downloaded
[13:14:35] Initial: 4B60; + 655360 bytes downloaded
[13:14:35] Initial: 5405; + 665600 bytes downloaded
[13:14:35] Initial: 1005; + 675840 bytes downloaded
[13:14:35] Initial: 311A; + 686080 bytes downloaded
[13:14:35] Initial: 5F86; + 696320 bytes downloaded
[13:14:35] Initial: A83E; + 706560 bytes downloaded
[13:14:35] Initial: 3426; + 716800 bytes downloaded
[13:14:35] Initial: 7489; + 727040 bytes downloaded
[13:14:35] Initial: BF49; + 737280 bytes downloaded
[13:14:35] Initial: 2F5A; + 747520 bytes downloaded
[13:14:35] Initial: BF36; + 757760 bytes downloaded
[13:14:35] Initial: 4120; + 768000 bytes downloaded
[13:14:35] Initial: ABAF; + 778240 bytes downloaded
[13:14:35] Initial: 3CD0; + 788480 bytes downloaded
[13:14:35] Initial: 39BF; + 798720 bytes downloaded
[13:14:35] Initial: 0EDC; + 808960 bytes downloaded
[13:14:35] Initial: BA99; + 819200 bytes downloaded
[13:14:35] Initial: 718D; + 829440 bytes downloaded
[13:14:35] Initial: 87BF; + 839680 bytes downloaded
[13:14:35] Initial: 87AE; + 849920 bytes downloaded
[13:14:35] Initial: 7C3B; + 860160 bytes downloaded
[13:14:35] Initial: 3E6D; + 870400 bytes downloaded
[13:14:35] Initial: D63B; + 880640 bytes downloaded
[13:14:35] Initial: CCAE; + 890880 bytes downloaded
[13:14:35] Initial: EAE0; + 901120 bytes downloaded
[13:14:35] Initial: 2D01; + 911360 bytes downloaded
[13:14:35] Initial: 4A00; + 921600 bytes downloaded
[13:14:35] Initial: 7EF1; + 931840 bytes downloaded
[13:14:35] Initial: C64D; + 942080 bytes downloaded
[13:14:35] Initial: DB24; + 952320 bytes downloaded
[13:14:35] Initial: 0E09; + 962560 bytes downloaded
[13:14:35] Initial: 083A; + 972800 bytes downloaded
[13:14:36] Initial: 8F16; + 983040 bytes downloaded
[13:14:36] Initial: 6F1A; + 993280 bytes downloaded
[13:14:36] Initial: BE3E; + 1003520 bytes downloaded
[13:14:36] Initial: 5339; + 1013760 bytes downloaded
[13:14:36] Initial: 5801; + 1024000 bytes downloaded
[13:14:36] Initial: 1191; + 1034240 bytes downloaded
[13:14:36] Initial: 2CB1; + 1044480 bytes downloaded
[13:14:36] Initial: E022; + 1054720 bytes downloaded
[13:14:36] Initial: 0000; + 1064960 bytes downloaded
[13:14:36] Initial: 260A; + 1075200 bytes downloaded
[13:14:36] Initial: 4ABF; + 1085440 bytes downloaded
[13:14:36] Initial: DF88; + 1095680 bytes downloaded
[13:14:36] Initial: 1D09; + 1105920 bytes downloaded
[13:14:36] Initial: 185E; + 1116160 bytes downloaded
[13:14:36] Initial: 6717; + 1126400 bytes downloaded
[13:14:36] Initial: 8D4D; + 1136640 bytes downloaded
[13:14:36] Initial: 0D13; + 1146880 bytes downloaded
[13:14:36] Initial: 04B9; + 1157120 bytes downloaded
[13:14:36] Initial: 4B8C; + 1167360 bytes downloaded
[13:14:36] Initial: E148; + 1177600 bytes downloaded
[13:14:36] Initial: 785E; + 1187840 bytes downloaded
[13:14:36] Initial: 24EF; + 1198080 bytes downloaded
[13:14:36] Initial: 1E91; + 1208320 bytes downloaded
[13:14:36] Initial: 9460; + 1218560 bytes downloaded
[13:14:36] Initial: 8C4C; + 1228800 bytes downloaded
[13:14:36] Initial: 5447; + 1239040 bytes downloaded
[13:14:36] Initial: BBB9; + 1249280 bytes downloaded
[13:14:36] Initial: ED1B; + 1259520 bytes downloaded
[13:14:36] Initial: 294B; + 1269760 bytes downloaded
[13:14:36] Initial: C105; + 1280000 bytes downloaded
[13:14:36] Initial: 2E08; + 1290240 bytes downloaded
[13:14:36] Initial: 264D; + 1300480 bytes downloaded
[13:14:36] Initial: 2089; + 1310720 bytes downloaded
[13:14:36] Initial: 2220; + 1320960 bytes downloaded
[13:14:36] Initial: 7FAE; + 1331200 bytes downloaded
[13:14:36] Initial: 965D; + 1341440 bytes downloaded
[13:14:36] Initial: 1F5E; + 1351680 bytes downloaded
[13:14:36] Initial: 8198; + 1361920 bytes downloaded
[13:14:36] Initial: E782; + 1372160 bytes downloaded
[13:14:36] Initial: FFFF; + 1382400 bytes downloaded
[13:14:36] Initial: 56C0; + 1392640 bytes downloaded
[13:14:36] Initial: 9B12; + 1402880 bytes downloaded
[13:14:36] Initial: 1729; + 1413120 bytes downloaded
[13:14:36] Initial: 9031; + 1423360 bytes downloaded
[13:14:36] Initial: 9C23; + 1433600 bytes downloaded
[13:14:36] Initial: E73F; + 1443840 bytes downloaded
[13:14:36] Initial: B822; + 1454080 bytes downloaded
[13:14:36] Initial: EF66; + 1464320 bytes downloaded
[13:14:36] Initial: 9278; + 1474560 bytes downloaded
[13:14:36] Initial: 9FAF; + 1484800 bytes downloaded
[13:14:36] Initial: 3C9E; + 1495040 bytes downloaded
[13:14:36] Initial: C589; + 1505280 bytes downloaded
[13:14:36] Initial: FE0B; + 1515520 bytes downloaded
[13:14:36] Initial: 55CC; + 1525760 bytes downloaded
[13:14:36] Initial: 306E; + 1536000 bytes downloaded
[13:14:36] Initial: 5D53; + 1546240 bytes downloaded
[13:14:36] Initial: 085B; + 1556480 bytes downloaded
[13:14:36] Initial: 2D59; + 1559166 bytes downloaded
[13:14:36] Verifying core Core_15.fah...
[13:14:36] Signature is VALID
[13:14:36] 
[13:14:36] Trying to unzip core FahCore_15.exe
[13:14:36] Decompressed FahCore_15.exe (4685824 bytes) successfully
[13:14:41] + Core successfully engaged
[13:14:46] 
[13:14:46] + Processing work unit
[13:14:46] Core required: FahCore_15.exe
[13:14:46] Core found.
[13:14:46] Working on queue slot 01 [May 10 13:14:46 UTC]
[13:14:46] + Working ...
[13:14:46] - Calling '.\FahCore_15.exe -dir work/ -suffix 01 -nice 19 -checkpoint 15 -verbose -lifeline 3104 -version 641'

[13:14:46] 
[13:14:46] *------------------------------*
[13:14:46] Folding@Home GPU Core
[13:14:46] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[13:14:46] Build host             SimbiosNvdWin7
[13:14:46] Board Type             NVIDIA/CUDA
[13:14:46] Core                   15
[13:14:46] 
[13:14:46] Window's signal control handler registered.
[13:14:46] Preparing to commence simulation
[13:14:46] - Looking at optimizations...
[13:14:46] DeleteFrameFiles: successfully deleted file=work/wudata_01.ckp
[13:14:46] - Created dyn
[13:14:46] - Files status OK
[13:14:46] sizeof(CORE_PACKET_HDR) = 512 file=<>
[13:14:46] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[13:14:46] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[13:14:46] - Digital signature verified
[13:14:46] 
[13:14:46] Project: 7643 (Run 490, Clone 0, Gen 19)
[13:14:46] 
[13:14:46] Assembly optimizations on if available.
[13:14:46] Entering M.D.
[13:14:49] Tpr hash work/wudata_01.tpr:  2722680634 1728328070 1454610611 2632785485 3344963210
[13:14:49] GPU device info: vendor=0 device=0 name=<NA> match=0
[13:14:49] Working on Protein in water
[13:14:49] Client config found, loading data.
[13:14:49] Starting GUI Server
[13:16:30] Setting checkpoint frequency: 25000
[13:16:30] Completed         3 out of 2500000 steps (0%).
[13:26:10] Completed     25000 out of 2500000 steps (1%).
[13:51:14] Completed     50000 out of 2500000 steps (2%).
[13:51:14] mdrun_gpu returned 52
[13:51:14] NANs detected on GPU
[13:51:14] 
[13:51:14] Folding@home Core Shutdown: UNSTABLE_MACHINE
[13:51:17] CoreStatus = 7A (122)
[13:51:17] Sending work to server
[13:51:17] Project: 7643 (Run 490, Clone 0, Gen 19)
[13:51:17] - Read packet limit of 540015616... Set to 524286976.
[13:51:17] - Error: Could not get length of results file work/wuresults_01.dat
[13:51:17] - Error: Could not read unit 01 file. Removing from queue.
[13:51:17] Trying to send all finished work units
[13:51:17] + No unsent completed units remaining.
[13:51:17] - Preparing to get new work unit...
[13:51:17] Cleaning up work directory
[13:51:17] + Attempting to get work packet
[13:51:17] Passkey found
[13:51:17] - Will indicate memory of 4192 MB
[13:51:17] Gpu type=3 species=21.
[13:51:17] - Connecting to assignment server
[13:51:17] Connecting to http://assign-GPU.stanford.edu:8080/
[13:51:18] Posted data.
[13:51:18] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[13:51:18] + News From Folding@Home: Welcome to Folding@Home
[13:51:18] Loaded queue successfully.
[13:51:18] Gpu type=3 species=21.
[13:51:18] Sent data
[13:51:18] Connecting to http://171.64.65.93:8080/
[13:51:18] Posted data.
[13:51:18] Initial: 0000; - Receiving payload (expected size: 551953)
[13:51:18] Conversation time very short, giving reduced weight in bandwidth avg
[13:51:18] - Downloaded at ~1078 kB/s
[13:51:18] - Averaged speed for that direction ~718 kB/s
[13:51:18] + Received work.
[13:51:18] Trying to send all finished work units
[13:51:18] + No unsent completed units remaining.
[13:51:18] + Closed connections
[13:51:23] 
[13:51:23] + Processing work unit
[13:51:23] Core required: FahCore_15.exe
[13:51:23] Core found.
[13:51:23] Working on queue slot 02 [May 10 13:51:23 UTC]
[13:51:23] + Working ...
[13:51:23] - Calling '.\FahCore_15.exe -dir work/ -suffix 02 -nice 19 -checkpoint 15 -verbose -lifeline 3104 -version 641'

[13:51:24] 
[13:51:24] *------------------------------*
[13:51:24] Folding@Home GPU Core
[13:51:24] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[13:51:24] Build host             SimbiosNvdWin7
[13:51:24] Board Type             NVIDIA/CUDA
[13:51:24] Core                   15
[13:51:24] 
[13:51:24] Window's signal control handler registered.
[13:51:24] Preparing to commence simulation
[13:51:24] - Looking at optimizations...
[13:51:24] DeleteFrameFiles: successfully deleted file=work/wudata_02.ckp
[13:51:24] - Created dyn
[13:51:24] - Files status OK
[13:51:24] sizeof(CORE_PACKET_HDR) = 512 file=<>
[13:51:24] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[13:51:24] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[13:51:24] - Digital signature verified
[13:51:24] 
[13:51:24] Project: 7643 (Run 490, Clone 0, Gen 19)
[13:51:24] 

Should this have been posted in the problems with WU's thread? My apologies if so, and the management may move this where it's
appropriate with my blessings. The driver being run, without fail the best Fermi GPU driver there is, is 285.62, Client is either 6.31
or 6.23, I can't recall which one exactly because this is the 1st time in 1.5 years with this GPU that I've had any issues at all, so I've
not had occasion to change anything up to now with this bad WU, apparently. I have tried running this at its traditional overclocking
of 980/1960/2170 and at default Mhz 900/1800/2106 with the same, exact results...it makes no difference about the clocking.

Can anyone offer some insight as to this particular WU problem, and what I should do about it, sooner rather than later?

Thank you for any advice in advance, and again, I hope this is the correct place to have filed such a report.

rexrzer

[rexrzer]

I've been having a terrible time with one specific WU in my PC No.3 desktop FAHome
dedicated PC, running Win7 64-bit SP1 build 7601, and i cannot seem to get rid of the
darn thing, and it has disabled my nVidia GTX 560Ti video card for folding presently.
I have tried all the traditional manners of stopping it, without anything good to report...
it just keeps downloading over and over again, even after repeatedly dismissing the
Work file, Core file, and both Info files within the FAHGPU-1a file itself.

I am having similar issues with WU's on PC No.1, and the Core i7 laptop that I have
for folding when I'm not using it for other things...I don't understand why all of a sudden
we're having BAD WU's happen all over the place with the GPU clients!

I am without words to describe the frustration of having a perfectly good Fermi GPU
sitting idle because of a bad WU like this. This machine has been folding GPU WU's
for more than one solid year, without any glitches or anomalies up to now, and given
the chance would complete any other WU but things like this quickly, without incidents
of any type, and frankly I am at a loss at to what to do next.

Here is what it looks like in the aggregate from the FAH Log file for almost 2 days worth
of attempts at folding this GPU WU, but now it sits idle because it seems the machine will
only continue downloading this specific WU over and over again, given the chance...

Code: Select all

Launch directory: C:\Users\poweruser\FAHGPU-1
Executable: C:\Users\poweruser\FAHGPU-1\FAHGPU-1a.exe
Arguments: -gpu 0 -verbosity 9 

[05:05:25] - Ask before connecting: No
[05:05:25] - User name: rexrzer (Team 111065)
[05:05:25] - User ID: 71038D4567CAF5F8
[05:05:25] - Machine ID: 11
[05:05:25] 
[05:05:25] Gpu type=3 species=21.
[05:05:25] Loaded queue successfully.
[05:05:25] - Preparing to get new work unit...
[05:05:25] Cleaning up work directory
[05:05:25] - Autosending finished units... [May 10 05:05:25 UTC]
[05:05:25] Trying to send all finished work units
[05:05:25] + No unsent completed units remaining.
[05:05:25] - Autosend completed
[05:05:25] + Attempting to get work packet
[05:05:25] Passkey found
[05:05:25] - Will indicate memory of 4192 MB
[05:05:25] Gpu type=3 species=21.
[05:05:25] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 10, Stepping: 5
[05:05:25] - Connecting to assignment server
[05:05:25] Connecting to http://assign-GPU.stanford.edu:8080/
[05:05:26] Posted data.
[05:05:26] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[05:05:26] + News From Folding@Home: Welcome to Folding@Home
[05:05:26] Loaded queue successfully.
[05:05:26] Gpu type=3 species=21.
[05:05:26] Sent data
[05:05:26] Connecting to http://171.64.65.93:8080/
[05:05:26] Posted data.
[05:05:26] Initial: 0000; - Receiving payload (expected size: 551953)
[05:05:27] - Downloaded at ~539 kB/s
[05:05:27] - Averaged speed for that direction ~503 kB/s
[05:05:27] + Received work.
[05:05:27] + Closed connections
[05:05:27] 
[05:05:27] + Processing work unit
[05:05:27] Core required: FahCore_15.exe
[05:05:27] Core found.
[05:05:27] Working on queue slot 05 [May 10 05:05:27 UTC]
[05:05:27] + Working ...
[05:05:27] - Calling '.\FahCore_15.exe -dir work/ -suffix 05 -nice 19 -checkpoint 15 -verbose -lifeline 4400 -version 641'

[05:05:27] 
[05:05:27] *------------------------------*
[05:05:27] Folding@Home GPU Core
[05:05:27] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[05:05:27] Build host             SimbiosNvdWin7
[05:05:27] Board Type             NVIDIA/CUDA
[05:05:27] Core                   15
[05:05:27] 
[05:05:27] Window's signal control handler registered.
[05:05:27] Preparing to commence simulation
[05:05:27] - Looking at optimizations...
[05:05:27] DeleteFrameFiles: successfully deleted file=work/wudata_05.ckp
[05:05:27] - Created dyn
[05:05:27] - Files status OK
[05:05:27] sizeof(CORE_PACKET_HDR) = 512 file=<>
[05:05:27] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[05:05:27] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[05:05:27] - Digital signature verified
[05:05:27] 
[05:05:27] Project: 7643 (Run 490, Clone 0, Gen 19)
[05:05:27] 
[05:05:27] Assembly optimizations on if available.
[05:05:27] Entering M.D.
[05:05:29] Tpr hash work/wudata_05.tpr:  2722680634 1728328070 1454610611 2632785485 3344963210
[05:05:29] GPU device info: vendor=0 device=0 name=<NA> match=0
[05:05:29] Working on Protein in water
[05:05:29] Client config found, loading data.
[05:05:29] Starting GUI Server
[05:07:09] Setting checkpoint frequency: 25000
[05:07:09] Completed         3 out of 2500000 steps (0%).
[05:16:08] Completed     25000 out of 2500000 steps (1%).
[05:25:06] Completed     50000 out of 2500000 steps (2%).
[05:34:02] Completed     75000 out of 2500000 steps (3%).
[05:52:22] Completed    100000 out of 2500000 steps (4%).
[05:52:23] mdrun_gpu returned 52
[05:52:23] NANs detected on GPU
[05:52:23] 
[05:52:23] Folding@home Core Shutdown: UNSTABLE_MACHINE
[05:52:26] CoreStatus = 7A (122)
[05:52:26] Sending work to server
[05:52:26] Project: 7643 (Run 490, Clone 0, Gen 19)
[05:52:26] - Read packet limit of 540015616... Set to 524286976.
[05:52:26] - Error: Could not get length of results file work/wuresults_05.dat
[05:52:26] - Error: Could not read unit 05 file. Removing from queue.
[05:52:26] Trying to send all finished work units
[05:52:26] + No unsent completed units remaining.
[05:52:26] - Preparing to get new work unit...
[05:52:26] Cleaning up work directory
[05:52:26] + Attempting to get work packet
[05:52:26] Passkey found
[05:52:26] - Will indicate memory of 4192 MB
[05:52:26] Gpu type=3 species=21.
[05:52:26] - Connecting to assignment server
[05:52:26] Connecting to http://assign-GPU.stanford.edu:8080/
[05:52:26] Posted data.
[05:52:26] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[05:52:26] + News From Folding@Home: Welcome to Folding@Home
[05:52:26] Loaded queue successfully.
[05:52:26] Gpu type=3 species=21.
[05:52:26] Sent data
[05:52:26] Connecting to http://171.64.65.93:8080/
[05:52:26] Posted data.
[05:52:26] Initial: 0000; - Receiving payload (expected size: 551953)
[05:52:27] - Downloaded at ~539 kB/s
[05:52:27] - Averaged speed for that direction ~510 kB/s
[05:52:27] + Received work.
[05:52:27] Trying to send all finished work units
[05:52:27] + No unsent completed units remaining.
[05:52:27] + Closed connections
[05:52:32] 
[05:52:32] + Processing work unit
[05:52:32] Core required: FahCore_15.exe
[05:52:32] Core found.
[05:52:32] Working on queue slot 06 [May 10 05:52:32 UTC]
[05:52:32] + Working ...
[05:52:32] - Calling '.\FahCore_15.exe -dir work/ -suffix 06 -nice 19 -checkpoint 15 -verbose -lifeline 4400 -version 641'

[05:52:32] 
[05:52:32] *------------------------------*
[05:52:32] Folding@Home GPU Core
[05:52:32] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[05:52:32] Build host             SimbiosNvdWin7
[05:52:32] Board Type             NVIDIA/CUDA
[05:52:32] Core                   15
[05:52:32] 
[05:52:32] Window's signal control handler registered.
[05:52:32] Preparing to commence simulation
[05:52:32] - Looking at optimizations...
[05:52:32] DeleteFrameFiles: successfully deleted file=work/wudata_06.ckp
[05:52:32] - Created dyn
[05:52:32] - Files status OK
[05:52:32] sizeof(CORE_PACKET_HDR) = 512 file=<>
[05:52:32] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[05:52:32] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[05:52:32] - Digital signature verified
[05:52:32] 
[05:52:32] Project: 7643 (Run 490, Clone 0, Gen 19)
[05:52:32] 
[05:52:32] Assembly optimizations on if available.
[05:52:32] Entering M.D.
[05:52:34] Tpr hash work/wudata_06.tpr:  2722680634 1728328070 1454610611 2632785485 3344963210
[05:52:34] GPU device info: vendor=0 device=0 name=<NA> match=0
[05:52:34] Working on Protein in water
[05:52:34] Client config found, loading data.
[05:52:34] Starting GUI Server
[05:54:14] Setting checkpoint frequency: 25000
[05:54:14] Completed         3 out of 2500000 steps (0%).
[06:21:52] Completed     25000 out of 2500000 steps (1%).
[06:21:52] mdrun_gpu returned 52
[06:21:52] NANs detected on GPU
[06:21:52] 
[06:21:52] Folding@home Core Shutdown: UNSTABLE_MACHINE
[06:21:56] CoreStatus = 7A (122)
[06:21:56] Sending work to server
[06:21:56] Project: 7643 (Run 490, Clone 0, Gen 19)
[06:21:56] - Read packet limit of 540015616... Set to 524286976.
[06:21:56] - Error: Could not get length of results file work/wuresults_06.dat
[06:21:56] - Error: Could not read unit 06 file. Removing from queue.
[06:21:56] Trying to send all finished work units
[06:21:56] + No unsent completed units remaining.
[06:21:56] - Preparing to get new work unit...
[06:21:56] Cleaning up work directory
[06:21:56] + Attempting to get work packet
[06:21:56] Passkey found
[06:21:56] - Will indicate memory of 4192 MB
[06:21:56] Gpu type=3 species=21.
[06:21:56] - Connecting to assignment server
[06:21:56] Connecting to http://assign-GPU.stanford.edu:8080/
[06:21:57] Posted data.
[06:21:57] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[06:21:57] + News From Folding@Home: Welcome to Folding@Home
[06:21:57] Loaded queue successfully.
[06:21:57] Gpu type=3 species=21.
[06:21:57] Sent data
[06:21:57] Connecting to http://171.64.65.93:8080/
[06:21:57] Posted data.
[06:21:57] Initial: 0000; - Receiving payload (expected size: 551953)
[06:21:58] - Downloaded at ~539 kB/s
[06:21:58] - Averaged speed for that direction ~516 kB/s
[06:21:58] + Received work.
[06:21:58] Trying to send all finished work units
[06:21:58] + No unsent completed units remaining.
[06:21:58] + Closed connections
[06:22:03] 
[06:22:03] + Processing work unit
[06:22:03] Core required: FahCore_15.exe
[06:22:03] Core found.
[06:22:03] Working on queue slot 07 [May 10 06:22:03 UTC]
[06:22:03] + Working ...
[06:22:03] - Calling '.\FahCore_15.exe -dir work/ -suffix 07 -nice 19 -checkpoint 15 -verbose -lifeline 4400 -version 641'

[06:22:03] 
[06:22:03] *------------------------------*
[06:22:03] Folding@Home GPU Core
[06:22:03] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[06:22:03] Build host             SimbiosNvdWin7
[06:22:03] Board Type             NVIDIA/CUDA
[06:22:03] Core                   15
[06:22:03] 
[06:22:03] Window's signal control handler registered.
[06:22:03] Preparing to commence simulation
[06:22:03] - Looking at optimizations...
[06:22:03] DeleteFrameFiles: successfully deleted file=work/wudata_07.ckp
[06:22:03] - Created dyn
[06:22:03] - Files status OK
[06:22:03] sizeof(CORE_PACKET_HDR) = 512 file=<>
[06:22:03] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[06:22:03] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[06:22:03] - Digital signature verified
[06:22:03] 
[06:22:03] Project: 7643 (Run 490, Clone 0, Gen 19)
[06:22:03] 
[06:22:03] Assembly optimizations on if available.
[06:22:03] Entering M.D.
[06:22:05] Tpr hash work/wudata_07.tpr:  2722680634 1728328070 1454610611 2632785485 3344963210
[06:22:05] GPU device info: vendor=0 device=0 name=<NA> match=0
[06:22:05] Working on Protein in water
[06:22:05] Client config found, loading data.
[06:22:05] Starting GUI Server
[06:23:45] Setting checkpoint frequency: 25000
[06:23:45] Completed         3 out of 2500000 steps (0%).
[06:32:44] Completed     25000 out of 2500000 steps (1%).
[06:58:14] Completed     50000 out of 2500000 steps (2%).
[06:58:14] mdrun_gpu returned 52
[06:58:14] NANs detected on GPU
[06:58:14] 
[06:58:14] Folding@home Core Shutdown: UNSTABLE_MACHINE
[06:58:17] CoreStatus = 7A (122)
[06:58:17] Sending work to server
[06:58:17] Project: 7643 (Run 490, Clone 0, Gen 19)
[06:58:17] - Read packet limit of 540015616... Set to 524286976.
[06:58:17] - Error: Could not get length of results file work/wuresults_07.dat
[06:58:17] - Error: Could not read unit 07 file. Removing from queue.
[06:58:17] Trying to send all finished work units
[06:58:17] + No unsent completed units remaining.
[06:58:17] - Preparing to get new work unit...
[06:58:17] Cleaning up work directory
[06:58:17] + Attempting to get work packet
[06:58:17] Passkey found
[06:58:17] - Will indicate memory of 4192 MB
[06:58:17] Gpu type=3 species=21.
[06:58:17] - Connecting to assignment server
[06:58:17] Connecting to http://assign-GPU.stanford.edu:8080/
[06:58:18] Posted data.
[06:58:18] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[06:58:18] + News From Folding@Home: Welcome to Folding@Home
[06:58:18] Loaded queue successfully.
[06:58:18] Gpu type=3 species=21.
[06:58:18] Sent data
[06:58:18] Connecting to http://171.64.65.93:8080/
[06:58:18] Posted data.
[06:58:18] Initial: 0000; - Receiving payload (expected size: 551953)
[06:58:18] Conversation time very short, giving reduced weight in bandwidth avg
[06:58:18] - Downloaded at ~1078 kB/s
[06:58:18] - Averaged speed for that direction ~578 kB/s
[06:58:18] + Received work.
[06:58:18] Trying to send all finished work units
[06:58:18] + No unsent completed units remaining.
[06:58:18] + Closed connections
[06:58:23] 
[06:58:23] + Processing work unit
[06:58:23] Core required: FahCore_15.exe
[06:58:23] Core found.
[06:58:23] Working on queue slot 08 [May 10 06:58:23 UTC]
[06:58:23] + Working ...
[06:58:23] - Calling '.\FahCore_15.exe -dir work/ -suffix 08 -nice 19 -checkpoint 15 -verbose -lifeline 4400 -version 641'

[06:58:23] 
[06:58:23] *------------------------------*
[06:58:23] Folding@Home GPU Core
[06:58:23] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[06:58:23] Build host             SimbiosNvdWin7
[06:58:23] Board Type             NVIDIA/CUDA
[06:58:23] Core                   15
[06:58:23] 
[06:58:23] Window's signal control handler registered.
[06:58:23] Preparing to commence simulation
[06:58:23] - Looking at optimizations...
[06:58:23] DeleteFrameFiles: successfully deleted file=work/wudata_08.ckp
[06:58:23] - Created dyn
[06:58:23] - Files status OK
[06:58:23] sizeof(CORE_PACKET_HDR) = 512 file=<>
[06:58:23] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[06:58:23] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[06:58:23] - Digital signature verified
[06:58:23] 
[06:58:23] Project: 7643 (Run 490, Clone 0, Gen 19)
[06:58:23] 
[06:58:23] Assembly optimizations on if available.
[06:58:23] Entering M.D.
[06:58:25] Tpr hash work/wudata_08.tpr:  2722680634 1728328070 1454610611 2632785485 3344963210
[06:58:25] GPU device info: vendor=0 device=0 name=<NA> match=0
[06:58:26] Working on Protein in water
[06:58:26] Client config found, loading data.
[06:58:26] Starting GUI Server
[07:00:04] Setting checkpoint frequency: 25000
[07:00:04] Completed         3 out of 2500000 steps (0%).
[07:08:58] Completed     25000 out of 2500000 steps (1%).
[07:17:54] Completed     50000 out of 2500000 steps (2%).
[07:45:04] Completed     75000 out of 2500000 steps (3%).
[07:45:05] mdrun_gpu returned 52
[07:45:05] NANs detected on GPU
[07:45:05] 
[07:45:05] Folding@home Core Shutdown: UNSTABLE_MACHINE
[07:45:08] CoreStatus = 7A (122)
[07:45:08] Sending work to server
[07:45:08] Project: 7643 (Run 490, Clone 0, Gen 19)
[07:45:08] - Read packet limit of 540015616... Set to 524286976.
[07:45:08] - Error: Could not get length of results file work/wuresults_08.dat
[07:45:08] - Error: Could not read unit 08 file. Removing from queue.
[07:45:08] Trying to send all finished work units
[07:45:08] + No unsent completed units remaining.
[07:45:08] - Preparing to get new work unit...
[07:45:08] Cleaning up work directory
[07:45:08] + Attempting to get work packet
[07:45:08] Passkey found
[07:45:08] - Will indicate memory of 4192 MB
[07:45:08] Gpu type=3 species=21.
[07:45:08] - Connecting to assignment server
[07:45:08] Connecting to http://assign-GPU.stanford.edu:8080/
[07:45:08] Posted data.
[07:45:08] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[07:45:08] + News From Folding@Home: Welcome to Folding@Home
[07:45:09] Loaded queue successfully.
[07:45:09] Gpu type=3 species=21.
[07:45:09] Sent data
[07:45:09] Connecting to http://171.64.65.93:8080/
[07:45:09] Posted data.
[07:45:09] Initial: 0000; - Receiving payload (expected size: 551953)
[07:45:09] Conversation time very short, giving reduced weight in bandwidth avg
[07:45:09] - Downloaded at ~1078 kB/s
[07:45:09] - Averaged speed for that direction ~634 kB/s
[07:45:09] + Received work.
[07:45:09] Trying to send all finished work units
[07:45:09] + No unsent completed units remaining.
[07:45:09] + Closed connections
[07:45:14] 
[07:45:14] + Processing work unit
[07:45:14] Core required: FahCore_15.exe
[07:45:14] Core found.
[07:45:14] Working on queue slot 09 [May 10 07:45:14 UTC]
[07:45:14] + Working ...
[07:45:14] - Calling '.\FahCore_15.exe -dir work/ -suffix 09 -nice 19 -checkpoint 15 -verbose -lifeline 4400 -version 641'

[07:45:14] 
[07:45:14] *------------------------------*
[07:45:14] Folding@Home GPU Core
[07:45:14] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[07:45:14] Build host             SimbiosNvdWin7
[07:45:14] Board Type             NVIDIA/CUDA
[07:45:14] Core                   15
[07:45:14] 
[07:45:14] Window's signal control handler registered.
[07:45:14] Preparing to commence simulation
[07:45:14] - Looking at optimizations...
[07:45:14] DeleteFrameFiles: successfully deleted file=work/wudata_09.ckp
[07:45:14] - Created dyn
[07:45:14] - Files status OK
[07:45:14] sizeof(CORE_PACKET_HDR) = 512 file=<>
[07:45:14] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[07:45:14] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[07:45:14] - Digital signature verified
[07:45:14] 
[07:45:14] Project: 7643 (Run 490, Clone 0, Gen 19)
[07:45:14] 
[07:45:14] Assembly optimizations on if available.
[07:45:14] Entering M.D.
[07:45:16] Tpr hash work/wudata_09.tpr:  2722680634 1728328070 1454610611 2632785485 3344963210
[07:45:16] GPU device info: vendor=0 device=0 name=<NA> match=0
[07:45:17] Working on Protein in water
[07:45:17] Client config found, loading data.
[07:45:17] Starting GUI Server
[07:46:57] Setting checkpoint frequency: 25000
[07:46:57] Completed         3 out of 2500000 steps (0%).
[08:10:21] Completed     25000 out of 2500000 steps (1%).
[08:10:22] mdrun_gpu returned 52
[08:10:22] NANs detected on GPU
[08:10:22] 
[08:10:22] Folding@home Core Shutdown: UNSTABLE_MACHINE
[08:10:25] CoreStatus = 7A (122)
[08:10:25] Sending work to server
[08:10:25] Project: 7643 (Run 490, Clone 0, Gen 19)
[08:10:25] - Read packet limit of 540015616... Set to 524286976.
[08:10:25] - Error: Could not get length of results file work/wuresults_09.dat
[08:10:25] - Error: Could not read unit 09 file. Removing from queue.
[08:10:25] EUE limit exceeded. Pausing 24 hours.
[11:05:25] - Autosending finished units... [May 10 11:05:25 UTC]
[11:05:25] Trying to send all finished work units
[11:05:25] + No unsent completed units remaining.
[11:05:25] - Autosend completed
[13:12:11] ***** Got a SIGTERM signal (2)
[13:12:11] Killing all core threads

Folding@Home Client Shutdown.


--- Opening Log file [May 10 13:14:34 UTC] 


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.41r2

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Users\poweruser\FAHGPU-1
Executable: C:\Users\poweruser\FAHGPU-1\FAHGPU-1a.exe
Arguments: -gpu 0 -verbosity 9 

[13:14:34] - Ask before connecting: No
[13:14:34] - User name: rexrzer (Team 111065)
[13:14:34] - User ID: 71038D4567CAF5F8
[13:14:34] - Machine ID: 11
[13:14:34] 
[13:14:34] Gpu type=3 species=21.
[13:14:34] Work directory not found. Creating...
[13:14:34] Could not open work queue, generating new queue...
[13:14:34] - Preparing to get new work unit...
[13:14:34] - Autosending finished units... [May 10 13:14:34 UTC]
[13:14:34] Cleaning up work directory
[13:14:34] Trying to send all finished work units
[13:14:34] + Attempting to get work packet
[13:14:34] + No unsent completed units remaining.
[13:14:34] Passkey found
[13:14:34] - Autosend completed
[13:14:34] - Will indicate memory of 4192 MB
[13:14:34] Gpu type=3 species=21.
[13:14:34] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 10, Stepping: 5
[13:14:34] - Connecting to assignment server
[13:14:34] Connecting to http://assign-GPU.stanford.edu:8080/
[13:14:34] Posted data.
[13:14:34] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[13:14:34] + News From Folding@Home: Welcome to Folding@Home
[13:14:34] Loaded queue successfully.
[13:14:34] Gpu type=3 species=21.
[13:14:34] Sent data
[13:14:34] Connecting to http://171.64.65.93:8080/
[13:14:34] Posted data.
[13:14:34] Initial: 0000; - Receiving payload (expected size: 551953)
[13:14:35] - Downloaded at ~539 kB/s
[13:14:35] - Averaged speed for that direction ~539 kB/s
[13:14:35] + Received work.
[13:14:35] + Closed connections
[13:14:35] 
[13:14:35] + Processing work unit
[13:14:35] Core required: FahCore_15.exe
[13:14:35] Core not found.
[13:14:35] - Core is not present or corrupted.
[13:14:35] - Attempting to download new core...
[13:14:35] + Downloading new core: FahCore_15.exe
[13:14:35] Downloading core (/~pande/Win32/x86/NVIDIA/Fermi/Core_15.fah from www.stanford.edu)
[13:14:35] Initial: AFDE; + 10240 bytes downloaded
[13:14:35] Initial: B149; + 20480 bytes downloaded
[13:14:35] Initial: F258; + 30720 bytes downloaded
[13:14:35] Initial: 3445; + 40960 bytes downloaded
[13:14:35] Initial: D51F; + 51200 bytes downloaded
[13:14:35] Initial: 8320; + 61440 bytes downloaded
[13:14:35] Initial: 6857; + 71680 bytes downloaded
[13:14:35] Initial: 8B0D; + 81920 bytes downloaded
[13:14:35] Initial: 5CC2; + 92160 bytes downloaded
[13:14:35] Initial: 49D2; + 102400 bytes downloaded
[13:14:35] Initial: 7422; + 112640 bytes downloaded
[13:14:35] Initial: 1089; + 122880 bytes downloaded
[13:14:35] Initial: 432F; + 133120 bytes downloaded
[13:14:35] Initial: 269E; + 143360 bytes downloaded
[13:14:35] Initial: 6958; + 153600 bytes downloaded
[13:14:35] Initial: A0EA; + 163840 bytes downloaded
[13:14:35] Initial: 28C4; + 174080 bytes downloaded
[13:14:35] Initial: 4000; + 184320 bytes downloaded
[13:14:35] Initial: 6390; + 194560 bytes downloaded
[13:14:35] Initial: A0A9; + 204800 bytes downloaded
[13:14:35] Initial: 8BB6; + 215040 bytes downloaded
[13:14:35] Initial: EF7E; + 225280 bytes downloaded
[13:14:35] Initial: B00E; + 235520 bytes downloaded
[13:14:35] Initial: 21E9; + 245760 bytes downloaded
[13:14:35] Initial: CBE4; + 256000 bytes downloaded
[13:14:35] Initial: 8E95; + 266240 bytes downloaded
[13:14:35] Initial: 4680; + 276480 bytes downloaded
[13:14:35] Initial: AD7E; + 286720 bytes downloaded
[13:14:35] Initial: 286B; + 296960 bytes downloaded
[13:14:35] Initial: CF0F; + 307200 bytes downloaded
[13:14:35] Initial: 9232; + 317440 bytes downloaded
[13:14:35] Initial: 1560; + 327680 bytes downloaded
[13:14:35] Initial: 1EEA; + 337920 bytes downloaded
[13:14:35] Initial: 3405; + 348160 bytes downloaded
[13:14:35] Initial: DC5B; + 358400 bytes downloaded
[13:14:35] Initial: F98E; + 368640 bytes downloaded
[13:14:35] Initial: 586D; + 378880 bytes downloaded
[13:14:35] Initial: EBD3; + 389120 bytes downloaded
[13:14:35] Initial: 55CE; + 399360 bytes downloaded
[13:14:35] Initial: 9783; + 409600 bytes downloaded
[13:14:35] Initial: 354C; + 419840 bytes downloaded
[13:14:35] Initial: 9ED3; + 430080 bytes downloaded
[13:14:35] Initial: 4724; + 440320 bytes downloaded
[13:14:35] Initial: 595F; + 450560 bytes downloaded
[13:14:35] Initial: 3C30; + 460800 bytes downloaded
[13:14:35] Initial: 6DCC; + 471040 bytes downloaded
[13:14:35] Initial: 4C51; + 481280 bytes downloaded
[13:14:35] Initial: 0AC2; + 491520 bytes downloaded
[13:14:35] Initial: BAF8; + 501760 bytes downloaded
[13:14:35] Initial: ECEA; + 512000 bytes downloaded
[13:14:35] Initial: 9F17; + 522240 bytes downloaded
[13:14:35] Initial: 9FDA; + 532480 bytes downloaded
[13:14:35] Initial: 9C9D; + 542720 bytes downloaded
[13:14:35] Initial: E006; + 552960 bytes downloaded
[13:14:35] Initial: 29C4; + 563200 bytes downloaded
[13:14:35] Initial: 7460; + 573440 bytes downloaded
[13:14:35] Initial: 2157; + 583680 bytes downloaded
[13:14:35] Initial: 93F1; + 593920 bytes downloaded
[13:14:35] Initial: 8EFC; + 604160 bytes downloaded
[13:14:35] Initial: 7329; + 614400 bytes downloaded
[13:14:35] Initial: 80F2; + 624640 bytes downloaded
[13:14:35] Initial: 9A1F; + 634880 bytes downloaded
[13:14:35] Initial: 4C46; + 645120 bytes downloaded
[13:14:35] Initial: 4B60; + 655360 bytes downloaded
[13:14:35] Initial: 5405; + 665600 bytes downloaded
[13:14:35] Initial: 1005; + 675840 bytes downloaded
[13:14:35] Initial: 311A; + 686080 bytes downloaded
[13:14:35] Initial: 5F86; + 696320 bytes downloaded
[13:14:35] Initial: A83E; + 706560 bytes downloaded
[13:14:35] Initial: 3426; + 716800 bytes downloaded
[13:14:35] Initial: 7489; + 727040 bytes downloaded
[13:14:35] Initial: BF49; + 737280 bytes downloaded
[13:14:35] Initial: 2F5A; + 747520 bytes downloaded
[13:14:35] Initial: BF36; + 757760 bytes downloaded
[13:14:35] Initial: 4120; + 768000 bytes downloaded
[13:14:35] Initial: ABAF; + 778240 bytes downloaded
[13:14:35] Initial: 3CD0; + 788480 bytes downloaded
[13:14:35] Initial: 39BF; + 798720 bytes downloaded
[13:14:35] Initial: 0EDC; + 808960 bytes downloaded
[13:14:35] Initial: BA99; + 819200 bytes downloaded
[13:14:35] Initial: 718D; + 829440 bytes downloaded
[13:14:35] Initial: 87BF; + 839680 bytes downloaded
[13:14:35] Initial: 87AE; + 849920 bytes downloaded
[13:14:35] Initial: 7C3B; + 860160 bytes downloaded
[13:14:35] Initial: 3E6D; + 870400 bytes downloaded
[13:14:35] Initial: D63B; + 880640 bytes downloaded
[13:14:35] Initial: CCAE; + 890880 bytes downloaded
[13:14:35] Initial: EAE0; + 901120 bytes downloaded
[13:14:35] Initial: 2D01; + 911360 bytes downloaded
[13:14:35] Initial: 4A00; + 921600 bytes downloaded
[13:14:35] Initial: 7EF1; + 931840 bytes downloaded
[13:14:35] Initial: C64D; + 942080 bytes downloaded
[13:14:35] Initial: DB24; + 952320 bytes downloaded
[13:14:35] Initial: 0E09; + 962560 bytes downloaded
[13:14:35] Initial: 083A; + 972800 bytes downloaded
[13:14:36] Initial: 8F16; + 983040 bytes downloaded
[13:14:36] Initial: 6F1A; + 993280 bytes downloaded
[13:14:36] Initial: BE3E; + 1003520 bytes downloaded
[13:14:36] Initial: 5339; + 1013760 bytes downloaded
[13:14:36] Initial: 5801; + 1024000 bytes downloaded
[13:14:36] Initial: 1191; + 1034240 bytes downloaded
[13:14:36] Initial: 2CB1; + 1044480 bytes downloaded
[13:14:36] Initial: E022; + 1054720 bytes downloaded
[13:14:36] Initial: 0000; + 1064960 bytes downloaded
[13:14:36] Initial: 260A; + 1075200 bytes downloaded
[13:14:36] Initial: 4ABF; + 1085440 bytes downloaded
[13:14:36] Initial: DF88; + 1095680 bytes downloaded
[13:14:36] Initial: 1D09; + 1105920 bytes downloaded
[13:14:36] Initial: 185E; + 1116160 bytes downloaded
[13:14:36] Initial: 6717; + 1126400 bytes downloaded
[13:14:36] Initial: 8D4D; + 1136640 bytes downloaded
[13:14:36] Initial: 0D13; + 1146880 bytes downloaded
[13:14:36] Initial: 04B9; + 1157120 bytes downloaded
[13:14:36] Initial: 4B8C; + 1167360 bytes downloaded
[13:14:36] Initial: E148; + 1177600 bytes downloaded
[13:14:36] Initial: 785E; + 1187840 bytes downloaded
[13:14:36] Initial: 24EF; + 1198080 bytes downloaded
[13:14:36] Initial: 1E91; + 1208320 bytes downloaded
[13:14:36] Initial: 9460; + 1218560 bytes downloaded
[13:14:36] Initial: 8C4C; + 1228800 bytes downloaded
[13:14:36] Initial: 5447; + 1239040 bytes downloaded
[13:14:36] Initial: BBB9; + 1249280 bytes downloaded
[13:14:36] Initial: ED1B; + 1259520 bytes downloaded
[13:14:36] Initial: 294B; + 1269760 bytes downloaded
[13:14:36] Initial: C105; + 1280000 bytes downloaded
[13:14:36] Initial: 2E08; + 1290240 bytes downloaded
[13:14:36] Initial: 264D; + 1300480 bytes downloaded
[13:14:36] Initial: 2089; + 1310720 bytes downloaded
[13:14:36] Initial: 2220; + 1320960 bytes downloaded
[13:14:36] Initial: 7FAE; + 1331200 bytes downloaded
[13:14:36] Initial: 965D; + 1341440 bytes downloaded
[13:14:36] Initial: 1F5E; + 1351680 bytes downloaded
[13:14:36] Initial: 8198; + 1361920 bytes downloaded
[13:14:36] Initial: E782; + 1372160 bytes downloaded
[13:14:36] Initial: FFFF; + 1382400 bytes downloaded
[13:14:36] Initial: 56C0; + 1392640 bytes downloaded
[13:14:36] Initial: 9B12; + 1402880 bytes downloaded
[13:14:36] Initial: 1729; + 1413120 bytes downloaded
[13:14:36] Initial: 9031; + 1423360 bytes downloaded
[13:14:36] Initial: 9C23; + 1433600 bytes downloaded
[13:14:36] Initial: E73F; + 1443840 bytes downloaded
[13:14:36] Initial: B822; + 1454080 bytes downloaded
[13:14:36] Initial: EF66; + 1464320 bytes downloaded
[13:14:36] Initial: 9278; + 1474560 bytes downloaded
[13:14:36] Initial: 9FAF; + 1484800 bytes downloaded
[13:14:36] Initial: 3C9E; + 1495040 bytes downloaded
[13:14:36] Initial: C589; + 1505280 bytes downloaded
[13:14:36] Initial: FE0B; + 1515520 bytes downloaded
[13:14:36] Initial: 55CC; + 1525760 bytes downloaded
[13:14:36] Initial: 306E; + 1536000 bytes downloaded
[13:14:36] Initial: 5D53; + 1546240 bytes downloaded
[13:14:36] Initial: 085B; + 1556480 bytes downloaded
[13:14:36] Initial: 2D59; + 1559166 bytes downloaded
[13:14:36] Verifying core Core_15.fah...
[13:14:36] Signature is VALID
[13:14:36] 
[13:14:36] Trying to unzip core FahCore_15.exe
[13:14:36] Decompressed FahCore_15.exe (4685824 bytes) successfully
[13:14:41] + Core successfully engaged
[13:14:46] 
[13:14:46] + Processing work unit
[13:14:46] Core required: FahCore_15.exe
[13:14:46] Core found.
[13:14:46] Working on queue slot 01 [May 10 13:14:46 UTC]
[13:14:46] + Working ...
[13:14:46] - Calling '.\FahCore_15.exe -dir work/ -suffix 01 -nice 19 -checkpoint 15 -verbose -lifeline 3104 -version 641'

[13:14:46] 
[13:14:46] *------------------------------*
[13:14:46] Folding@Home GPU Core
[13:14:46] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[13:14:46] Build host             SimbiosNvdWin7
[13:14:46] Board Type             NVIDIA/CUDA
[13:14:46] Core                   15
[13:14:46] 
[13:14:46] Window's signal control handler registered.
[13:14:46] Preparing to commence simulation
[13:14:46] - Looking at optimizations...
[13:14:46] DeleteFrameFiles: successfully deleted file=work/wudata_01.ckp
[13:14:46] - Created dyn
[13:14:46] - Files status OK
[13:14:46] sizeof(CORE_PACKET_HDR) = 512 file=<>
[13:14:46] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[13:14:46] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[13:14:46] - Digital signature verified
[13:14:46] 
[13:14:46] Project: 7643 (Run 490, Clone 0, Gen 19)
[13:14:46] 
[13:14:46] Assembly optimizations on if available.
[13:14:46] Entering M.D.
[13:14:49] Tpr hash work/wudata_01.tpr:  2722680634 1728328070 1454610611 2632785485 3344963210
[13:14:49] GPU device info: vendor=0 device=0 name=<NA> match=0
[13:14:49] Working on Protein in water
[13:14:49] Client config found, loading data.
[13:14:49] Starting GUI Server
[13:16:30] Setting checkpoint frequency: 25000
[13:16:30] Completed         3 out of 2500000 steps (0%).
[13:26:10] Completed     25000 out of 2500000 steps (1%).
[13:51:14] Completed     50000 out of 2500000 steps (2%).
[13:51:14] mdrun_gpu returned 52
[13:51:14] NANs detected on GPU
[13:51:14] 
[13:51:14] Folding@home Core Shutdown: UNSTABLE_MACHINE
[13:51:17] CoreStatus = 7A (122)
[13:51:17] Sending work to server
[13:51:17] Project: 7643 (Run 490, Clone 0, Gen 19)
[13:51:17] - Read packet limit of 540015616... Set to 524286976.
[13:51:17] - Error: Could not get length of results file work/wuresults_01.dat
[13:51:17] - Error: Could not read unit 01 file. Removing from queue.
[13:51:17] Trying to send all finished work units
[13:51:17] + No unsent completed units remaining.
[13:51:17] - Preparing to get new work unit...
[13:51:17] Cleaning up work directory
[13:51:17] + Attempting to get work packet
[13:51:17] Passkey found
[13:51:17] - Will indicate memory of 4192 MB
[13:51:17] Gpu type=3 species=21.
[13:51:17] - Connecting to assignment server
[13:51:17] Connecting to http://assign-GPU.stanford.edu:8080/
[13:51:18] Posted data.
[13:51:18] Initial: 40AB; - Successful: assigned to (171.64.65.93).
[13:51:18] + News From Folding@Home: Welcome to Folding@Home
[13:51:18] Loaded queue successfully.
[13:51:18] Gpu type=3 species=21.
[13:51:18] Sent data
[13:51:18] Connecting to http://171.64.65.93:8080/
[13:51:18] Posted data.
[13:51:18] Initial: 0000; - Receiving payload (expected size: 551953)
[13:51:18] Conversation time very short, giving reduced weight in bandwidth avg
[13:51:18] - Downloaded at ~1078 kB/s
[13:51:18] - Averaged speed for that direction ~718 kB/s
[13:51:18] + Received work.
[13:51:18] Trying to send all finished work units
[13:51:18] + No unsent completed units remaining.
[13:51:18] + Closed connections
[13:51:23] 
[13:51:23] + Processing work unit
[13:51:23] Core required: FahCore_15.exe
[13:51:23] Core found.
[13:51:23] Working on queue slot 02 [May 10 13:51:23 UTC]
[13:51:23] + Working ...
[13:51:23] - Calling '.\FahCore_15.exe -dir work/ -suffix 02 -nice 19 -checkpoint 15 -verbose -lifeline 3104 -version 641'

[13:51:24] 
[13:51:24] *------------------------------*
[13:51:24] Folding@Home GPU Core
[13:51:24] Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
[13:51:24] Build host             SimbiosNvdWin7
[13:51:24] Board Type             NVIDIA/CUDA
[13:51:24] Core                   15
[13:51:24] 
[13:51:24] Window's signal control handler registered.
[13:51:24] Preparing to commence simulation
[13:51:24] - Looking at optimizations...
[13:51:24] DeleteFrameFiles: successfully deleted file=work/wudata_02.ckp
[13:51:24] - Created dyn
[13:51:24] - Files status OK
[13:51:24] sizeof(CORE_PACKET_HDR) = 512 file=<>
[13:51:24] - Expanded 551441 -> 896732 (decompressed 162.6 percent)
[13:51:24] Called DecompressByteArray: compressed_data_size=551441 data_size=896732, decompressed_data_size=896732 diff=0
[13:51:24] - Digital signature verified
[13:51:24] 
[13:51:24] Project: 7643 (Run 490, Clone 0, Gen 19)
[13:51:24]

The driver being run, without fail the best Fermi GPU driver there is, is 285.62. This is the 1st time in 1.5 years with this GPU that I've had any issues at all, so I've not had occasion to change anything up to now with this bad WU, apparently. I have tried running this at its traditional overclocking of 980/1960/2170 and at default Mhz 900/1800/2106 with the same, exact results...it makes no difference about the clocking.

Can anyone offer some insight as to this particular WU problem, and what I should do about it, sooner rather than later?

rexrzer

[bruce]

PM sent

[7im]

It's been running fine for 1.5 years, because that's how long it took for Pande Group to come out with larger work units.

76xx work units are much larger in size, and thus push the GPUs harder. What was once a stable overclock for a long time may no longer be stable. NaN errors are typically caused by hardware being pushed too hard or too hot. Adjust accordingly... like try underclocking the card to 850?

[rexrzer]

It's been running fine for 1.5 years, because that's how long it took for Pande Group to come out with larger work units.

76xx work units are much larger in size, and thus push the GPUs harder. What was once a stable overclock for a long time may no longer be stable. NaN errors are typically caused by hardware being pushed too hard or too hot. Adjust accordingly... like try underclocking the card to 850?

If it were that simple, even I would have figured out that underclocking the GPU would be the answer, and it isn't.

First , the GPU runs at barely over its normal folding temps for this WU, and the temps, by my logs anyway, did not
*increase* at all during the piddly 1-2% it takes to generate a "NANs" error, or UNSTABLE_MACHINE error with this WU.
If the trouble is RAM allocation, I have that idea covered in that my 6-Core 970-24GB RAM-- just clocked to 4215Mhz these days,
since I got tired of seeing any temps over 65-degrees in my expensive (to me it IS expensive) 970 CPU setup, albeit one
running the best air CPU cooler I know of, with 9 Scythe specialty fans with the "Sony liquid bearings" all around, so the
CPU is not showing any wear and tear at all right now. Neither is the GPU for that matter.

As for the 560Ti SC, it passed the heaviest test of them all-plus AIDA 64, and all others with flying colors (the one in the PC No.3 not
the 6-Core), and since the 6-Core's is a duplicate of the setup in the PC No.3, I found it useless to test that GPU also.

If you mean to say that these new 76xx WU's are not finding our GPU's acceptable as folders for some reason or other, I
would be inclined to agree with you, what with two out of 3 machines here with the Big WU's being folded showing
signs of not liking the new GPU WU's either, for that matter! But I repeat with all due respect 7im, because you are capable
of giving very good advice overall and I have no problem with the sarcasm and wit et al in the posts either, find them actually
entertaining overall, anyway with all respects due I see an alarming trend here if this is the case with the new 76xx WU's as
a group, if simple 560Ti SC's can't fold them for some reason or other. As for other GPU's not folding them successfully either,
I have seen reading in the Forum of late, that other GPU folders are having the same trouble as I am experiencing...an interesting
trend, no?

Is Stanford telling us menial folders with the average to low/high tech units like the 560Ti's in my stable, telling us that they don't
want us folding anymore? That is what it comes down to with me, personally, unfortunately. I never had issues with WU's until the
past couple weeks, and that is very strange to experience after years and years of doing this stuff! I am not going to go out and
spend $1K on a new nVidia 690 just to do folding, you dig? Nor any other of the newest, expensive video boards, as I have just about
worn myself out doing the experimentation for Stanford over the years for the well and good of the Folding Gods, and the buck stops
here with my 560Ti SC's unfortunately, again.

My apology for any rash tones here, as I don't mean anything personal to anybody here at all, absolutely nothing of that nature
here in my post at all. This is pure bizness, and the folding bizness is going to be missing people like myself I am certain, if we aren't
there in a vast, bigger sense of the word "Folding@Home" because the WU's are "too large and complex" for us to be folding them...
don't you think that's a bad thing rather than a good thing? I do, frankly and I don't see how you could see differently either.

There's nothing wrong with my setups, nothing wrong with my video boards or cooling mechanisms, and the cards DO NOT RUN HOT
in any definition of that phrase: they are cool, stable, and normal even when being refused by these finicky, complex and large WU's
that we are all having trouble with, apparently.

Thanks for your advice also Bruce, and it's always appreciated when you are involved with a subject/topic here. Thanks
for offering up the suggestions about my programming of the WU coding, and I am sorry to say it's made no difference in my case
at all...bad luck there.

Anyway boys and girls, I have better things to do than fight with the Stanford Folding Gods about what we are being asked to fold
these days, and if they are beyond the abilities of quite normal video boards such as mine, then I feel sorry for Stanford, not the
other way around...understand me well there, it's a case of having folders willing to fold WU's with good equipment and decent if
not xlnt setups to boot. but the WU's we're being asked to fold are NOT BEING COOPERATIVE with our hardware, end of story.

Thanks for your time, and thoughts here, I will wait it out a couple more days and see if anything changes, but those same
WU's are GLUED LIKE EPOXY to my video boards, I cannot get rid of them as they get assigned over and over until Infinity Freezes
Over apparently, and I don't think that's right either, but such is life with Stanford Folding@Home.

rexrzer

[bruce]

I've had some similar troubles with 76xx projects. I don't know exactly what the problem is.

[{RaW}Eagle1]

I have noticed a couple of posts about people potentially abandoning GPU folding due to issues relating to 76xx projects. In the interest of encouraging people to continue to fold on their GPUs I'd like to comment that I am currently running F@H GPU v6.41 (console) on two GTX480s with the -advmethods flag enabled. They are currently folding project 8009 not 76xx.

Perhaps the answer (at least until the problem with these units can be identified) is to use the v6.41 client with the -advmethods flag, rather than abandoning F@H altogether? Clearing the "work" directory, deleting "queue.dat and adding or removing the -advmethods flag without change of client version has also worked for at least 1 other person (in that; they have been able to pick up different work units).

Eagle

[bruce]

Do we actually know that projects 76xx are assigned only to V7?

The Pande Group is aware of this problem and it's also possible that the entire series of projects has been suspended until some changes can be made.

[rexrzer]

Do we actually know that projects 76xx are assigned only to V7?

The Pande Group is aware of this problem and it's also possible that the entire series of projects has been suspended until some changes can be made.

Well Bruce, my friend, we can see the light of the tunnel opening here at my folding digs also this afternoon, so Hurray for that happening!
I am currently re-programming all my WU's to have the Adv tag running full bore on the 6.31 (I finally checked on the version-that's what I have) Clients that I have in the stable here, which is all of my 560Ti SC's, and I'll be willing to try it on my laptop also at some point later tonight, but right now I am using the laptop for a little web page building and can't relinquish it presently. It too has been disabled by the 76xx WU's, by the way. So I've had ZERO points happening for 3-4 of my GPU's for the past week, and my average PPD has dropped down to less than 100K from 150K+/- just two weeks ago.

Thank you Pande Group for recognizing a major problem somewhere in these 76xx WU's, as we just randomly cannot fold them with good equipment, end of story. I hope that they get it right as I don't mind the challenge of a larger WU on my GPU's or CPU's--they have handled everything else up to now and doubt they will fail anything, within reason, that the Pande Group comes up with for us to fold.

Thanks for concurring with me Bruce, also about these WU's we're all having trouble with recently. It was bound to happen sometime, I guess, as it's reasonable to assume that even the best programmers that Pande Group has working making WU's for us are capable of making an error of terrible proportions, unfortunately. Up to now this idea has been minimal to none in existence with my folding machines, and I hope that the group can "man up" and get some decently conceived 76xx's for us to fold at some point. But for now, thanks for pulling them out of the mix here!

I really appreciate being able to contribute to science in my small way once again, with these newer WU's being much more involved and complex than the previous WU's we had in the mix. I think it would be a sad day for some of us to call it quits because of some bad WU's in the mix, but that's what I was facing with the 76xx's popping up constantly in my workflow.

Better days are ahead, I guess... once again! Thanks to everyone for getting together on the issues of the 76xx's WU's, it's appreciated.

rexrzer

[bruce]

Changing all your clients to "advanced" may not be necessary. One of the first steps the Pande Group has probably taken is to suspend projects 76xx until they can diagnose the actual problem. That would probably been useful on a machine that already had received a bad one, but it's not clear whether changing them all is useful.