Stock GTX-670 started randomly failing all WU's

[mattlowe01`]

Hello again all,

It's finally getting colder again, so I started to fire up some folding. My rig is a 3930k, 1 GTX-670 FTW, 2 x GTS 250's. I tried configuring v7 client again, but I couldn't for the life of me get it to work with my multiple GPU's. My machine reports the 670 as GPU0, and the GTS 250's as GPU 1&2, but v7 reports the 670 as GPU2, and the 250's at 0 & 1. I spent a few hours messing with the GPU client index numbers as well as the cuda, but couldn't get them to fold properly, so I downloaded FAH GPU Tracker v2, which correctly configured all of my video cards out of the box.

The 670 was folding fine for a few days, now it's failing every WU that get's sent to it saying UNSTABLE MACHINE. It's at stock clocks. I even bumped the voltage up for a bit to try to make it stable, but it didn't work. Here are a few of the logs.

Most Recent:

Code: Select all


--- Opening Log file [October 25 20:00:51 UTC]


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.30r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Users\Matt\Desktop\FAH GPU Tracker V2\GPU0
Executable: C:\Users\Matt\Desktop\FAH GPU Tracker V2\FAH_GPU3.exe
Arguments: -oneunit -forcegpu nvidia_fermi -verbosity 9 -gpu 0

[20:00:51] - Ask before connecting: No
[20:00:51] - User name: MattLowe (Team 50959)
[20:00:51] - User ID: 4E9F93E95F93C178
[20:00:51] - Machine ID: 3
[20:00:51]
[20:00:51] Gpu type=3 species=40.
[20:00:51] Work directory not found. Creating...
[20:00:51] Could not open work queue, generating new queue...
[20:00:51] - Preparing to get new work unit...
[20:00:51] Cleaning up work directory
[20:00:51] + Attempting to get work packet
[20:00:51] - Will indicate memory of 32743 MB
[20:00:51] Gpu type=3 species=40.
[20:00:51] - Detect CPU.[20:00:51] - Autosending finished units... [October 25 20:00:51 UTC]
 Vendor: GenuineIntel, Family: 6, Model: 13, Stepping: 7
[20:00:51] - Connecting to assignment server
[20:00:51] Connecting to http://assign-GPU.stanford.edu:8080/
[20:00:51] Trying to send all finished work units
[20:00:51] + No unsent completed units remaining.
[20:00:51] - Autosend completed
[20:00:52] Posted data.
[20:00:52] Initial: 40AB; - Successful: assigned to (171.64.65.105).
[20:00:52] + News From Folding@Home: Welcome to Folding@Home
[20:00:52] Loaded queue successfully.
[20:00:52] Gpu type=3 species=40.
[20:00:52] Empty passkey
[20:00:52] Connecting to http://171.64.65.105:8080/
[20:00:52] Posted data.
[20:00:52] Initial: 0000; - Receiving payload (expected size: 118539)
[20:00:53] - Downloaded at ~115 kB/s
[20:00:53] - Averaged speed for that direction ~115 kB/s
[20:00:53] + Received work.
[20:00:53] + Closed connections
[20:00:53]
[20:00:53] + Processing work unit
[20:00:53] Core required: FahCore_15.exe
[20:00:53] Core found.
[20:00:53] Working on queue slot 01 [October 25 20:00:53 UTC]
[20:00:53] + Working ...
[20:00:53] - Calling '.\FahCore_15.exe -dir work/ -suffix 01 -nice 19 -priority 96 -nocpulock -checkpoint 3 -verbose -lifeline 7560 -version 630'

[20:00:53]
[20:00:53] *------------------------------*
[20:00:53] Folding@Home GPU Core
[20:00:53] Version                2.25 (Wed May 9 17:03:01 EDT 2012)
[20:00:53] Build host             AmoebaRemote
[20:00:53] Board Type             NVIDIA/CUDA
[20:00:53] Core                   15
[20:00:53]
[20:00:53] Window's signal control handler registered.
[20:00:53] Preparing to commence simulation
[20:00:53] - Looking at optimizations...
[20:00:53] DeleteFrameFiles: successfully deleted file=work/wudata_01.ckp
[20:00:53] - Created dyn
[20:00:53] - Files status OK
[20:00:53] sizeof(CORE_PACKET_HDR) = 512 file=<>
[20:00:53] - Expanded 118027 -> 501826 (decompressed 425.1 percent)
[20:00:53] Called DecompressByteArray: compressed_data_size=118027 data_size=501826, decompressed_data_size=501826 diff=0
[20:00:53] - Digital signature verified
[20:00:53]
[20:00:53] Project: 7623 (Run 661, Clone 1, Gen 0)
[20:00:53]
[20:00:53] Assembly optimizations on if available.
[20:00:53] Entering M.D.
[20:00:55] Tpr hash work/wudata_01.tpr:  656191293 3420210527 850023636 1984152685 3354270320
[20:00:55] GPU device id=0
[20:00:55] Working on Protein
[20:00:55] Client config found, loading data.
[20:00:55] Starting GUI Server
[20:01:57] Setting checkpoint frequency: 400000
[20:01:57] Completed         3 out of 40000000 steps (0%).
[20:06:26] Completed    400000 out of 40000000 steps (1%).
[20:06:26] mdrun_gpu returned 52
[20:06:26] NANs detected on GPU
[20:06:26]
[20:06:26] Folding@home Core Shutdown: UNSTABLE_MACHINE
[20:06:29] CoreStatus = 7A (122)
[20:06:29] Sending work to server
[20:06:29] Project: 7623 (Run 661, Clone 1, Gen 0)
[20:06:29] - Read packet limit of 540015616... Set to 524286976.
[20:06:29] - Error: Could not get length of results file work/wuresults_01.dat
[20:06:29] - Error: Could not read unit 01 file. Removing from queue.
[20:06:29] Trying to send all finished work units
[20:06:29] + No unsent completed units remaining.
[20:06:29] + -oneunit flag given and have now finished a unit. Exiting.***** Got a SIGTERM signal (2)
[20:06:29] Killing all core threads

Folding@Home Client Shutdown.


Code: Select all


--- Opening Log file [October 25 16:15:56 UTC]


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.30r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Users\Matt\Desktop\FAH GPU Tracker V2\GPU0
Executable: C:\Users\Matt\Desktop\FAH GPU Tracker V2\FAH_GPU3.exe
Arguments: -oneunit -forcegpu nvidia_fermi -advmethods -verbosity 9 -gpu 0

[16:15:56] - Ask before connecting: No
[16:15:56] - User name: MattLowe (Team 50959)
[16:15:56] - User ID: 4E9F93E95F93C178
[16:15:56] - Machine ID: 3
[16:15:56]
[16:15:56] Gpu type=3 species=40.
[16:15:56] Could not open work queue, generating new queue...
[16:15:56] - Preparing to get new work unit...
[16:15:56] Cleaning up work directory
[16:15:56] - Autosending finished units... [October 25 16:15:56 UTC]
[16:15:56] Trying to send all finished work units
[16:15:56] + No unsent completed units remaining.
[16:15:56] - Autosend completed
[16:15:56] + Attempting to get work packet
[16:15:56] - Will indicate memory of 32743 MB
[16:15:56] Gpu type=3 species=40.
[16:15:56] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 13, Stepping: 7
[16:15:56] - Connecting to assignment server
[16:15:56] Connecting to http://assign-GPU.stanford.edu:8080/
[16:15:57] Posted data.
[16:15:57] Initial: 40AB; - Successful: assigned to (171.64.65.105).
[16:15:57] + News From Folding@Home: Welcome to Folding@Home
[16:15:57] Loaded queue successfully.
[16:15:57] Gpu type=3 species=40.
[16:15:57] Empty passkey
[16:15:57] Connecting to http://171.64.65.105:8080/
[16:15:58] Posted data.
[16:15:58] Initial: 0000; - Receiving payload (expected size: 118539)
[16:15:58] Conversation time very short, giving reduced weight in bandwidth avg
[16:15:58] - Downloaded at ~231 kB/s
[16:15:58] - Averaged speed for that direction ~231 kB/s
[16:15:58] + Received work.
[16:15:58] + Closed connections
[16:15:58]
[16:15:58] + Processing work unit
[16:15:58] Core required: FahCore_15.exe
[16:15:58] Core found.
[16:15:58] Working on queue slot 01 [October 25 16:15:58 UTC]
[16:15:58] + Working ...
[16:15:58] - Calling '.\FahCore_15.exe -dir work/ -suffix 01 -nice 19 -priority 96 -nocpulock -checkpoint 3 -verbose -lifeline 10404 -version 630'

[16:21:37] CoreStatus = 7A (122)
[16:21:37] Sending work to server
[16:21:37] Project: 7623 (Run 661, Clone 1, Gen 0)
[16:21:37] - Read packet limit of 540015616... Set to 524286976.
[16:21:37] - Error: Could not get length of results file work/wuresults_01.dat
[16:21:37] - Error: Could not read unit 01 file. Removing from queue.
[16:21:37] Trying to send all finished work units
[16:21:37] + No unsent completed units remaining.
[16:21:37] + -oneunit flag given and have now finished a unit. Exiting.***** Got a SIGTERM signal (2)
[16:21:37] Killing all core threads

Folding@Home Client Shutdown.


--- Opening Log file [October 25 16:21:37 UTC]


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.30r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Users\Matt\Desktop\FAH GPU Tracker V2\GPU0
Executable: C:\Users\Matt\Desktop\FAH GPU Tracker V2\FAH_GPU3.exe
Arguments: -oneunit -forcegpu nvidia_fermi -advmethods -verbosity 9 -gpu 0

[16:21:37] - Ask before connecting: No
[16:21:37] - User name: MattLowe (Team 50959)
[16:21:37] - User ID: 4E9F93E95F93C178
[16:21:37] - Machine ID: 3
[16:21:37]
[16:21:37] Gpu type=3 species=40.
[16:21:37] Loaded queue successfully.
[16:21:37] - Preparing to get new work unit...
[16:21:37] Cleaning up work directory
[16:21:37] + Attempting to get work packet
[16:21:37] - Autosending finished units... [October 25 16:21:37 UTC]
[16:21:37] - Will indicate memory of 32743 MB
[16:21:37] Trying to send all finished work units
[16:21:37] Gpu type=3 species=40.
[16:21:37] + No unsent completed units remaining.
[16:21:37] - Detect CPU.[16:21:37] - Autosend completed
 Vendor: GenuineIntel, Family: 6, Model: 13, Stepping: 7
[16:21:37] - Connecting to assignment server
[16:21:37] Connecting to http://assign-GPU.stanford.edu:8080/
[16:21:38] Posted data.
[16:21:38] Initial: 40AB; - Successful: assigned to (171.64.65.105).
[16:21:38] + News From Folding@Home: Welcome to Folding@Home
[16:21:38] Loaded queue successfully.
[16:21:38] Gpu type=3 species=40.
[16:21:38] Empty passkey
[16:21:38] Connecting to http://171.64.65.105:8080/
[16:21:38] Posted data.
[16:21:38] Initial: 0000; - Receiving payload (expected size: 118539)
[16:21:39] - Downloaded at ~115 kB/s
[16:21:39] - Averaged speed for that direction ~173 kB/s
[16:21:39] + Received work.
[16:21:39] + Closed connections
[16:21:39]
[16:21:39] + Processing work unit
[16:21:39] Core required: FahCore_15.exe
[16:21:39] Core found.
[16:21:39] Working on queue slot 02 [October 25 16:21:39 UTC]
[16:21:39] + Working ...
[16:21:39] - Calling '.\FahCore_15.exe -dir work/ -suffix 02 -nice 19 -priority 96 -nocpulock -checkpoint 3 -verbose -lifeline 4232 -version 630'

[16:21:39]
[16:21:39] *------------------------------*
[16:21:39] Folding@Home GPU Core
[16:21:39] Version                2.25 (Wed May 9 17:03:01 EDT 2012)
[16:21:39] Build host             AmoebaRemote
[16:21:39] Board Type             NVIDIA/CUDA
[16:21:39] Core                   15
[16:21:39]
[16:21:39] Window's signal control handler registered.
[16:21:39] Preparing to commence simulation
[16:21:39] - Looking at optimizations...
[16:21:39] DeleteFrameFiles: successfully deleted file=work/wudata_02.ckp
[16:21:39] - Created dyn
[16:21:39] - Files status OK
[16:21:39] sizeof(CORE_PACKET_HDR) = 512 file=<>
[16:21:39] - Expanded 118027 -> 501826 (decompressed 425.1 percent)
[16:21:39] Called DecompressByteArray: compressed_data_size=118027 data_size=501826, decompressed_data_size=501826 diff=0
[16:21:39] - Digital signature verified
[16:21:39]
[16:21:39] Project: 7623 (Run 661, Clone 1, Gen 0)
[16:21:39]
[16:21:39] Assembly optimizations on if available.
[16:21:39] Entering M.D.
[16:21:41] Tpr hash work/wudata_02.tpr:  656191293 3420210527 850023636 1984152685 3354270320
[16:21:41] GPU device id=0
[16:21:41] Working on Protein
[16:21:41] Client config found, loading data.
[16:21:42] Starting GUI Server
[16:22:43] Setting checkpoint frequency: 400000
[16:22:43] Completed         3 out of 40000000 steps (0%).
[16:27:23] Completed    400000 out of 40000000 steps (1%).
[16:27:23] mdrun_gpu returned 52
[16:27:23] NANs detected on GPU
[16:27:23]
[16:27:23] Folding@home Core Shutdown: UNSTABLE_MACHINE
[16:27:25] CoreStatus = 7A (122)
[16:27:25] Sending work to server
[16:27:25] Project: 7623 (Run 661, Clone 1, Gen 0)
[16:27:25] - Read packet limit of 540015616... Set to 524286976.
[16:27:25] - Error: Could not get length of results file work/wuresults_02.dat
[16:27:25] - Error: Could not read unit 02 file. Removing from queue.
[16:27:25] Trying to send all finished work units
[16:27:25] + No unsent completed units remaining.
[16:27:25] + -oneunit flag given and have now finished a unit. Exiting.***** Got a SIGTERM signal (2)
[16:27:25] Killing all core threads

Folding@Home Client Shutdown.


Code: Select all


--- Opening Log file [October 25 10:18:37 UTC]


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.30r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Users\Matt\Desktop\FAH GPU Tracker V2\GPU0
Executable: C:\Users\Matt\Desktop\FAH GPU Tracker V2\FAH_GPU3.exe
Arguments: -oneunit -forcegpu nvidia_fermi -advmethods -verbosity 9 -gpu 0

[10:18:37] - Ask before connecting: No
[10:18:37] - User name: MattLowe (Team 50959)
[10:18:37] - User ID: 4E9F93E95F93C178
[10:18:37] - Machine ID: 3
[10:18:37]
[10:18:37] Gpu type=3 species=40.
[10:18:37] Could not open work queue, generating new queue...
[10:18:37] - Preparing to get new work unit...
[10:18:37] Cleaning up work directory
[10:18:37] - Autosending finished units... [October 25 10:18:37 UTC]
[10:18:37] Trying to send all finished work units
[10:18:37] + No unsent completed units remaining.
[10:18:37] - Autosend completed
[10:18:37] + Attempting to get work packet
[10:18:37] - Will indicate memory of 32743 MB
[10:18:37] Gpu type=3 species=40.
[10:18:37] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 13, Stepping: 7
[10:18:37] - Connecting to assignment server
[10:18:37] Connecting to http://assign-GPU.stanford.edu:8080/
[10:18:39] Posted data.
[10:18:39] Initial: 40AB; - Successful: assigned to (171.64.65.105).
[10:18:39] + News From Folding@Home: Welcome to Folding@Home
[10:18:40] Loaded queue successfully.
[10:18:40] Gpu type=3 species=40.
[10:18:40] Empty passkey
[10:18:40] Connecting to http://171.64.65.105:8080/
[10:18:41] Posted data.
[10:18:41] Initial: 0000; - Receiving payload (expected size: 118539)
[10:18:55] - Downloaded at ~8 kB/s
[10:18:55] - Averaged speed for that direction ~8 kB/s
[10:18:55] + Received work.
[10:18:55] + Closed connections
[10:18:55]
[10:18:55] + Processing work unit
[10:18:55] Core required: FahCore_15.exe
[10:18:55] Core found.
[10:18:55] Working on queue slot 01 [October 25 10:18:55 UTC]
[10:18:55] + Working ...
[10:18:55] - Calling '.\FahCore_15.exe -dir work/ -suffix 01 -nice 19 -priority 96 -nocpulock -checkpoint 3 -verbose -lifeline 10256 -version 630'

[10:24:34] CoreStatus = 7A (122)
[10:24:34] Sending work to server
[10:24:34] Project: 7623 (Run 661, Clone 1, Gen 0)
[10:24:34] - Read packet limit of 540015616... Set to 524286976.
[10:24:34] - Error: Could not get length of results file work/wuresults_01.dat
[10:24:34] - Error: Could not read unit 01 file. Removing from queue.
[10:24:34] Trying to send all finished work units
[10:24:34] + No unsent completed units remaining.
[10:24:34] + -oneunit flag given and have now finished a unit. Exiting.***** Got a SIGTERM signal (2)
[10:24:34] Killing all core threads

Folding@Home Client Shutdown.


--- Opening Log file [October 25 10:24:34 UTC]


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.30r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Users\Matt\Desktop\FAH GPU Tracker V2\GPU0
Executable: C:\Users\Matt\Desktop\FAH GPU Tracker V2\FAH_GPU3.exe
Arguments: -oneunit -forcegpu nvidia_fermi -advmethods -verbosity 9 -gpu 0

[10:24:34] - Ask before connecting: No
[10:24:34] - User name: MattLowe (Team 50959)
[10:24:34] - User ID: 4E9F93E95F93C178
[10:24:34] - Machine ID: 3
[10:24:34]
[10:24:34] Gpu type=3 species=40.
[10:24:34] Loaded queue successfully.
[10:24:34] - Preparing to get new work unit...
[10:24:34] - Autosending finished units... [October 25 10:24:34 UTC]
[10:24:34] Cleaning up work directory
[10:24:34] Trying to send all finished work units
[10:24:34] + No unsent completed units remaining.
[10:24:34] - Autosend completed
[10:24:34] + Attempting to get work packet
[10:24:34] - Will indicate memory of 32743 MB
[10:24:34] Gpu type=3 species=40.
[10:24:34] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 13, Stepping: 7
[10:24:34] - Connecting to assignment server
[10:24:34] Connecting to http://assign-GPU.stanford.edu:8080/
[10:24:36] Posted data.
[10:24:36] Initial: 40AB; - Successful: assigned to (171.64.65.105).
[10:24:36] + News From Folding@Home: Welcome to Folding@Home
[10:24:36] Loaded queue successfully.
[10:24:36] Gpu type=3 species=40.
[10:24:36] Empty passkey
[10:24:36] Connecting to http://171.64.65.105:8080/
[10:24:37] Posted data.
[10:24:37] Initial: 0000; - Receiving payload (expected size: 118539)
[10:24:43] - Downloaded at ~19 kB/s
[10:24:43] - Averaged speed for that direction ~13 kB/s
[10:24:43] + Received work.
[10:24:43] + Closed connections
[10:24:43]
[10:24:43] + Processing work unit
[10:24:43] Core required: FahCore_15.exe
[10:24:43] Core found.
[10:24:43] Working on queue slot 02 [October 25 10:24:43 UTC]
[10:24:43] + Working ...
[10:24:43] - Calling '.\FahCore_15.exe -dir work/ -suffix 02 -nice 19 -priority 96 -nocpulock -checkpoint 3 -verbose -lifeline 11212 -version 630'

[10:24:43]
[10:24:43] *------------------------------*
[10:24:43] Folding@Home GPU Core
[10:24:43] Version                2.25 (Wed May 9 17:03:01 EDT 2012)
[10:24:43] Build host             AmoebaRemote
[10:24:43] Board Type             NVIDIA/CUDA
[10:24:43] Core                   15
[10:24:43]
[10:24:43] Window's signal control handler registered.
[10:24:43] Preparing to commence simulation
[10:24:43] - Looking at optimizations...
[10:24:43] DeleteFrameFiles: successfully deleted file=work/wudata_02.ckp
[10:24:43] - Created dyn
[10:24:43] - Files status OK
[10:24:43] sizeof(CORE_PACKET_HDR) = 512 file=<>
[10:24:43] - Expanded 118027 -> 501826 (decompressed 425.1 percent)
[10:24:43] Called DecompressByteArray: compressed_data_size=118027 data_size=501826, decompressed_data_size=501826 diff=0
[10:24:43] - Digital signature verified
[10:24:43]
[10:24:43] Project: 7623 (Run 661, Clone 1, Gen 0)
[10:24:43]
[10:24:43] Assembly optimizations on if available.
[10:24:43] Entering M.D.
[10:24:45] Tpr hash work/wudata_02.tpr:  656191293 3420210527 850023636 1984152685 3354270320
[10:24:45] GPU device id=0
[10:24:45] Working on Protein
[10:24:45] Client config found, loading data.
[10:24:46] Starting GUI Server
[10:25:48] Setting checkpoint frequency: 400000
[10:25:48] Completed         3 out of 40000000 steps (0%).
[10:30:14] Completed    400000 out of 40000000 steps (1%).
[10:30:15] mdrun_gpu returned 52
[10:30:15] NANs detected on GPU
[10:30:15]
[10:30:15] Folding@home Core Shutdown: UNSTABLE_MACHINE
[10:30:18] CoreStatus = 7A (122)
[10:30:18] Sending work to server
[10:30:18] Project: 7623 (Run 661, Clone 1, Gen 0)
[10:30:18] - Read packet limit of 540015616... Set to 524286976.
[10:30:18] - Error: Could not get length of results file work/wuresults_02.dat
[10:30:18] - Error: Could not read unit 02 file. Removing from queue.
[10:30:18] Trying to send all finished work units
[10:30:18] + No unsent completed units remaining.
[10:30:18] + -oneunit flag given and have now finished a unit. Exiting.***** Got a SIGTERM signal (2)
[10:30:18] Killing all core threads

Folding@Home Client Shutdown.


Code: Select all


--- Opening Log file [October 25 00:45:54 UTC]


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.30r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Users\Matt\Desktop\FAH GPU Tracker V2\GPU0
Executable: C:\Users\Matt\Desktop\FAH GPU Tracker V2\FAH_GPU3.exe
Arguments: -oneunit -forcegpu nvidia_fermi -advmethods -verbosity 9 -gpu 0

[00:45:54] - Ask before connecting: No
[00:45:54] - User name: MattLowe (Team 50959)
[00:45:54] - User ID: 4E9F93E95F93C178
[00:45:54] - Machine ID: 3
[00:45:54]
[00:45:54] Gpu type=3 species=40.
[00:45:54] Could not open work queue, generating new queue...
[00:45:54] - Preparing to get new work unit...
[00:45:54] Cleaning up work directory
[00:45:54] - Autosending finished units... [October 25 00:45:54 UTC]
[00:45:54] Trying to send all finished work units
[00:45:54] + No unsent completed units remaining.
[00:45:54] - Autosend completed
[00:45:54] + Attempting to get work packet
[00:45:54] - Will indicate memory of 32743 MB
[00:45:54] Gpu type=3 species=40.
[00:45:54] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 13, Stepping: 7
[00:45:54] - Connecting to assignment server
[00:45:54] Connecting to http://assign-GPU.stanford.edu:8080/
[00:45:56] Posted data.
[00:45:56] Initial: 40AB; - Successful: assigned to (171.64.65.105).
[00:45:56] + News From Folding@Home: Welcome to Folding@Home
[00:45:56] Loaded queue successfully.
[00:45:56] Gpu type=3 species=40.
[00:45:56] Empty passkey
[00:45:56] Connecting to http://171.64.65.105:8080/
[00:46:00] Posted data.
[00:46:00] Initial: 0000; - Receiving payload (expected size: 118539)
[00:46:08] - Downloaded at ~14 kB/s
[00:46:08] - Averaged speed for that direction ~14 kB/s
[00:46:08] + Received work.
[00:46:08] + Closed connections
[00:46:08]
[00:46:08] + Processing work unit
[00:46:08] Core required: FahCore_15.exe
[00:46:08] Core found.
[00:46:08] Working on queue slot 01 [October 25 00:46:08 UTC]
[00:46:08] + Working ...
[00:46:08] - Calling '.\FahCore_15.exe -dir work/ -suffix 01 -nice 19 -priority 96 -nocpulock -checkpoint 3 -verbose -lifeline 12100 -version 630'

[00:51:45] CoreStatus = 7A (122)
[00:51:45] Sending work to server
[00:51:45] Project: 7623 (Run 661, Clone 1, Gen 0)
[00:51:45] - Read packet limit of 540015616... Set to 524286976.
[00:51:45] - Error: Could not get length of results file work/wuresults_01.dat
[00:51:45] - Error: Could not read unit 01 file. Removing from queue.
[00:51:45] Trying to send all finished work units
[00:51:45] + No unsent completed units remaining.
[00:51:45] + -oneunit flag given and have now finished a unit. Exiting.***** Got a SIGTERM signal (2)
[00:51:45] Killing all core threads

Folding@Home Client Shutdown.


--- Opening Log file [October 25 00:51:45 UTC]


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.30r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Users\Matt\Desktop\FAH GPU Tracker V2\GPU0
Executable: C:\Users\Matt\Desktop\FAH GPU Tracker V2\FAH_GPU3.exe
Arguments: -oneunit -forcegpu nvidia_fermi -advmethods -verbosity 9 -gpu 0

[00:51:45] - Ask before connecting: No
[00:51:45] - User name: MattLowe (Team 50959)
[00:51:45] - User ID: 4E9F93E95F93C178
[00:51:45] - Machine ID: 3
[00:51:45]
[00:51:45] Gpu type=3 species=40.
[00:51:45] Loaded queue successfully.
[00:51:45] - Preparing to get new work unit...
[00:51:45] - Autosending finished units... [October 25 00:51:45 UTC]
[00:51:45] Cleaning up work directory
[00:51:45] Trying to send all finished work units
[00:51:45] + No unsent completed units remaining.
[00:51:45] - Autosend completed
[00:51:45] + Attempting to get work packet
[00:51:45] - Will indicate memory of 32743 MB
[00:51:45] Gpu type=3 species=40.
[00:51:45] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 13, Stepping: 7
[00:51:45] - Connecting to assignment server
[00:51:45] Connecting to http://assign-GPU.stanford.edu:8080/
[00:51:47] Posted data.
[00:51:47] Initial: 40AB; - Successful: assigned to (171.64.65.105).
[00:51:47] + News From Folding@Home: Welcome to Folding@Home
[00:51:47] Loaded queue successfully.
[00:51:47] Gpu type=3 species=40.
[00:51:47] Empty passkey
[00:51:47] Connecting to http://171.64.65.105:8080/
[00:51:48] Posted data.
[00:51:48] Initial: 0000; - Receiving payload (expected size: 118539)
[00:51:52] - Downloaded at ~28 kB/s
[00:51:52] - Averaged speed for that direction ~21 kB/s
[00:51:52] + Received work.
[00:51:52] + Closed connections
[00:51:52]
[00:51:52] + Processing work unit
[00:51:52] Core required: FahCore_15.exe
[00:51:52] Core found.
[00:51:52] Working on queue slot 02 [October 25 00:51:52 UTC]
[00:51:52] + Working ...
[00:51:52] - Calling '.\FahCore_15.exe -dir work/ -suffix 02 -nice 19 -priority 96 -nocpulock -checkpoint 3 -verbose -lifeline 5632 -version 630'

[00:51:52]
[00:51:52] *------------------------------*
[00:51:52] Folding@Home GPU Core
[00:51:52] Version                2.25 (Wed May 9 17:03:01 EDT 2012)
[00:51:52] Build host             AmoebaRemote
[00:51:52] Board Type             NVIDIA/CUDA
[00:51:52] Core                   15
[00:51:52]
[00:51:52] Window's signal control handler registered.
[00:51:52] Preparing to commence simulation
[00:51:52] - Looking at optimizations...
[00:51:52] DeleteFrameFiles: successfully deleted file=work/wudata_02.ckp
[00:51:52] - Created dyn
[00:51:52] - Files status OK
[00:51:52] sizeof(CORE_PACKET_HDR) = 512 file=<>
[00:51:52] - Expanded 118027 -> 501826 (decompressed 425.1 percent)
[00:51:52] Called DecompressByteArray: compressed_data_size=118027 data_size=501826, decompressed_data_size=501826 diff=0
[00:51:52] - Digital signature verified
[00:51:52]
[00:51:52] Project: 7623 (Run 661, Clone 1, Gen 0)
[00:51:52]
[00:51:52] Assembly optimizations on if available.
[00:51:52] Entering M.D.
[00:51:54] Tpr hash work/wudata_02.tpr:  656191293 3420210527 850023636 1984152685 3354270320
[00:51:54] GPU device id=0
[00:51:54] Working on Protein
[00:51:54] Client config found, loading data.
[00:51:54] Starting GUI Server
[00:52:56] Setting checkpoint frequency: 400000
[00:52:56] Completed         3 out of 40000000 steps (0%).
[00:57:28] Completed    400000 out of 40000000 steps (1%).
[00:57:28] mdrun_gpu returned 52
[00:57:28] NANs detected on GPU
[00:57:28]
[00:57:28] Folding@home Core Shutdown: UNSTABLE_MACHINE
[00:57:32] CoreStatus = 7A (122)
[00:57:32] Sending work to server
[00:57:32] Project: 7623 (Run 661, Clone 1, Gen 0)
[00:57:32] - Read packet limit of 540015616... Set to 524286976.
[00:57:32] - Error: Could not get length of results file work/wuresults_02.dat
[00:57:32] - Error: Could not read unit 02 file. Removing from queue.
[00:57:32] Trying to send all finished work units
[00:57:32] + No unsent completed units remaining.
[00:57:32] + -oneunit flag given and have now finished a unit. Exiting.***** Got a SIGTERM signal (2)
[00:57:32] Killing all core threads

Folding@Home Client Shutdown.


Code: Select all


--- Opening Log file [October 24 14:45:05 UTC]


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.30r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Users\Matt\Desktop\FAH GPU Tracker V2\GPU0
Executable: C:\Users\Matt\Desktop\FAH GPU Tracker V2\FAH_GPU3.exe
Arguments: -oneunit -forcegpu nvidia_fermi -advmethods -verbosity 9 -gpu 0

[14:45:05] - Ask before connecting: No
[14:45:05] - User name: MattLowe (Team 50959)
[14:45:05] - User ID: 4E9F93E95F93C178
[14:45:05] - Machine ID: 3
[14:45:05]
[14:45:05] Gpu type=3 species=40.
[14:45:05] Work directory not found. Creating...
[14:45:05] Could not open work queue, generating new queue...
[14:45:05] - Preparing to get new work unit...
[14:45:05] Cleaning up work directory
[14:45:05] - Autosending finished units... [October 24 14:45:05 UTC]
[14:45:05] Trying to send all finished work units
[14:45:05] + Attempting to get work packet
[14:45:05] + No unsent completed units remaining.
[14:45:05] - Will indicate memory of 32743 MB
[14:45:05] - Autosend completed
[14:45:05] Gpu type=3 species=40.
[14:45:05] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 13, Stepping: 7
[14:45:05] - Connecting to assignment server
[14:45:05] Connecting to http://assign-GPU.stanford.edu:8080/
[14:45:06] Posted data.
[14:45:06] Initial: 40AB; - Successful: assigned to (171.64.65.105).
[14:45:06] + News From Folding@Home: Welcome to Folding@Home
[14:45:06] Loaded queue successfully.
[14:45:06] Gpu type=3 species=40.
[14:45:06] Empty passkey
[14:45:06] Connecting to http://171.64.65.105:8080/
[14:45:07] Posted data.
[14:45:07] Initial: 0000; - Receiving payload (expected size: 118539)
[14:45:07] Conversation time very short, giving reduced weight in bandwidth avg
[14:45:07] - Downloaded at ~231 kB/s
[14:45:07] - Averaged speed for that direction ~231 kB/s
[14:45:07] + Received work.
[14:45:07] + Closed connections
[14:45:07]
[14:45:07] + Processing work unit
[14:45:07] Core required: FahCore_15.exe
[14:45:07] Core found.
[14:45:07] Working on queue slot 01 [October 24 14:45:07 UTC]
[14:45:07] + Working ...
[14:45:07] - Calling '.\FahCore_15.exe -dir work/ -suffix 01 -nice 19 -priority 96 -nocpulock -checkpoint 3 -verbose -lifeline 11956 -version 630'

[14:45:08]
[14:45:08] *------------------------------*
[14:45:08] Folding@Home GPU Core
[14:45:08] Version                2.25 (Wed May 9 17:03:01 EDT 2012)
[14:45:08] Build host             AmoebaRemote
[14:45:08] Board Type             NVIDIA/CUDA
[14:45:08] Core                   15
[14:45:08]
[14:45:08] Window's signal control handler registered.
[14:45:08] Preparing to commence simulation
[14:45:08] - Looking at optimizations...
[14:45:08] DeleteFrameFiles: successfully deleted file=work/wudata_01.ckp
[14:45:08] - Created dyn
[14:45:08] - Files status OK
[14:45:08] sizeof(CORE_PACKET_HDR) = 512 file=<>
[14:45:08] - Expanded 118027 -> 501826 (decompressed 425.1 percent)
[14:45:08] Called DecompressByteArray: compressed_data_size=118027 data_size=501826, decompressed_data_size=501826 diff=0
[14:45:08] - Digital signature verified
[14:45:08]
[14:45:08] Project: 7623 (Run 661, Clone 1, Gen 0)
[14:45:08]
[14:45:08] Assembly optimizations on if available.
[14:45:08] Entering M.D.
[14:45:10] Tpr hash work/wudata_01.tpr:  656191293 3420210527 850023636 1984152685 3354270320
[14:45:10] GPU device id=0
[14:45:10] Working on Protein
[14:45:10] Client config found, loading data.
[14:45:10] Starting GUI Server
[14:46:12] Setting checkpoint frequency: 400000
[14:46:12] Completed         3 out of 40000000 steps (0%).
[14:50:42] Completed    400000 out of 40000000 steps (1%).
[14:50:42] mdrun_gpu returned 52
[14:50:42] NANs detected on GPU
[14:50:42]
[14:50:42] Folding@home Core Shutdown: UNSTABLE_MACHINE
[14:50:44] CoreStatus = 7A (122)
[14:50:44] Sending work to server
[14:50:44] Project: 7623 (Run 661, Clone 1, Gen 0)
[14:50:44] - Read packet limit of 540015616... Set to 524286976.
[14:50:44] - Error: Could not get length of results file work/wuresults_01.dat
[14:50:44] - Error: Could not read unit 01 file. Removing from queue.
[14:50:44] Trying to send all finished work units
[14:50:44] + No unsent completed units remaining.
[14:50:44] + -oneunit flag given and have now finished a unit. Exiting.***** Got a SIGTERM signal (2)
[14:50:44] Killing all core threads

Folding@Home Client Shutdown.


Here are a few random logs I grabbed from the last few days... any ideas? I've updated the Nvidia driver from 301 to 306. I installed windows updates. I deleted the cores and downloaded new ones. Still can't figure it out.

[bruce]

You're running a V6 client and you're posting in a topic for the V7 client.

Update your client to V7 -- preferably V7.2.9.

[7im]

Any idea if the new v2.25 GPU fahcore still supports v6 switches like -forcegpu nvidia_fermi? Or does that stuff get dropped, assuming newer hardware would typically run newer clients?

[bruce]

FahCores don't recognize that data.

[mattlowe01`]

I can't get the V7.2.9 to correctly identify my GPU's. It lists

GPU:0:G92 [GeForce GTS 250]
GPU:1:G92 [GeFOrce GTS 250]
GPU:2:GK107 [GeForce GTX 670]

Cuda-Z and GPU-Z list them in the order of

GPU:0 GTX 670
GPU:1 GTS 250
GPU:2 GTS 250

If I manually change the GTX 670 gpu-index to 0, it changes the description to GTS-250 and won't fold with Cuda-Index 0,1,2, or 3. If I leave it with Gpu-index 2, it reports as a GTX-670, but still won't fold with cuda-index as 0,1,2,3. Here's a log file with gpu-index -1, cuda index 0

Code: Select all

*********************** Log Started 2012-10-26T02:03:13Z ***********************
02:03:13:WU01:FS00:Starting
02:03:13:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Matt/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 01 -suffix 01 -version 702 -lifeline 12272 -checkpoint 15 -gpu 0
02:03:13:WU01:FS00:Started FahCore on PID 11732
02:03:13:WU01:FS00:Core PID:4412
02:03:13:WU01:FS00:FahCore 0x15 started
02:03:14:WU01:FS00:0x15:
02:03:14:WU01:FS00:0x15:*------------------------------*
02:03:14:WU01:FS00:0x15:Folding@Home GPU Core
02:03:14:WU01:FS00:0x15:Version                2.22 (Thu Dec 8 17:08:05 PST 2011)
02:03:14:WU01:FS00:0x15:Build host             SimbiosNvdWin7
02:03:14:WU01:FS00:0x15:Board Type             NVIDIA/CUDA
02:03:14:WU01:FS00:0x15:Core                   15
02:03:14:WU01:FS00:0x15:
02:03:14:WU01:FS00:0x15:Window's signal control handler registered.
02:03:14:WU01:FS00:0x15:Preparing to commence simulation
02:03:14:WU01:FS00:0x15:- Ensuring status. Please wait.
02:03:23:WU01:FS00:0x15:- Looking at optimizations...
02:03:23:WU01:FS00:0x15:- Working with standard loops on this execution.
02:03:23:WU01:FS00:0x15:- Previous termination of core was improper.
02:03:23:WU01:FS00:0x15:- Files status OK
02:03:23:WU01:FS00:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
02:03:23:WU01:FS00:0x15:- Expanded 60221 -> 264278 (decompressed 438.8 percent)
02:03:23:WU01:FS00:0x15:Called DecompressByteArray: compressed_data_size=60221 data_size=264278, decompressed_data_size=264278 diff=0
02:03:23:WU01:FS00:0x15:- Digital signature verified
02:03:23:WU01:FS00:0x15:
02:03:23:WU01:FS00:0x15:Project: 8054 (Run 0, Clone 1096, Gen 13)
02:03:23:WU01:FS00:0x15:
02:03:23:WU01:FS00:0x15:Entering M.D.
02:03:25:WU01:FS00:0x15:Tpr hash 01/wudata_01.tpr:  813274246 585860560 476112754 1095487504 2072512956
02:03:25:WU01:FS00:0x15:GPU device info: vendor=0 device=0 name=<NA> match=0
02:03:25:WU01:FS00:0x15:Working on Good ROcking Metal Altar for Chronical Sinners
02:03:25:WU01:FS00:0x15:Client config unavailable.
02:03:25:WU01:FS00:0x15:Starting GUI Server


Here's the log files of both gts 250's currently folding and working

Code: Select all

*********************** Log Started 2012-10-26T02:03:13Z ***********************
02:03:14:WU00:FS01:Cleaning up
02:03:14:WU00:FS01:Connecting to assign-GPU.stanford.edu:80
02:03:15:WU00:FS01:News: Welcome to Folding@Home
02:03:15:WU00:FS01:Assigned to work server 171.67.108.21
02:03:15:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:"G92 [GeForce GTS 250]" from 171.67.108.21
02:03:15:WU00:FS01:Connecting to 171.67.108.21:8080
02:03:16:WU00:FS01:Downloading 61.78KiB
02:03:17:WU00:FS01:Download complete
02:03:17:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:10503 run:294 clone:0 gen:577 core:0x11 unit:0x000005a06652eda54b716bfb000072d7
02:03:17:WU00:FS01:Starting
02:03:17:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Matt/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 00 -suffix 01 -version 702 -lifeline 12272 -checkpoint 15 -gpu 2
02:03:17:WU00:FS01:Started FahCore on PID 8632
02:03:17:WU00:FS01:Core PID:6676
02:03:17:WU00:FS01:FahCore 0x11 started
02:03:18:WU00:FS01:0x11:
02:03:18:WU00:FS01:0x11:*------------------------------*
02:03:18:WU00:FS01:0x11:Folding@Home GPU Core
02:03:18:WU00:FS01:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
02:03:18:WU00:FS01:0x11:
02:03:18:WU00:FS01:0x11:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
02:03:18:WU00:FS01:0x11:Build host: amoeba
02:03:18:WU00:FS01:0x11:Board Type: Nvidia
02:03:18:WU00:FS01:0x11:Core      :
02:03:18:WU00:FS01:0x11:Preparing to commence simulation
02:03:18:WU00:FS01:0x11:- Looking at optimizations...
02:03:18:WU00:FS01:0x11:DeleteFrameFiles: successfully deleted file=00/wudata_01.ckp
02:03:18:WU00:FS01:0x11:- Created dyn
02:03:18:WU00:FS01:0x11:- Files status OK
02:03:18:WU00:FS01:0x11:- Expanded 62755 -> 336799 (decompressed 536.6 percent)
02:03:18:WU00:FS01:0x11:Called DecompressByteArray: compressed_data_size=62755 data_size=336799, decompressed_data_size=336799 diff=0
02:03:18:WU00:FS01:0x11:- Digital signature verified
02:03:18:WU00:FS01:0x11:
02:03:18:WU00:FS01:0x11:Project: 10503 (Run 294, Clone 0, Gen 577)
02:03:18:WU00:FS01:0x11:
02:03:18:WU00:FS01:0x11:Assembly optimizations on if available.
02:03:18:WU00:FS01:0x11:Entering M.D.
02:03:24:WU00:FS01:0x11:Tpr hash 00/wudata_01.tpr:  2845377120 1335140037 4193494972 3166974631 488547969
02:03:24:WU00:FS01:0x11:
02:03:24:WU00:FS01:0x11:Calling fah_main args: 14 usage=100
02:03:24:WU00:FS01:0x11:
02:03:24:WU00:FS01:0x11:Working on Protein
02:03:25:WU00:FS01:0x11:Client config unavailable.
02:03:25:WU00:FS01:0x11:Starting GUI Server
02:04:55:WU00:FS01:0x11:Completed 1%
02:06:25:WU00:FS01:0x11:Completed 2%
02:07:55:WU00:FS01:0x11:Completed 3%
02:09:24:WU00:FS01:0x11:Completed 4%
02:11:10:WU00:FS01:0x11:Completed 5%
02:12:45:WU00:FS01:0x11:Completed 6%


Code: Select all

*********************** Log Started 2012-10-26T02:03:13Z ***********************
02:03:13:WU02:FS02:Starting
02:03:13:WU02:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Matt/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 02 -suffix 01 -version 702 -lifeline 12272 -checkpoint 15 -gpu 1
02:03:13:WU02:FS02:Started FahCore on PID 9828
02:03:13:WU02:FS02:Core PID:9808
02:03:14:WU02:FS02:FahCore 0x11 started
02:03:14:WU02:FS02:0x11:
02:03:14:WU02:FS02:0x11:*------------------------------*
02:03:14:WU02:FS02:0x11:Folding@Home GPU Core
02:03:14:WU02:FS02:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
02:03:14:WU02:FS02:0x11:
02:03:14:WU02:FS02:0x11:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
02:03:14:WU02:FS02:0x11:Build host: amoeba
02:03:14:WU02:FS02:0x11:Board Type: Nvidia
02:03:14:WU02:FS02:0x11:Core      :
02:03:14:WU02:FS02:0x11:Preparing to commence simulation
02:03:14:WU02:FS02:0x11:- Looking at optimizations...
02:03:14:WU02:FS02:0x11:- Files status OK
02:03:14:WU02:FS02:0x11:- Expanded 62824 -> 336799 (decompressed 536.0 percent)
02:03:14:WU02:FS02:0x11:Called DecompressByteArray: compressed_data_size=62824 data_size=336799, decompressed_data_size=336799 diff=0
02:03:14:WU02:FS02:0x11:- Digital signature verified
02:03:14:WU02:FS02:0x11:
02:03:14:WU02:FS02:0x11:Project: 10504 (Run 281, Clone 0, Gen 674)
02:03:14:WU02:FS02:0x11:
02:03:14:WU02:FS02:0x11:Assembly optimizations on if available.
02:03:14:WU02:FS02:0x11:Entering M.D.
02:03:20:WU02:FS02:0x11:Will resume from checkpoint file
02:03:20:WU02:FS02:0x11:Tpr hash 02/wudata_01.tpr:  451806238 3008412301 925716413 2554549308 939811750
02:03:20:WU02:FS02:0x11:
02:03:20:WU02:FS02:0x11:Calling fah_main args: 14 usage=100
02:03:20:WU02:FS02:0x11:
02:03:20:WU02:FS02:0x11:Working on Protein
02:03:21:WU02:FS02:0x11:Client config unavailable.
02:03:21:WU02:FS02:0x11:Resuming from checkpoint
02:03:21:WU02:FS02:0x11:fcCheckPointResume: retreived and current tpr file hash:
02:03:21:WU02:FS02:0x11:   0    451806238    451806238
02:03:21:WU02:FS02:0x11:   1   3008412301   3008412301
02:03:21:WU02:FS02:0x11:   2    925716413    925716413
02:03:21:WU02:FS02:0x11:   3   2554549308   2554549308
02:03:21:WU02:FS02:0x11:   4    939811750    939811750
02:03:21:WU02:FS02:0x11:fcCheckPointResume: file hashes same.
02:03:21:WU02:FS02:0x11:fcCheckPointResume: state restored.
02:03:21:WU02:FS02:0x11:Verified 02/wudata_01.log
02:03:21:WU02:FS02:0x11:Verified 02/wudata_01.edr
02:03:21:WU02:FS02:0x11:Verified 02/wudata_01.xtc
02:03:21:WU02:FS02:0x11:Completed 68%
02:03:21:WU02:FS02:0x11:Starting GUI Server
02:04:50:WU02:FS02:0x11:Completed 69%
02:06:19:WU02:FS02:0x11:Completed 70%
02:07:47:WU02:FS02:0x11:Completed 71%
02:09:16:WU02:FS02:0x11:Completed 72%
02:10:45:WU02:FS02:0x11:Completed 73%
02:12:22:WU02:FS02:0x11:Completed 74%


[mattlowe01`]

Okay... So I installed v7.2.9 fresh, to a different location. I let it automatically assign indexs to all of the slots. Here is the first log file from each slot after letting it run for a few minutes. Currently all GPU sensors reporting 0% activity. SMP crunching away.

ID 00 - gpu:0:G92 [GeForce GTS 250]

Code: Select all

*********************** Log Started 2012-10-26T04:29:15Z ***********************
04:29:17:WU00:FS00:Connecting to assign-GPU.stanford.edu:80
04:29:17:WU00:FS00:Connecting to assign-GPU.stanford.edu:80
04:29:17:WU00:FS00:News: Welcome to Folding@Home
04:29:17:WU00:FS00:Assigned to work server 171.67.108.21
04:29:17:WU00:FS00:Requesting new work unit for slot 00: READY gpu:0:"G92 [GeForce GTS 250]" from 171.67.108.21
04:29:17:WU00:FS00:Connecting to 171.67.108.21:8080
04:29:18:WU00:FS00:Downloading 61.93KiB
04:29:19:WU00:FS00:Download complete
04:29:19:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:10504 run:31 clone:1 gen:1219 core:0x11 unit:0x00000a656652eda54b75ae7800001860
04:29:19:WU00:FS00:Downloading project 10504 description
04:29:19:WU00:FS00:Connecting to fah-web.stanford.edu:80
04:29:19:WU00:FS00:Project 10504 description downloaded successfully
04:29:22:WU00:FS00:Starting
04:29:22:WU00:FS00:Running FahCore: C:\Users\Matt\Documents\Folding\FAHClient/FAHCoreWrapper.exe C:/Users/Matt/Documents/Folding/AppData/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 00 -suffix 01 -version 702 -lifeline 1344 -checkpoint 15 -gpu 0
04:29:22:WU00:FS00:Started FahCore on PID 9472
04:29:22:WU00:FS00:Core PID:6476
04:29:22:WU00:FS00:FahCore 0x11 started
04:29:23:WU00:FS00:0x11:
04:29:23:WU00:FS00:0x11:*------------------------------*
04:29:23:WU00:FS00:0x11:Folding@Home GPU Core
04:29:23:WU00:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
04:29:23:WU00:FS00:0x11:
04:29:23:WU00:FS00:0x11:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
04:29:23:WU00:FS00:0x11:Build host: amoeba
04:29:23:WU00:FS00:0x11:Board Type: Nvidia
04:29:23:WU00:FS00:0x11:Core      :
04:29:23:WU00:FS00:0x11:Preparing to commence simulation
04:29:23:WU00:FS00:0x11:- Looking at optimizations...
04:29:23:WU00:FS00:0x11:DeleteFrameFiles: successfully deleted file=00/wudata_01.ckp
04:29:23:WU00:FS00:0x11:- Created dyn
04:29:23:WU00:FS00:0x11:- Files status OK
04:29:23:WU00:FS00:0x11:- Expanded 62901 -> 336799 (decompressed 535.4 percent)
04:29:23:WU00:FS00:0x11:Called DecompressByteArray: compressed_data_size=62901 data_size=336799, decompressed_data_size=336799 diff=0
04:29:23:WU00:FS00:0x11:- Digital signature verified
04:29:23:WU00:FS00:0x11:
04:29:23:WU00:FS00:0x11:Project: 10504 (Run 31, Clone 1, Gen 1219)
04:29:23:WU00:FS00:0x11:
04:29:23:WU00:FS00:0x11:Assembly optimizations on if available.
04:29:23:WU00:FS00:0x11:Entering M.D.
04:29:29:WU00:FS00:0x11:Tpr hash 00/wudata_01.tpr:  2974029469 687552728 4091093292 2264149243 3736431257
04:29:29:WU00:FS00:0x11:
04:29:29:WU00:FS00:0x11:Calling fah_main args: 14 usage=100
04:29:29:WU00:FS00:0x11:


ID 01 - gpu:1:g92 [GeForce GTS 250]

Code: Select all

*********************** Log Started 2012-10-26T04:29:15Z ***********************
04:29:17:WU01:FS01:Connecting to assign-GPU.stanford.edu:80
04:29:17:WU01:FS01:Connecting to assign-GPU.stanford.edu:80
04:29:17:WU01:FS01:News: Welcome to Folding@Home
04:29:17:WU01:FS01:Assigned to work server 171.67.108.21
04:29:17:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:"G92 [GeForce GTS 250]" from 171.67.108.21
04:29:17:WU01:FS01:Connecting to 171.67.108.21:8080
04:29:18:WU01:FS01:Downloading 61.98KiB
04:29:18:WU01:FS01:Download complete
04:29:18:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:10501 run:21 clone:0 gen:1238 core:0x11 unit:0x00000aca6652eda54b6ea66c00000835
04:29:19:WU01:FS01:Downloading core from http://www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah
04:29:19:WU01:FS01:Connecting to www.stanford.edu:80
04:29:19:WU01:FS01:FahCore 11: Downloading 648.82KiB
04:29:22:WU01:FS01:FahCore 11: Download complete
04:29:22:WU01:FS01:Valid core signature
04:29:22:WU01:FS01:Unpacked 1.82MiB to cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe
04:29:22:WU01:FS01:Starting
04:29:22:WU01:FS01:Running FahCore: C:\Users\Matt\Documents\Folding\FAHClient/FAHCoreWrapper.exe C:/Users/Matt/Documents/Folding/AppData/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 01 -suffix 01 -version 702 -lifeline 1344 -checkpoint 15 -gpu 1
04:29:22:WU01:FS01:Started FahCore on PID 5400
04:29:22:WU01:FS01:Core PID:11952
04:29:22:WU01:FS01:FahCore 0x11 started
04:29:22:WU01:FS01:Downloading project 10501 description
04:29:22:WU01:FS01:Connecting to fah-web.stanford.edu:80
04:29:23:WU01:FS01:Project 10501 description downloaded successfully
04:29:23:WU01:FS01:0x11:
04:29:23:WU01:FS01:0x11:*------------------------------*
04:29:23:WU01:FS01:0x11:Folding@Home GPU Core
04:29:23:WU01:FS01:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
04:29:23:WU01:FS01:0x11:
04:29:23:WU01:FS01:0x11:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
04:29:23:WU01:FS01:0x11:Build host: amoeba
04:29:23:WU01:FS01:0x11:Board Type: Nvidia
04:29:23:WU01:FS01:0x11:Core      :
04:29:23:WU01:FS01:0x11:Preparing to commence simulation
04:29:23:WU01:FS01:0x11:- Looking at optimizations...
04:29:23:WU01:FS01:0x11:DeleteFrameFiles: successfully deleted file=01/wudata_01.ckp
04:29:23:WU01:FS01:0x11:- Created dyn
04:29:23:WU01:FS01:0x11:- Files status OK
04:29:23:WU01:FS01:0x11:- Expanded 62958 -> 336763 (decompressed 534.9 percent)
04:29:23:WU01:FS01:0x11:Called DecompressByteArray: compressed_data_size=62958 data_size=336763, decompressed_data_size=336763 diff=0
04:29:23:WU01:FS01:0x11:- Digital signature verified
04:29:23:WU01:FS01:0x11:
04:29:23:WU01:FS01:0x11:Project: 10501 (Run 21, Clone 0, Gen 1238)
04:29:23:WU01:FS01:0x11:
04:29:23:WU01:FS01:0x11:Assembly optimizations on if available.
04:29:23:WU01:FS01:0x11:Entering M.D.
04:29:28:WU01:FS01:0x11:Tpr hash 01/wudata_01.tpr:  4135312308 2707865303 3455207179 3586408894 348670491
04:29:28:WU01:FS01:0x11:
04:29:28:WU01:FS01:0x11:Calling fah_main args: 14 usage=100
04:29:28:WU01:FS01:0x11:
04:29:29:WU01:FS01:0x11:Working on Protein
04:29:29:WU01:FS01:0x11:Client config unavailable.
04:29:30:WU01:FS01:0x11:Starting GUI Server


ID 02 - gpu:2:Gk107 [GeForce GTX 670]

Code: Select all

*********************** Log Started 2012-10-26T04:29:15Z ***********************
04:29:17:WU02:FS02:Connecting to assign-GPU.stanford.edu:80
04:29:17:WU02:FS02:Connecting to assign-GPU.stanford.edu:80
04:29:17:WU02:FS02:News: Welcome to Folding@Home
04:29:17:WU02:FS02:Assigned to work server 171.67.108.143
04:29:17:WU02:FS02:Requesting new work unit for slot 02: READY gpu:2:"GK107 [GeForce GTX 670]" from 171.67.108.143
04:29:17:WU02:FS02:Connecting to 171.67.108.143:8080
04:29:18:WU02:FS02:Downloading 59.36KiB
04:29:19:WU02:FS02:Download complete
04:29:19:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:8054 run:0 clone:1039 gen:24 core:0x15 unit:0x0000001a6953ee2f50626ad0292b90c4
04:29:19:WU02:FS02:Downloading core from http://www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah
04:29:19:WU02:FS02:Connecting to www.stanford.edu:80
04:29:19:WU02:FS02:FahCore 15: Downloading 1.88MiB
04:29:25:WU02:FS02:FahCore 15: 46.65%
04:29:31:WU02:FS02:FahCore 15: 96.63%
04:29:31:WU02:FS02:FahCore 15: Download complete
04:29:31:WU02:FS02:Valid core signature
04:29:31:WU02:FS02:Unpacked 7.71MiB to cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe
04:29:31:WU02:FS02:Starting
04:29:31:WU02:FS02:Running FahCore: C:\Users\Matt\Documents\Folding\FAHClient/FAHCoreWrapper.exe C:/Users/Matt/Documents/Folding/AppData/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 02 -suffix 01 -version 702 -lifeline 1344 -checkpoint 15 -gpu 2
04:29:31:WU02:FS02:Started FahCore on PID 10416
04:29:31:WU02:FS02:Core PID:9696
04:29:31:WU02:FS02:FahCore 0x15 started
04:29:31:WU02:FS02:Downloading project 8054 description
04:29:31:WU02:FS02:Connecting to fah-web.stanford.edu:80
04:29:31:WU02:FS02:0x15:
04:29:31:WU02:FS02:0x15:*------------------------------*
04:29:31:WU02:FS02:0x15:Folding@Home GPU Core
04:29:31:WU02:FS02:0x15:Version                2.25 (Wed May 9 17:03:01 EDT 2012)
04:29:31:WU02:FS02:0x15:Build host             AmoebaRemote
04:29:31:WU02:FS02:0x15:Board Type             NVIDIA/CUDA
04:29:31:WU02:FS02:0x15:Core                   15
04:29:31:WU02:FS02:0x15:GPU device info vendor=0 device=0 name=NA match=0 deviceId=2
04:29:31:WU02:FS02:0x15:
04:29:31:WU02:FS02:0x15:Window's signal control handler registered.
04:29:31:WU02:FS02:0x15:Preparing to commence simulation
04:29:31:WU02:FS02:0x15:- Looking at optimizations...
04:29:31:WU02:FS02:0x15:DeleteFrameFiles: successfully deleted file=02/wudata_01.ckp
04:29:31:WU02:FS02:0x15:- Created dyn
04:29:31:WU02:FS02:0x15:- Files status OK
04:29:31:WU02:FS02:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
04:29:31:WU02:FS02:0x15:- Expanded 60271 -> 264278 (decompressed 438.4 percent)
04:29:31:WU02:FS02:0x15:Called DecompressByteArray: compressed_data_size=60271 data_size=264278, decompressed_data_size=264278 diff=0
04:29:31:WU02:FS02:0x15:- Digital signature verified
04:29:31:WU02:FS02:0x15:
04:29:31:WU02:FS02:0x15:Project: 8054 (Run 0, Clone 1039, Gen 24)
04:29:31:WU02:FS02:0x15:
04:29:31:WU02:FS02:0x15:Assembly optimizations on if available.
04:29:31:WU02:FS02:0x15:Entering M.D.
04:29:32:WU02:FS02:Project 8054 description downloaded successfully
04:29:33:WU02:FS02:0x15:Tpr hash 02/wudata_01.tpr:  2727818051 495464078 155758220 2017881211 1730012960
04:29:33:WU02:FS02:0x15:GPU device id=2
04:29:33:WU02:FS02:0x15:Working on Good ROcking Metal Altar for Chronical Sinners
04:29:33:WU02:FS02:0x15:Client config unavailable.
04:29:33:WU02:FS02:0x15:Starting GUI Server
04:30:37:WU02:FS02:0x15:Finished fah_main status=59
04:30:37:WU02:FS02:0x15:mdrun_gpu returned 59
04:30:37:WU02:FS02:0x15:GPU memtest failure
04:30:37:WU02:FS02:0x15:
04:30:37:WU02:FS02:0x15:Folding@home Core Shutdown: GPU_MEMTEST_ERROR


ID 03 - smp:12

Code: Select all

*********************** Log Started 2012-10-26T04:29:15Z ***********************
04:29:17:WU03:FS03:Connecting to assign-GPU.stanford.edu:80
04:29:17:WU03:FS03:Connecting to assign3.stanford.edu:8080
04:29:17:WU03:FS03:News: Welcome to Folding@Home
04:29:17:WU03:FS03:Assigned to work server 171.67.108.58
04:29:17:WU03:FS03:Requesting new work unit for slot 03: READY smp:12 from 171.67.108.58
04:29:17:WU03:FS03:Connecting to 171.67.108.58:8080
04:29:18:WU03:FS03:Downloading 671.63KiB
04:29:20:WU03:FS03:Download complete
04:29:20:WU03:FS03:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:8055 run:947 clone:3 gen:26 core:0xa4 unit:0x000000226652edca506d0114aa703b52
04:29:20:WU03:FS03:Downloading core from http://www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah
04:29:20:WU03:FS03:Connecting to www.stanford.edu:80
04:29:20:WU03:FS03:FahCore a4: Downloading 2.89MiB
04:29:26:WU03:FS03:FahCore a4: 38.96%
04:29:32:WU03:FS03:FahCore a4: 71.42%
04:29:38:WU03:FS03:FahCore a4: 97.39%
04:29:39:WU03:FS03:FahCore a4: Download complete
04:29:39:WU03:FS03:Valid core signature
04:29:39:WU03:FS03:Unpacked 9.59MiB to cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe
04:29:39:WU03:FS03:Starting
04:29:39:WU03:FS03:Running FahCore: C:\Users\Matt\Documents\Folding\FAHClient/FAHCoreWrapper.exe C:/Users/Matt/Documents/Folding/AppData/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 03 -suffix 01 -version 702 -lifeline 1344 -checkpoint 15 -np 12
04:29:39:WU03:FS03:Started FahCore on PID 11296
04:29:39:WU03:FS03:Core PID:8316
04:29:39:WU03:FS03:FahCore 0xa4 started
04:29:39:WU03:FS03:Downloading project 8055 description
04:29:39:WU03:FS03:Connecting to fah-web.stanford.edu:80
04:29:39:WU03:FS03:0xa4:
04:29:39:WU03:FS03:0xa4:*------------------------------*
04:29:39:WU03:FS03:0xa4:Folding@Home Gromacs GB Core
04:29:39:WU03:FS03:0xa4:Version 2.27 (Dec. 15, 2010)
04:29:39:WU03:FS03:0xa4:
04:29:39:WU03:FS03:0xa4:Preparing to commence simulation
04:29:39:WU03:FS03:0xa4:- Looking at optimizations...
04:29:39:WU03:FS03:0xa4:- Created dyn
04:29:39:WU03:FS03:0xa4:- Files status OK
04:29:39:WU03:FS03:0xa4:- Expanded 687242 -> 1529840 (decompressed 222.6 percent)
04:29:39:WU03:FS03:0xa4:Called DecompressByteArray: compressed_data_size=687242 data_size=1529840, decompressed_data_size=1529840 diff=0
04:29:39:WU03:FS03:0xa4:- Digital signature verified
04:29:39:WU03:FS03:0xa4:
04:29:39:WU03:FS03:0xa4:Project: 8055 (Run 947, Clone 3, Gen 26)
04:29:39:WU03:FS03:0xa4:
04:29:39:WU03:FS03:0xa4:Assembly optimizations on if available.
04:29:39:WU03:FS03:0xa4:Entering M.D.
04:29:40:WU03:FS03:Project 8055 description downloaded successfully
04:29:45:WU03:FS03:0xa4:Mapping NT from 12 to 12
04:30:13:WU03:FS03:0xa4:Completed 0 out of 500000 steps  (0%)


At least I got v7 to report memtest error this time on the 670... but I still don't know why. It's at stock settings and was folding fine yesterday. But now, none of the 250's are folding. After setting the SMP to use 8 cores, one of the GTS 250's starts folding, the other is stuck at Calling fah_main args: 14 usage = 100 still, and the 670 still gives memtest error, but it never shows any load activity under GPU-Z.

I had all of these issues months ago when I originally got the 670, but they somehow resolved themselves. Now, they're even worse.

[Napoleon]

Could you post the contents of your config.xml, mattlowe01`? Might be helpful but take NOTE: be sure you edit out your passkey(s) before submitting, because config.xml stores them in plain text format. Here's my config.xml, for example:

Code: Select all

<config>
  <!-- FahCore Control -->
  <checkpoint v='30'/>

  <!-- Folding Slot Configuration -->
  <cause-pref v='CANCER'/>
  <client-type v='advanced'/>
  <extra-core-args v='-verbose -np 2'/>
  <max-packet-size v='big'/>

  <!-- Logging -->
  <log-rotate-max v='1000'/>
  <verbosity v='5'/>

  <!-- Network -->
  <proxy v=':8080'/>

  <!-- Slot Control -->
  <pause-on-start v='true'/>

  <!-- User Information -->
  <passkey v='MANUALLY BLANKED'/>
  <team v='191980'/>
  <user v='Napoleon'/>

  <!-- Work Unit Control -->
  <next-unit-percentage v='100'/>

  <!-- Folding Slots -->
  <slot id='0' type='GPU'>
    <cuda-index v='0'/>
    <gpu-index v='1'/>
    <opencl-index v='0'/>
    <passkey v='MANUALLY BLANKED'/>
    <user v='Zotac430'/>
  </slot>
  <slot id='1' type='GPU'>
    <cuda-index v='1'/>
    <gpu-index v='0'/>
    <opencl-index v='1'/>
    <passkey v='MANUALLY BLANKED'/>
    <user v='ION'/>
  </slot>
  <slot id='2' type='UNIPROCESSOR'/>
  <slot id='3' type='UNIPROCESSOR'/>
</config>

[bruce]

If you ask them to copy and paste the data from FAHControl, you'll get the config with the passwords masked.

[mattlowe01`]

Here's my config file

Code: Select all

<config>
  <!-- Folding Slot Configuration -->
  <gpu v='true'/>

  <!-- Network -->
  <proxy v=':8080'/>

  <!-- User Information -->
  <team v='50959'/>
  <user v='MattLowe'/>

  <!-- Folding Slots -->
  <slot id='0' type='GPU'/>
  <slot id='1' type='GPU'/>
  <slot id='2' type='GPU'/>
  <slot id='3' type='SMP'>
    <cpus v='8'/>
  </slot>
</config>

No passkeys used...

[Napoleon]

You'll want to get a passkey because it's required in order to receive SMP Quick Return Bonus points. Becomes even more essential once the GPU QRB is in place. Anyway, earlier you mentioned this (my comments):

I can't get the V7.2.9 to correctly identify my GPU's. It lists
GPU:0:G92 [GeForce GTS 250] (gpu-index = 0)
GPU:1:G92 [GeFOrce GTS 250] (gpu-index = 1)
GPU:2:GK107 [GeForce GTX 670] (gpu-index = 2)

Cuda-Z and GPU-Z list them in the order of

GPU:0 GTX 670 (cuda-index = 0)
GPU:1 GTS 250 (cuda-index = 1)
GPU:2 GTS 250 (cuda-index = 2)

So, if you set up your config like this, it should work.

Code: Select all

<config>
  <!-- Folding Slot Configuration -->
  <gpu v='true'/>

  <!-- Network -->
  <proxy v=':8080'/>

  <!-- User Information -->
  <team v='50959'/>
  <user v='MattLowe'/>

  <!-- Folding Slots -->
  <slot id='0' type='GPU'>
    <client-type v='advanced'/>
    <cuda-index v='0'/>
    <gpu-index v='2'/>
    <opencl-index v='0'/>
  </slot>
  <slot id='1' type='GPU'>
    <cuda-index v='1'/>
    <gpu-index v='1'/>
    <opencl-index v='1'/>
  </slot>
  <slot id='2' type='GPU'>
    <cuda-index v='2'/>
    <gpu-index v='0'/>
    <opencl-index v='2'/>
  </slot>
  <slot id='3' type='SMP'>
    <cpus v='8'/>
  </slot>
</config>

Folding slot order should now be GTX670, GTS250, GTS250, SMP. I also set opencl-index to the same value as cuda-index for each slot. May not be necessary, but shouldn't hurt either. BTW, you don't need to edit the config.xml manually, all these adjustments can be done in Configure ==> Slots editing dialogs for the slots. Better to do all the changes in one go before saving, and you may need to restart FAHClient after that. Note that I added Advanced client type override for the GTX670 because of recent Projects 762x Testing Core v2.25 on Adv announcement. So you should be getting the Kepler-compatible core_15 v2.25 and P762x WUs for your GTX670. I presume that's the best option for Kepler folding at the moment, a controlled environment, so to speak. Other slots should receive Normal WUs. Let us know if this works.

If it does work and you want to check which GTS250 folding slot corresponds to which GPU-z slot exactly, pause folding, then start the GPU slots one by one while monitoring GPU utilization for each GPU. If the GTS250 folding slots still aren't in the same order as in GPUz, swap the gpu-index values in the GTS250 folding slots if you like.

[mattlowe01`]

You sir, are a god send.

All cores up and running, Gtx 670 crunching away on project 7623. Just created a passkey as well.

It's odd, because I entered these settings exactly this way from inside the client, and it didn't work. But replacing the XML worked instantly.

Thanks again!