SOLVED: Unstable machine problem with new card: GTX 660ti

[aoeu]

The first card failed totally and has been RMAed

I need to reopen this problem as I am still experiencing issues.
There are two failure modes.
One case is a video driver failure followed by dropping a WU. I can live with the occasional dropped WU.
The other just happened again and involves the entire machine rebooting by surprise. That really needs to stop.
Since my first post I have replaced the other GTS450 with another Asus GTX 660ti. They are not SLIed.
I have also replaced the power supply.
I have a 12" fan blowing on the video card area.
Driver is 306.97
Mainboard is a Gigabyte P55A-UD3

F@H is identifying the card closest to the CPU as Slot 01. At the moment the only monitor on that card is the one the BIOS displays on and the only window open on it is FAHControl.
Failure only occurs when two conditions exist: Firefox is running and F@H is working on a 76xx unit in Slot 01.
No other program need be running.
Switching the cards does not matter, it happens to both of them.
Running my main monitors on the card in Slot 00 does not matter either. That is the experiment that just produced failure.

I'm running out of ideas and am down to three:
Reload Windows from scratch on the theory that I have a bad case of cruft
Replace the mainboard.
Something else. Any ideas?


This is the current log

Code: Select all

*********************** Log Started 2013-03-06T13:52:51Z ***********************
13:52:51:************************* Folding@home Client *************************
13:52:51:      Website: http://folding.stanford.edu/
13:52:51:    Copyright: (c) 2009-2012 Stanford University
13:52:51:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
13:52:51:         Args: --lifeline 3804 --command-port=36330
13:52:51:       Config: C:/Users/aoeu/AppData/Roaming/FAHClient/config.xml
13:52:51:******************************** Build ********************************
13:52:51:      Version: 7.2.9
13:52:51:         Date: Oct 3 2012
13:52:51:         Time: 18:05:48
13:52:51:      SVN Rev: 3578
13:52:51:       Branch: fah/trunk/client
13:52:51:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
13:52:51:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
13:52:51:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
13:52:51:     Platform: win32 XP
13:52:51:         Bits: 32
13:52:51:         Mode: Release
13:52:51:******************************* System ********************************
13:52:51:          CPU: Intel(R) Core(TM) i5 CPU 750 @ 2.67GHz
13:52:51:       CPU ID: GenuineIntel Family 6 Model 30 Stepping 5
13:52:51:         CPUs: 4
13:52:51:       Memory: 7.99GiB
13:52:51:  Free Memory: 6.75GiB
13:52:51:      Threads: WINDOWS_THREADS
13:52:51:   On Battery: false
13:52:51:   UTC offset: -5
13:52:51:          PID: 4040
13:52:51:          CWD: C:/Users/aoeu/AppData/Roaming/FAHClient
13:52:51:           OS: Windows 7 Professional
13:52:51:      OS Arch: AMD64
13:52:51:         GPUs: 2
13:52:51:        GPU 0: NVIDIA:3 GK104 [GeForce GTX 660 Ti]
13:52:51:        GPU 1: NVIDIA:3 GK104 [GeForce GTX 660 Ti]
13:52:51:         CUDA: 3.0
13:52:51:  CUDA Driver: 5000
13:52:51:Win32 Service: false
13:52:51:***********************************************************************
13:52:51:<config>
13:52:51:  <!-- Folding Slot Configuration -->
13:52:51:  <gpu v='true'/>
13:52:51:
13:52:51:  <!-- Network -->
13:52:51:  <proxy v=':8080'/>
13:52:51:
13:52:51:  <!-- User Information -->
13:52:51:  <passkey v='********************************'/>
13:52:51:  <team v='48083'/>
13:52:51:  <user v='aoeu'/>
13:52:51:
13:52:51:  <!-- Folding Slots -->
13:52:51:  <slot id='0' type='GPU'/>
13:52:51:  <slot id='2' type='SMP'/>
13:52:51:  <slot id='1' type='GPU'/>
13:52:51:</config>
13:52:51:Trying to access database...
13:52:52:Successfully acquired database lock
13:52:52:Enabled folding slot 00: READY gpu:0:"GK104 [GeForce GTX 660 Ti]"
13:52:52:Enabled folding slot 02: READY smp:4
13:52:52:Enabled folding slot 01: READY gpu:1:"GK104 [GeForce GTX 660 Ti]"
13:52:52:WU00:FS02:Starting
13:52:52:WU00:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/aoeu/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 702 -lifeline 4040 -checkpoint 15 -np 4
13:52:52:WU00:FS02:Started FahCore on PID 2592
13:52:52:WU00:FS02:Core PID:3092
13:52:52:WU00:FS02:FahCore 0xa4 started
13:52:52:WU03:FS01:Starting
13:52:52:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/aoeu/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 03 -suffix 01 -version 702 -lifeline 4040 -checkpoint 15 -gpu 1
13:52:52:WU03:FS01:Started FahCore on PID 2788
13:52:52:WU03:FS01:Core PID:2800
13:52:52:WU03:FS01:FahCore 0x15 started
13:52:52:WU01:FS00:Starting
13:52:52:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/aoeu/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 01 -suffix 01 -version 702 -lifeline 4040 -checkpoint 15 -gpu 0
13:52:52:WU01:FS00:Started FahCore on PID 2772
13:52:52:WU01:FS00:Core PID:3124
13:52:52:WU01:FS00:FahCore 0x15 started
13:52:52:WU00:FS02:0xa4:
13:52:52:WU00:FS02:0xa4:*------------------------------*
13:52:52:WU00:FS02:0xa4:Folding@Home Gromacs GB Core
13:52:52:WU00:FS02:0xa4:Version 2.27 (Dec. 15, 2010)
13:52:52:WU00:FS02:0xa4:
13:52:52:WU00:FS02:0xa4:Preparing to commence simulation
13:52:52:WU00:FS02:0xa4:- Ensuring status. Please wait.
13:52:52:WU03:FS01:0x15:
13:52:52:WU03:FS01:0x15:*------------------------------*
13:52:52:WU03:FS01:0x15:Folding@Home GPU Core
13:52:52:WU03:FS01:0x15:Version                2.25 (Wed May 9 17:03:01 EDT 2012)
13:52:52:WU03:FS01:0x15:Build host             AmoebaRemote
13:52:52:WU03:FS01:0x15:Board Type             NVIDIA/CUDA
13:52:52:WU03:FS01:0x15:Core                   15
13:52:52:WU03:FS01:0x15:GPU device info vendor=0 device=0 name=NA match=0 deviceId=1
13:52:52:WU03:FS01:0x15:
13:52:52:WU03:FS01:0x15:Window's signal control handler registered.
13:52:52:WU03:FS01:0x15:Preparing to commence simulation
13:52:52:WU03:FS01:0x15:- Ensuring status. Please wait.
13:52:52:WU01:FS00:0x15:
13:52:52:WU01:FS00:0x15:*------------------------------*
13:52:52:WU01:FS00:0x15:Folding@Home GPU Core
13:52:52:WU01:FS00:0x15:Version                2.25 (Wed May 9 17:03:01 EDT 2012)
13:52:52:WU01:FS00:0x15:Build host             AmoebaRemote
13:52:52:WU01:FS00:0x15:Board Type             NVIDIA/CUDA
13:52:52:WU01:FS00:0x15:Core                   15
13:52:52:WU01:FS00:0x15:
13:52:52:WU01:FS00:0x15:Window's signal control handler registered.
13:52:52:WU01:FS00:0x15:Preparing to commence simulation
13:52:52:WU01:FS00:0x15:- Ensuring status. Please wait.
13:52:54:Server connection id=1 on 0.0.0.0:36330 from 127.0.0.1
13:53:01:WU00:FS02:0xa4:- Looking at optimizations...
13:53:01:WU00:FS02:0xa4:- Working with standard loops on this execution.
13:53:01:WU00:FS02:0xa4:- Previous termination of core was improper.
13:53:01:WU00:FS02:0xa4:- Going to use standard loops.
13:53:01:WU00:FS02:0xa4:- Files status OK
13:53:01:WU00:FS02:0xa4:- Expanded 547363 -> 846804 (decompressed 154.7 percent)
13:53:01:WU00:FS02:0xa4:Called DecompressByteArray: compressed_data_size=547363 data_size=846804, decompressed_data_size=846804 diff=0
13:53:02:WU00:FS02:0xa4:- Digital signature verified
13:53:02:WU00:FS02:0xa4:
13:53:02:WU00:FS02:0xa4:Project: 7647 (Run 201, Clone 0, Gen 102)
13:53:02:WU00:FS02:0xa4:
13:53:02:WU00:FS02:0xa4:Entering M.D.
13:53:02:WU03:FS01:0x15:- Looking at optimizations...
13:53:02:WU03:FS01:0x15:- Working with standard loops on this execution.
13:53:02:WU03:FS01:0x15:- Previous termination of core was improper.
13:53:02:WU03:FS01:0x15:- Files status OK
13:53:02:WU03:FS01:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
13:53:02:WU03:FS01:0x15:- Expanded 126212 -> 507182 (decompressed 401.8 percent)
13:53:02:WU03:FS01:0x15:Called DecompressByteArray: compressed_data_size=126212 data_size=507182, decompressed_data_size=507182 diff=0
13:53:02:WU03:FS01:0x15:- Digital signature verified
13:53:02:WU03:FS01:0x15:
13:53:02:WU03:FS01:0x15:Project: 7624 (Run 269, Clone 0, Gen 149)
13:53:02:WU03:FS01:0x15:
13:53:02:WU03:FS01:0x15:Entering M.D.
13:53:02:WU01:FS00:0x15:- Looking at optimizations...
13:53:02:WU01:FS00:0x15:- Working with standard loops on this execution.
13:53:02:WU01:FS00:0x15:- Previous termination of core was improper.
13:53:02:WU01:FS00:0x15:- Files status OK
13:53:02:WU01:FS00:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
13:53:02:WU01:FS00:0x15:- Expanded 58336 -> 257358 (decompressed 441.1 percent)
13:53:02:WU01:FS00:0x15:Called DecompressByteArray: compressed_data_size=58336 data_size=257358, decompressed_data_size=257358 diff=0
13:53:02:WU01:FS00:0x15:- Digital signature verified
13:53:02:WU01:FS00:0x15:
13:53:02:WU01:FS00:0x15:Project: 8070 (Run 87, Clone 11, Gen 22)
13:53:02:WU01:FS00:0x15:
13:53:02:WU01:FS00:0x15:Entering M.D.
13:53:04:WU03:FS01:0x15:Will resume from checkpoint file 03/wudata_01.ckp
13:53:04:WU03:FS01:0x15:Tpr hash 03/wudata_01.tpr:  3770453588 404081512 1468488701 2457642417 808595515
13:53:04:WU03:FS01:0x15:GPU device id=1
13:53:04:WU03:FS01:0x15:Working on Protein
13:53:04:WU03:FS01:0x15:Client config unavailable.
13:53:04:WU03:FS01:0x15:Starting GUI Server
13:53:04:WU01:FS00:0x15:Will resume from checkpoint file 01/wudata_01.ckp
13:53:04:WU01:FS00:0x15:Tpr hash 01/wudata_01.tpr:  2073289338 1261486888 905855471 2745521801 3138994534
13:53:04:WU01:FS00:0x15:GPU device id=0
13:53:04:WU01:FS00:0x15:Working on Gallium Rubidium Oxygen Manganese Argon Carbon Silicon t=  87.00000
13:53:04:WU01:FS00:0x15:Client config unavailable.
13:53:04:WU01:FS00:0x15:Starting GUI Server
13:53:07:WU00:FS02:0xa4:Using Gromacs checkpoints
13:53:07:WU00:FS02:0xa4:Mapping NT from 4 to 4
13:53:08:WU00:FS02:0xa4:Resuming from checkpoint
13:53:08:WU00:FS02:0xa4:Verified 00/wudata_01.log
13:53:08:WU00:FS02:0xa4:Verified 00/wudata_01.trr
13:53:08:WU00:FS02:0xa4:Verified 00/wudata_01.xtc
13:53:08:WU00:FS02:0xa4:Verified 00/wudata_01.edr
13:53:10:WU00:FS02:0xa4:Completed 2256390 out of 2500000 steps  (90%)
13:54:05:WU01:FS00:0x15:Resuming from checkpoint
13:54:05:WU01:FS00:0x15:fcCheckPointResume: retreived and current tpr file hash:
13:54:05:WU01:FS00:0x15:   0   2073289338   2073289338
13:54:05:WU01:FS00:0x15:   1   1261486888   1261486888
13:54:05:WU01:FS00:0x15:   2    905855471    905855471
13:54:05:WU01:FS00:0x15:   3   2745521801   2745521801
13:54:05:WU01:FS00:0x15:   4   3138994534   3138994534
13:54:05:WU01:FS00:0x15:fcCheckPointResume: file hashes same.
13:54:05:WU01:FS00:0x15:fcCheckPointResume: state restored.
13:54:05:WU01:FS00:0x15:fcCheckPointResume: name 01/wudata_01.log Verified 01/wudata_01.log
13:54:05:WU01:FS00:0x15:fcCheckPointResume: name 01/wudata_01.trr Verified 01/wudata_01.trr
13:54:05:WU01:FS00:0x15:fcCheckPointResume: name 01/wudata_01.xtc Verified 01/wudata_01.xtc
13:54:05:WU01:FS00:0x15:fcCheckPointResume: name 01/wudata_01.edr Verified 01/wudata_01.edr
13:54:05:WU01:FS00:0x15:fcCheckPointResume: state restored 2
13:54:05:WU01:FS00:0x15:Resumed from checkpoint
13:54:05:WU01:FS00:0x15:Setting checkpoint frequency: 500000
13:54:05:WU01:FS00:0x15:Completed  32500001 out of 50000000 steps (65%).
13:54:06:WARNING:WU01:FS00:Detected clock skew (1 mins 05 secs), adjusting time estimates
13:54:07:WU03:FS01:0x15:Resuming from checkpoint
13:54:07:WU03:FS01:0x15:fcCheckPointResume: retreived and current tpr file hash:
13:54:07:WU03:FS01:0x15:   0   3770453588   3770453588
13:54:07:WU03:FS01:0x15:   1    404081512    404081512
13:54:07:WU03:FS01:0x15:   2   1468488701   1468488701
13:54:07:WU03:FS01:0x15:   3   2457642417   2457642417
13:54:07:WU03:FS01:0x15:   4    808595515    808595515
13:54:07:WU03:FS01:0x15:fcCheckPointResume: file hashes same.
13:54:07:WU03:FS01:0x15:fcCheckPointResume: state restored.
13:54:07:WU03:FS01:0x15:fcCheckPointResume: name 03/wudata_01.log Verified 03/wudata_01.log
13:54:07:WU03:FS01:0x15:fcCheckPointResume: name 03/wudata_01.trr Verified 03/wudata_01.trr
13:54:07:WU03:FS01:0x15:fcCheckPointResume: name 03/wudata_01.xtc Verified 03/wudata_01.xtc
13:54:07:WU03:FS01:0x15:fcCheckPointResume: name 03/wudata_01.edr Verified 03/wudata_01.edr
13:54:07:WU03:FS01:0x15:fcCheckPointResume: state restored 2
13:54:07:WU03:FS01:0x15:Resumed from checkpoint
13:54:07:WU03:FS01:0x15:Setting checkpoint frequency: 400000
13:54:07:WU03:FS01:0x15:Completed  19200001 out of 40000000 steps (48%).
13:54:07:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
13:56:25:WU01:FS00:0x15:Completed  33000000 out of 50000000 steps (66%).
13:58:46:WU01:FS00:0x15:Completed  33500000 out of 50000000 steps (67%).
13:59:49:WU03:FS01:0x15:Completed  19600000 out of 40000000 steps (49%).
14:01:07:WU01:FS00:0x15:Completed  34000000 out of 50000000 steps (68%).
14:03:27:WU01:FS00:0x15:Completed  34500000 out of 50000000 steps (69%).
14:05:33:WU03:FS01:0x15:Completed  20000000 out of 40000000 steps (50%).
14:05:49:WU01:FS00:0x15:Completed  35000000 out of 50000000 steps (70%).
14:08:10:WU01:FS00:0x15:Completed  35500000 out of 50000000 steps (71%).
14:10:31:WU01:FS00:0x15:Completed  36000000 out of 50000000 steps (72%).
14:11:17:WU03:FS01:0x15:Completed  20400000 out of 40000000 steps (51%).
14:12:52:WU01:FS00:0x15:Completed  36500000 out of 50000000 steps (73%).
14:15:12:WU01:FS00:0x15:Completed  37000000 out of 50000000 steps (74%).
14:17:01:WU03:FS01:0x15:Completed  20800000 out of 40000000 steps (52%).
14:17:33:WU01:FS00:0x15:Completed  37500000 out of 50000000 steps (75%).
14:19:55:WU01:FS00:0x15:Completed  38000000 out of 50000000 steps (76%).
14:22:16:WU01:FS00:0x15:Completed  38500000 out of 50000000 steps (77%).
14:22:45:WU03:FS01:0x15:Completed  21200000 out of 40000000 steps (53%).
14:24:36:WU01:FS00:0x15:Completed  39000000 out of 50000000 steps (78%).
14:24:38:WU00:FS02:0xa4:Completed 2275000 out of 2500000 steps  (91%)
14:26:57:WU01:FS00:0x15:Completed  39500000 out of 50000000 steps (79%).
14:28:29:WU03:FS01:0x15:Completed  21600000 out of 40000000 steps (54%).
14:29:18:WU01:FS00:0x15:Completed  40000000 out of 50000000 steps (80%).
14:31:39:WU01:FS00:0x15:Completed  40500000 out of 50000000 steps (81%).
14:34:00:WU01:FS00:0x15:Completed  41000000 out of 50000000 steps (82%).
14:34:12:WU03:FS01:0x15:Completed  22000000 out of 40000000 steps (55%).


This is the log at the time of most recent failure

Code: Select all

*********************** Log Started 2013-03-03T16:55:33Z ***********************
16:55:33:************************* Folding@home Client *************************
16:55:33:      Website: http://folding.stanford.edu/
16:55:33:    Copyright: (c) 2009-2012 Stanford University
16:55:33:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:55:33:         Args: --lifeline 3724 --command-port=36330
16:55:33:       Config: C:/Users/aoeu/AppData/Roaming/FAHClient/config.xml
16:55:33:******************************** Build ********************************
16:55:33:      Version: 7.2.9
16:55:33:         Date: Oct 3 2012
16:55:33:         Time: 18:05:48
16:55:33:      SVN Rev: 3578
16:55:33:       Branch: fah/trunk/client
16:55:33:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
16:55:33:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
16:55:33:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
16:55:33:     Platform: win32 XP
16:55:33:         Bits: 32
16:55:33:         Mode: Release
16:55:33:******************************* System ********************************
16:55:33:          CPU: Intel(R) Core(TM) i5 CPU 750 @ 2.67GHz
16:55:33:       CPU ID: GenuineIntel Family 6 Model 30 Stepping 5
16:55:33:         CPUs: 4
16:55:33:       Memory: 7.99GiB
16:55:33:  Free Memory: 6.76GiB
16:55:33:      Threads: WINDOWS_THREADS
16:55:33:   On Battery: false
16:55:33:   UTC offset: -5
16:55:33:          PID: 3928
16:55:33:          CWD: C:/Users/aoeu/AppData/Roaming/FAHClient
16:55:33:           OS: Windows 7 Professional
16:55:33:      OS Arch: AMD64
16:55:33:         GPUs: 2
16:55:33:        GPU 0: NVIDIA:3 GK104 [GeForce GTX 660 Ti]
16:55:33:        GPU 1: NVIDIA:3 GK104 [GeForce GTX 660 Ti]
16:55:33:         CUDA: 3.0
16:55:33:  CUDA Driver: 5000
16:55:33:Win32 Service: false
16:55:33:***********************************************************************
16:55:33:<config>
16:55:33:  <!-- Folding Slot Configuration -->
16:55:33:  <gpu v='true'/>
16:55:33:
16:55:33:  <!-- Network -->
16:55:33:  <proxy v=':8080'/>
16:55:33:
16:55:33:  <!-- User Information -->
16:55:33:  <passkey v='********************************'/>
16:55:33:  <team v='48083'/>
16:55:33:  <user v='aoeu'/>
16:55:33:
16:55:33:  <!-- Folding Slots -->
16:55:33:  <slot id='0' type='GPU'/>
16:55:33:  <slot id='2' type='SMP'/>
16:55:33:  <slot id='1' type='GPU'/>
16:55:33:</config>
16:55:33:Trying to access database...
16:55:33:Successfully acquired database lock
16:55:33:Enabled folding slot 00: READY gpu:0:"GK104 [GeForce GTX 660 Ti]"
16:55:33:Enabled folding slot 02: READY smp:4
16:55:33:Enabled folding slot 01: READY gpu:1:"GK104 [GeForce GTX 660 Ti]"
16:55:33:WU03:FS01:Starting
16:55:33:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/aoeu/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 03 -suffix 01 -version 702 -lifeline 3928 -checkpoint 15 -gpu 1
16:55:33:WU03:FS01:Started FahCore on PID 3980
16:55:33:WU03:FS01:Core PID:3992
16:55:33:WU03:FS01:FahCore 0x15 started
16:55:33:WU00:FS02:Starting
16:55:33:WU00:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/aoeu/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 702 -lifeline 3928 -checkpoint 15 -np 4
16:55:33:WU00:FS02:Started FahCore on PID 4000
16:55:33:WU00:FS02:Core PID:4012
16:55:33:WU00:FS02:FahCore 0xa4 started
16:55:33:WU01:FS00:Starting
16:55:33:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/aoeu/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 01 -suffix 01 -version 702 -lifeline 3928 -checkpoint 15 -gpu 0
16:55:33:WU01:FS00:Started FahCore on PID 4024
16:55:33:WU01:FS00:Core PID:4036
16:55:33:WU01:FS00:FahCore 0x15 started
16:55:33:WU03:FS01:0x15:
16:55:33:WU03:FS01:0x15:*------------------------------*
16:55:33:WU03:FS01:0x15:Folding@Home GPU Core
16:55:33:WU03:FS01:0x15:Version                2.25 (Wed May 9 17:03:01 EDT 2012)
16:55:33:WU03:FS01:0x15:Build host             AmoebaRemote
16:55:33:WU03:FS01:0x15:Board Type             NVIDIA/CUDA
16:55:33:WU03:FS01:0x15:Core                   15
16:55:33:WU03:FS01:0x15:GPU device info vendor=0 device=0 name=NA match=0 deviceId=1
16:55:33:WU03:FS01:0x15:
16:55:33:WU03:FS01:0x15:Window's signal control handler registered.
16:55:33:WU03:FS01:0x15:Preparing to commence simulation
16:55:33:WU03:FS01:0x15:- Ensuring status. Please wait.
16:55:33:WU00:FS02:0xa4:
16:55:33:WU00:FS02:0xa4:*------------------------------*
16:55:33:WU00:FS02:0xa4:Folding@Home Gromacs GB Core
16:55:33:WU00:FS02:0xa4:Version 2.27 (Dec. 15, 2010)
16:55:33:WU00:FS02:0xa4:
16:55:33:WU00:FS02:0xa4:Preparing to commence simulation
16:55:33:WU00:FS02:0xa4:- Ensuring status. Please wait.
16:55:34:WU01:FS00:0x15:
16:55:34:WU01:FS00:0x15:*------------------------------*
16:55:34:WU01:FS00:0x15:Folding@Home GPU Core
16:55:34:WU01:FS00:0x15:Version                2.25 (Wed May 9 17:03:01 EDT 2012)
16:55:34:WU01:FS00:0x15:Build host             AmoebaRemote
16:55:34:WU01:FS00:0x15:Board Type             NVIDIA/CUDA
16:55:34:WU01:FS00:0x15:Core                   15
16:55:34:WU01:FS00:0x15:
16:55:34:WU01:FS00:0x15:Window's signal control handler registered.
16:55:34:WU01:FS00:0x15:Preparing to commence simulation
16:55:34:WU01:FS00:0x15:- Ensuring status. Please wait.
16:55:36:Server connection id=1 on 0.0.0.0:36330 from 127.0.0.1
16:55:43:WU01:FS00:0x15:- Looking at optimizations...
16:55:43:WU01:FS00:0x15:- Working with standard loops on this execution.
16:55:43:WU01:FS00:0x15:- Previous termination of core was improper.
16:55:43:WU01:FS00:0x15:- Going to use standard loops.
16:55:43:WU01:FS00:0x15:- Files status OK
16:55:43:WU01:FS00:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
16:55:43:WU03:FS01:0x15:- Looking at optimizations...
16:55:43:WU03:FS01:0x15:- Working with standard loops on this execution.
16:55:43:WU03:FS01:0x15:- Previous termination of core was improper.
16:55:43:WU03:FS01:0x15:- Going to use standard loops.
16:55:43:WU03:FS01:0x15:- Files status OK
16:55:43:WU03:FS01:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
16:55:43:WU03:FS01:0x15:- Expanded 125183 -> 502918 (decompressed 401.7 percent)
16:55:43:WU03:FS01:0x15:Called DecompressByteArray: compressed_data_size=125183 data_size=502918, decompressed_data_size=502918 diff=0
16:55:43:WU03:FS01:0x15:- Digital signature verified
16:55:43:WU03:FS01:0x15:
16:55:43:WU03:FS01:0x15:Project: 7625 (Run 51, Clone 0, Gen 81)
16:55:43:WU03:FS01:0x15:
16:55:43:WU03:FS01:0x15:Entering M.D.
16:55:43:WU00:FS02:0xa4:- Looking at optimizations...
16:55:43:WU00:FS02:0xa4:- Working with standard loops on this execution.
16:55:43:WU00:FS02:0xa4:- Previous termination of core was improper.
16:55:43:WU00:FS02:0xa4:- Going to use standard loops.
16:55:43:WU00:FS02:0xa4:- Files status OK
16:55:43:WU00:FS02:0xa4:- Expanded 547363 -> 846804 (decompressed 154.7 percent)
16:55:43:WU00:FS02:0xa4:Called DecompressByteArray: compressed_data_size=547363 data_size=846804, decompressed_data_size=846804 diff=0
16:55:43:WU00:FS02:0xa4:- Digital signature verified
16:55:43:WU00:FS02:0xa4:
16:55:43:WU00:FS02:0xa4:Project: 7647 (Run 201, Clone 0, Gen 102)
16:55:43:WU00:FS02:0xa4:
16:55:43:WU00:FS02:0xa4:Entering M.D.
16:55:43:WU01:FS00:0x15:- Expanded 79513 -> 307810 (decompressed 387.1 percent)
16:55:43:WU01:FS00:0x15:Called DecompressByteArray: compressed_data_size=79513 data_size=307810, decompressed_data_size=307810 diff=0
16:55:43:WU01:FS00:0x15:- Digital signature verified
16:55:43:WU01:FS00:0x15:
16:55:43:WU01:FS00:0x15:Project: 7660 (Run 1021, Clone 0, Gen 29)
16:55:43:WU01:FS00:0x15:
16:55:43:WU01:FS00:0x15:Entering M.D.
16:55:45:WU01:FS00:0x15:Will resume from checkpoint file 01/wudata_01.ckp
16:55:45:WU01:FS00:0x15:Tpr hash 01/wudata_01.tpr:  1297378395 3326411384 390537312 3251781890 3013743469
16:55:45:WU01:FS00:0x15:GPU device id=0
16:55:45:WU01:FS00:0x15:Working on Protein
16:55:45:WU01:FS00:0x15:Client config unavailable.
16:55:45:WU03:FS01:0x15:Will resume from checkpoint file 03/wudata_01.ckp
16:55:45:WU03:FS01:0x15:Tpr hash 03/wudata_01.tpr:  241767762 1335177123 138446583 1040823125 2958142504
16:55:45:WU03:FS01:0x15:GPU device id=1
16:55:45:WU03:FS01:0x15:Working on Protein
16:55:45:WU03:FS01:0x15:Client config unavailable.
16:55:45:WU03:FS01:0x15:Starting GUI Server
16:55:45:WU01:FS00:0x15:Starting GUI Server
16:55:49:WU00:FS02:0xa4:Using Gromacs checkpoints
16:55:49:WU00:FS02:0xa4:Mapping NT from 4 to 4
16:55:49:WU00:FS02:0xa4:Resuming from checkpoint
16:55:49:WU00:FS02:0xa4:Verified 00/wudata_01.log
16:55:49:WU00:FS02:0xa4:Verified 00/wudata_01.trr
16:55:49:WU00:FS02:0xa4:Verified 00/wudata_01.xtc
16:55:49:WU00:FS02:0xa4:Verified 00/wudata_01.edr
16:55:49:WU00:FS02:0xa4:Completed 193230 out of 2500000 steps  (7%)
16:56:47:WU01:FS00:0x15:Resuming from checkpoint
16:56:47:WU01:FS00:0x15:fcCheckPointResume: retreived and current tpr file hash:
16:56:47:WU01:FS00:0x15:   0   1297378395   1297378395
16:56:47:WU01:FS00:0x15:   1   3326411384   3326411384
16:56:47:WU01:FS00:0x15:   2    390537312    390537312
16:56:47:WU01:FS00:0x15:   3   3251781890   3251781890
16:56:47:WU01:FS00:0x15:   4   3013743469   3013743469
16:56:47:WU01:FS00:0x15:fcCheckPointResume: file hashes same.
16:56:47:WU01:FS00:0x15:fcCheckPointResume: state restored.
16:56:47:WU01:FS00:0x15:fcCheckPointResume: name 01/wudata_01.log Verified 01/wudata_01.log
16:56:47:WARNING:WU01:FS00:Detected clock skew (1 mins 04 secs), adjusting time estimates
16:56:47:WU01:FS00:0x15:fcCheckPointResume: name 01/wudata_01.trr Verified 01/wudata_01.trr
16:56:47:WU01:FS00:0x15:fcCheckPointResume: name 01/wudata_01.xtc Verified 01/wudata_01.xtc
16:56:47:WU01:FS00:0x15:fcCheckPointResume: name 01/wudata_01.edr Verified 01/wudata_01.edr
16:56:47:WU01:FS00:0x15:fcCheckPointResume: state restored 2
16:56:47:WU01:FS00:0x15:Resumed from checkpoint
16:56:47:WU01:FS00:0x15:Setting checkpoint frequency: 400000
16:56:47:WU01:FS00:0x15:Completed  12800001 out of 40000000 steps (32%).
16:56:48:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
16:56:48:WU03:FS01:0x15:Resuming from checkpoint
16:56:48:WU03:FS01:0x15:fcCheckPointResume: retreived and current tpr file hash:
16:56:48:WU03:FS01:0x15:   0    241767762    241767762
16:56:48:WU03:FS01:0x15:   1   1335177123   1335177123
16:56:48:WU03:FS01:0x15:   2    138446583    138446583
16:56:48:WU03:FS01:0x15:   3   1040823125   1040823125
16:56:48:WU03:FS01:0x15:   4   2958142504   2958142504
16:56:48:WU03:FS01:0x15:fcCheckPointResume: file hashes same.
16:56:48:WU03:FS01:0x15:fcCheckPointResume: state restored.
16:56:48:WU03:FS01:0x15:fcCheckPointResume: name 03/wudata_01.log Verified 03/wudata_01.log
16:56:48:WU03:FS01:0x15:fcCheckPointResume: name 03/wudata_01.trr Verified 03/wudata_01.trr
16:56:48:WU03:FS01:0x15:fcCheckPointResume: name 03/wudata_01.xtc Verified 03/wudata_01.xtc
16:56:48:WU03:FS01:0x15:fcCheckPointResume: name 03/wudata_01.edr Verified 03/wudata_01.edr
16:56:48:WU03:FS01:0x15:fcCheckPointResume: state restored 2
16:56:48:WU03:FS01:0x15:Resumed from checkpoint
16:56:48:WU03:FS01:0x15:Setting checkpoint frequency: 400000
16:56:48:WU03:FS01:0x15:Completed  11200001 out of 40000000 steps (28%).


SNIP


14:33:18:WU02:FS01:0x15:Completed  32500000 out of 50000000 steps (65%).
14:35:31:WU03:FS00:0x15:Completed  19500000 out of 50000000 steps (39%).
14:35:34:WU02:FS01:0x15:Completed  33000000 out of 50000000 steps (66%).
14:37:50:WU02:FS01:0x15:Completed  33500000 out of 50000000 steps (67%).
14:37:52:WU03:FS00:0x15:Completed  20000000 out of 50000000 steps (40%).
14:40:07:WU02:FS01:0x15:Completed  34000000 out of 50000000 steps (68%).
14:40:15:WU03:FS00:0x15:Completed  20500000 out of 50000000 steps (41%).
14:41:31:WARNING:WU02:FS01:FahCore crashed with Windows unhandled exception code 0xUNKNOWN_ENUM, searching for this code online may provide more information
14:41:31:WARNING:WU02:FS01:FahCore returned: UNKNOWN_ENUM (1073807364 = 0x40010004)
14:41:32:WARNING:WU03:FS00:FahCore crashed with Windows unhandled exception code 0xUNKNOWN_ENUM, searching for this code online may provide more information
14:41:32:WARNING:WU03:FS00:FahCore returned: UNKNOWN_ENUM (1073807364 = 0x40010004)
14:41:32:WU02:FS01:Starting
14:41:32:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/aoeu/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 02 -suffix 01 -version 702 -lifeline 3928 -checkpoint 15 -gpu 1
14:41:32:WU02:FS01:Started FahCore on PID 147496


**********************************************Below is older news*******************************************
DirectCU II OC.
Windows 7 64bit
Intel core I5-750
Old video configuration was 2 non SLIed GeForce GTS450sc cards and ppd was about 25k.
I replaced the primary card with the one in the title. I have never done anything to overclock anything in the computer. ppd on the new card with 76xx projects is about 35k but it drops them occasionally and it just happened again. Screens go black for a couple of seconds and then it starts over. Occasionally a project will run much slower on the new card and a reboot is required to fix it.

Please advise. Do I have a defective card?

Code: Select all

*********************** Log Started 2013-01-24T11:48:35Z ***********************
11:48:35:************************* Folding@home Client *************************
11:48:35:      Website: http://folding.stanford.edu/
11:48:35:    Copyright: (c) 2009-2012 Stanford University
11:48:35:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
11:48:35:         Args: --lifeline 4056 --command-port=36330
11:48:35:       Config: C:/Users/aoeu/AppData/Roaming/FAHClient/config.xml
11:48:35:******************************** Build ********************************
11:48:35:      Version: 7.2.9
11:48:35:         Date: Oct 3 2012
11:48:35:         Time: 18:05:48
11:48:35:      SVN Rev: 3578
11:48:35:       Branch: fah/trunk/client
11:48:35:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
11:48:35:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
11:48:35:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
11:48:35:     Platform: win32 XP
11:48:35:         Bits: 32
11:48:35:         Mode: Release
11:48:35:******************************* System ********************************
11:48:35:          CPU: Intel(R) Core(TM) i5 CPU 750 @ 2.67GHz
11:48:35:       CPU ID: GenuineIntel Family 6 Model 30 Stepping 5
11:48:35:         CPUs: 4
11:48:35:       Memory: 7.99GiB
11:48:35:  Free Memory: 6.64GiB
11:48:35:      Threads: WINDOWS_THREADS
11:48:35:   On Battery: false
11:48:35:   UTC offset: -5
11:48:35:          PID: 3844
11:48:35:          CWD: C:/Users/aoeu/AppData/Roaming/FAHClient
11:48:35:           OS: Windows 7 Professional
11:48:35:      OS Arch: AMD64
11:48:35:         GPUs: 2
11:48:35:        GPU 0: NVIDIA:3 GK104 [GeForce GTX 660 Ti]
11:48:35:        GPU 1: NVIDIA:2 GF116 [GeForce GTS 450]
11:48:35:         CUDA: 3.0
11:48:35:  CUDA Driver: 5000
11:48:35:Win32 Service: false
11:48:35:***********************************************************************
11:48:35:<config>
11:48:35:  <!-- Folding Slot Configuration -->
11:48:35:  <cause-pref v='ALZHEIMERS'/>
11:48:35:  <gpu v='true'/>
11:48:35:
11:48:35:  <!-- Network -->
11:48:35:  <proxy v=':8080'/>
11:48:35:
11:48:35:  <!-- User Information -->
11:48:35:  <passkey v='********************************'/>
11:48:35:  <team v='48083'/>
11:48:35:  <user v='aoeu'/>
11:48:35:
11:48:35:  <!-- Folding Slots -->
11:48:35:  <slot id='0' type='GPU'/>
11:48:35:  <slot id='1' type='GPU'/>
11:48:35:  <slot id='2' type='SMP'/>
11:48:35:</config>
11:48:35:Trying to access database...
11:48:35:Successfully acquired database lock
11:48:35:Enabled folding slot 00: READY gpu:0:"GK104 [GeForce GTX 660 Ti]"
11:48:35:Enabled folding slot 01: READY gpu:1:"GF116 [GeForce GTS 450]"
11:48:35:Enabled folding slot 02: READY smp:4
11:48:35:WU00:FS01:Starting
11:48:35:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/aoeu/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 00 -suffix 01 -version 702 -lifeline 3844 -checkpoint 15 -gpu 1
11:48:35:WU00:FS01:Started FahCore on PID 3916
11:48:35:WU00:FS01:Core PID:4040
11:48:35:WU00:FS01:FahCore 0x15 started
11:48:35:WU02:FS02:Starting
11:48:35:WU02:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/aoeu/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 02 -suffix 01 -version 702 -lifeline 3844 -checkpoint 15 -np 4
11:48:35:WU02:FS02:Started FahCore on PID 124
11:48:35:WU02:FS02:Core PID:3464
11:48:35:WU02:FS02:FahCore 0xa4 started
11:48:35:WU03:FS00:Starting
11:48:35:WU03:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/aoeu/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 03 -suffix 01 -version 702 -lifeline 3844 -checkpoint 15 -gpu 0
11:48:35:WU03:FS00:Started FahCore on PID 4100
11:48:35:WU03:FS00:Core PID:4112
11:48:35:WU03:FS00:FahCore 0x15 started
11:48:36:WU00:FS01:0x15:
11:48:36:WU00:FS01:0x15:*------------------------------*
11:48:36:WU00:FS01:0x15:Folding@Home GPU Core
11:48:36:WU00:FS01:0x15:Version                2.25 (Wed May 9 17:03:01 EDT 2012)
11:48:36:WU00:FS01:0x15:Build host             AmoebaRemote
11:48:36:WU00:FS01:0x15:Board Type             NVIDIA/CUDA
11:48:36:WU00:FS01:0x15:Core                   15
11:48:36:WU00:FS01:0x15:GPU device info vendor=0 device=0 name=NA match=0 deviceId=1
11:48:36:WU00:FS01:0x15:
11:48:36:WU00:FS01:0x15:Window's signal control handler registered.
11:48:36:WU00:FS01:0x15:Preparing to commence simulation
11:48:36:WU00:FS01:0x15:- Ensuring status. Please wait.
11:48:36:WU02:FS02:0xa4:
11:48:36:WU02:FS02:0xa4:*------------------------------*
11:48:36:WU02:FS02:0xa4:Folding@Home Gromacs GB Core
11:48:36:WU02:FS02:0xa4:Version 2.27 (Dec. 15, 2010)
11:48:36:WU02:FS02:0xa4:
11:48:36:WU02:FS02:0xa4:Preparing to commence simulation
11:48:36:WU02:FS02:0xa4:- Ensuring status. Please wait.
11:48:36:WU03:FS00:0x15:
11:48:36:WU03:FS00:0x15:*------------------------------*
11:48:36:WU03:FS00:0x15:Folding@Home GPU Core
11:48:36:WU03:FS00:0x15:Version                2.25 (Wed May 9 17:03:01 EDT 2012)
11:48:36:WU03:FS00:0x15:Build host             AmoebaRemote
11:48:36:WU03:FS00:0x15:Board Type             NVIDIA/CUDA
11:48:36:WU03:FS00:0x15:Core                   15
11:48:36:WU03:FS00:0x15:
11:48:36:WU03:FS00:0x15:Window's signal control handler registered.
11:48:36:WU03:FS00:0x15:Preparing to commence simulation
11:48:36:WU03:FS00:0x15:- Ensuring status. Please wait.
11:48:38:Server connection id=1 on 0.0.0.0:36330 from 127.0.0.1
11:48:45:WU03:FS00:0x15:- Looking at optimizations...
11:48:45:WU03:FS00:0x15:- Working with standard loops on this execution.
11:48:45:WU03:FS00:0x15:- Previous termination of core was improper.
11:48:45:WU03:FS00:0x15:- Files status OK
11:48:45:WU03:FS00:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
11:48:45:WU00:FS01:0x15:- Looking at optimizations...
11:48:45:WU00:FS01:0x15:- Working with standard loops on this execution.
11:48:45:WU00:FS01:0x15:- Previous termination of core was improper.
11:48:45:WU00:FS01:0x15:- Files status OK
11:48:45:WU00:FS01:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
11:48:45:WU00:FS01:0x15:- Expanded 125064 -> 502918 (decompressed 402.1 percent)
11:48:45:WU00:FS01:0x15:Called DecompressByteArray: compressed_data_size=125064 data_size=502918, decompressed_data_size=502918 diff=0
11:48:45:WU00:FS01:0x15:- Digital signature verified
11:48:45:WU00:FS01:0x15:
11:48:45:WU00:FS01:0x15:Project: 7625 (Run 608, Clone 0, Gen 54)
11:48:45:WU00:FS01:0x15:
11:48:45:WU00:FS01:0x15:Entering M.D.
11:48:45:WU02:FS02:0xa4:- Looking at optimizations...
11:48:45:WU02:FS02:0xa4:- Working with standard loops on this execution.
11:48:45:WU02:FS02:0xa4:- Previous termination of core was improper.
11:48:45:WU02:FS02:0xa4:- Going to use standard loops.
11:48:45:WU02:FS02:0xa4:- Files status OK
11:48:45:WU03:FS00:0x15:- Expanded 126469 -> 507182 (decompressed 401.0 percent)
11:48:45:WU03:FS00:0x15:Called DecompressByteArray: compressed_data_size=126469 data_size=507182, decompressed_data_size=507182 diff=0
11:48:45:WU03:FS00:0x15:- Digital signature verified
11:48:45:WU03:FS00:0x15:
11:48:45:WU03:FS00:0x15:Project: 7624 (Run 318, Clone 0, Gen 68)
11:48:45:WU03:FS00:0x15:
11:48:45:WU03:FS00:0x15:Entering M.D.
11:48:45:WU02:FS02:0xa4:- Expanded 2079206 -> 5386224 (decompressed 259.0 percent)
11:48:45:WU02:FS02:0xa4:Called DecompressByteArray: compressed_data_size=2079206 data_size=5386224, decompressed_data_size=5386224 diff=0
11:48:45:WU02:FS02:0xa4:- Digital signature verified
11:48:45:WU02:FS02:0xa4:
11:48:45:WU02:FS02:0xa4:Project: 7809 (Run 10, Clone 446, Gen 99)
11:48:45:WU02:FS02:0xa4:
11:48:45:WU02:FS02:0xa4:Entering M.D.
11:48:47:WU00:FS01:0x15:Will resume from checkpoint file 00/wudata_01.ckp
11:48:47:WU00:FS01:0x15:Tpr hash 00/wudata_01.tpr:  2601218723 1937364931 2057422821 633386024 1606444556
11:48:47:WU00:FS01:0x15:GPU device id=1
11:48:47:WU00:FS01:0x15:Working on Protein
11:48:47:WU00:FS01:0x15:Client config unavailable.
11:48:47:WU03:FS00:0x15:Will resume from checkpoint file 03/wudata_01.ckp
11:48:47:WU03:FS00:0x15:Tpr hash 03/wudata_01.tpr:  1968629152 1570462249 1646169697 3542696308 2846552521
11:48:47:WU03:FS00:0x15:GPU device id=0
11:48:47:WU03:FS00:0x15:Working on Protein
11:48:47:WU03:FS00:0x15:Client config unavailable.
11:48:47:WU00:FS01:0x15:Starting GUI Server
11:48:47:WU03:FS00:0x15:Starting GUI Server
11:48:51:WU02:FS02:0xa4:Using Gromacs checkpoints
11:48:51:WU02:FS02:0xa4:Mapping NT from 4 to 4
11:48:52:WU02:FS02:0xa4:Resuming from checkpoint
11:48:52:WU02:FS02:0xa4:Verified 02/wudata_01.log
11:48:52:WU02:FS02:0xa4:Verified 02/wudata_01.trr
11:48:52:WU02:FS02:0xa4:Verified 02/wudata_01.xtc
11:48:52:WU02:FS02:0xa4:Verified 02/wudata_01.edr
11:48:52:WU02:FS02:0xa4:Completed 890810 out of 1500000 steps  (59%)
11:49:50:WU03:FS00:0x15:Resuming from checkpoint
11:49:50:WU03:FS00:0x15:fcCheckPointResume: retreived and current tpr file hash:
11:49:50:WU03:FS00:0x15:   0   1968629152   1968629152
11:49:50:WU03:FS00:0x15:   1   1570462249   1570462249
11:49:50:WU03:FS00:0x15:   2   1646169697   1646169697
11:49:50:WU03:FS00:0x15:   3   3542696308   3542696308
11:49:50:WU03:FS00:0x15:   4   2846552521   2846552521
11:49:50:WU03:FS00:0x15:fcCheckPointResume: file hashes same.
11:49:50:WU03:FS00:0x15:fcCheckPointResume: state restored.
11:49:50:WU03:FS00:0x15:fcCheckPointResume: name 03/wudata_01.log Verified 03/wudata_01.log
11:49:50:WU03:FS00:0x15:fcCheckPointResume: name 03/wudata_01.trr Verified 03/wudata_01.trr
11:49:50:WU03:FS00:0x15:fcCheckPointResume: name 03/wudata_01.xtc Verified 03/wudata_01.xtc
11:49:50:WU03:FS00:0x15:fcCheckPointResume: name 03/wudata_01.edr Verified 03/wudata_01.edr
11:49:50:WU03:FS00:0x15:fcCheckPointResume: state restored 2
11:49:50:WU03:FS00:0x15:Resumed from checkpoint
11:49:50:WU03:FS00:0x15:Setting checkpoint frequency: 400000
11:49:50:WU03:FS00:0x15:Completed  11200001 out of 40000000 steps (28%).
11:49:51:WARNING:WU03:FS00:Detected clock skew (1 mins 06 secs), adjusting time estimates
11:49:51:WU00:FS01:0x15:Resuming from checkpoint
11:49:51:WU00:FS01:0x15:fcCheckPointResume: retreived and current tpr file hash:
11:49:51:WU00:FS01:0x15:   0   2601218723   2601218723
11:49:51:WU00:FS01:0x15:   1   1937364931   1937364931
11:49:51:WU00:FS01:0x15:   2   2057422821   2057422821
11:49:51:WU00:FS01:0x15:   3    633386024    633386024
11:49:51:WU00:FS01:0x15:   4   1606444556   1606444556
11:49:51:WU00:FS01:0x15:fcCheckPointResume: file hashes same.
11:49:51:WU00:FS01:0x15:fcCheckPointResume: state restored.
11:49:51:WU00:FS01:0x15:fcCheckPointResume: name 00/wudata_01.log Verified 00/wudata_01.log
11:49:51:WU00:FS01:0x15:fcCheckPointResume: name 00/wudata_01.trr Verified 00/wudata_01.trr
11:49:51:WU00:FS01:0x15:fcCheckPointResume: name 00/wudata_01.xtc Verified 00/wudata_01.xtc
11:49:51:WU00:FS01:0x15:fcCheckPointResume: name 00/wudata_01.edr Verified 00/wudata_01.edr
11:49:51:WU00:FS01:0x15:fcCheckPointResume: state restored 2
11:49:51:WU00:FS01:0x15:Resumed from checkpoint
11:49:51:WU00:FS01:0x15:Setting checkpoint frequency: 400000
11:49:51:WU00:FS01:0x15:Completed   9200001 out of 40000000 steps (23%).
11:49:52:WARNING:WU00:FS01:Detected clock skew (1 mins 07 secs), adjusting time estimates
11:55:33:WU03:FS00:0x15:Completed  11600000 out of 40000000 steps (29%).
11:59:51:WU02:FS02:0xa4:Completed 900000 out of 1500000 steps  (60%)
12:01:16:WU03:FS00:0x15:Completed  12000000 out of 40000000 steps (30%).
12:05:48:WU00:FS01:0x15:Completed   9600000 out of 40000000 steps (24%).
12:06:59:WU03:FS00:0x15:Completed  12400000 out of 40000000 steps (31%).
12:12:41:WU03:FS00:0x15:Completed  12800000 out of 40000000 steps (32%).
12:17:09:WU02:FS02:0xa4:Completed 915000 out of 1500000 steps  (61%)
12:18:24:WU03:FS00:0x15:Completed  13200000 out of 40000000 steps (33%).
12:21:43:WU00:FS01:0x15:Completed  10000000 out of 40000000 steps (25%).
12:24:07:WU03:FS00:0x15:Completed  13600000 out of 40000000 steps (34%).
12:29:50:WU03:FS00:0x15:Completed  14000000 out of 40000000 steps (35%).
12:34:27:WU02:FS02:0xa4:Completed 930000 out of 1500000 steps  (62%)
12:35:33:WU03:FS00:0x15:Completed  14400000 out of 40000000 steps (36%).
12:37:38:WU00:FS01:0x15:Completed  10400000 out of 40000000 steps (26%).
12:41:16:WU03:FS00:0x15:Completed  14800000 out of 40000000 steps (37%).
12:46:59:WU03:FS00:0x15:Completed  15200000 out of 40000000 steps (38%).
12:51:47:WU02:FS02:0xa4:Completed 945000 out of 1500000 steps  (63%)
12:52:42:WU03:FS00:0x15:Completed  15600000 out of 40000000 steps (39%).
12:53:33:WU00:FS01:0x15:Completed  10800000 out of 40000000 steps (27%).
12:58:27:WU03:FS00:0x15:Completed  16000000 out of 40000000 steps (40%).
13:04:12:WU03:FS00:0x15:Completed  16400000 out of 40000000 steps (41%).
13:09:48:WU00:FS01:0x15:Completed  11200000 out of 40000000 steps (28%).
13:09:57:WU03:FS00:0x15:Completed  16800000 out of 40000000 steps (42%).
13:10:21:WU02:FS02:0xa4:Completed 960000 out of 1500000 steps  (64%)
13:15:42:WU03:FS00:0x15:Completed  17200000 out of 40000000 steps (43%).
13:21:26:WU03:FS00:0x15:Completed  17600000 out of 40000000 steps (44%).
13:25:51:WU00:FS01:0x15:Completed  11600000 out of 40000000 steps (29%).
13:27:10:WU03:FS00:0x15:Completed  18000000 out of 40000000 steps (45%).
13:28:56:WU02:FS02:0xa4:Completed 975000 out of 1500000 steps  (65%)
13:32:55:WU03:FS00:0x15:Completed  18400000 out of 40000000 steps (46%).
13:38:38:WU03:FS00:0x15:Completed  18800000 out of 40000000 steps (47%).
13:42:01:WU00:FS01:0x15:Completed  12000000 out of 40000000 steps (30%).
13:44:23:WU03:FS00:0x15:Completed  19200000 out of 40000000 steps (48%).
13:47:28:WU02:FS02:0xa4:Completed 990000 out of 1500000 steps  (66%)
13:50:06:WU03:FS00:0x15:Completed  19600000 out of 40000000 steps (49%).
13:55:49:WU03:FS00:0x15:Completed  20000000 out of 40000000 steps (50%).
13:58:07:WU00:FS01:0x15:Completed  12400000 out of 40000000 steps (31%).
14:01:34:WU03:FS00:0x15:Completed  20400000 out of 40000000 steps (51%).
14:06:01:WU02:FS02:0xa4:Completed 1005000 out of 1500000 steps  (67%)
14:07:18:WU03:FS00:0x15:Completed  20800000 out of 40000000 steps (52%).
14:13:01:WU03:FS00:0x15:Completed  21200000 out of 40000000 steps (53%).
14:14:23:WU00:FS01:0x15:Completed  12800000 out of 40000000 steps (32%).
14:18:46:WU03:FS00:0x15:Completed  21600000 out of 40000000 steps (54%).
14:24:26:WU02:FS02:0xa4:Completed 1020000 out of 1500000 steps  (68%)
14:24:30:WU03:FS00:0x15:Completed  22000000 out of 40000000 steps (55%).
14:30:14:WU03:FS00:0x15:Completed  22400000 out of 40000000 steps (56%).
14:30:34:WU00:FS01:0x15:Completed  13200000 out of 40000000 steps (33%).
14:35:58:WU03:FS00:0x15:Completed  22800000 out of 40000000 steps (57%).
14:41:43:WU03:FS00:0x15:Completed  23200000 out of 40000000 steps (58%).
14:43:13:WU02:FS02:0xa4:Completed 1035000 out of 1500000 steps  (69%)
14:46:46:WU00:FS01:0x15:Completed  13600000 out of 40000000 steps (34%).
14:47:26:WU03:FS00:0x15:Completed  23600000 out of 40000000 steps (59%).
14:53:10:WU03:FS00:0x15:Completed  24000000 out of 40000000 steps (60%).
14:58:53:WU03:FS00:0x15:Completed  24400000 out of 40000000 steps (61%).
15:01:33:WU02:FS02:0xa4:Completed 1050000 out of 1500000 steps  (70%)
15:02:56:WU00:FS01:0x15:Completed  14000000 out of 40000000 steps (35%).
15:04:38:WU03:FS00:0x15:Completed  24800000 out of 40000000 steps (62%).
15:10:22:WU03:FS00:0x15:Completed  25200000 out of 40000000 steps (63%).
15:16:07:WU03:FS00:0x15:Completed  25600000 out of 40000000 steps (64%).
15:19:16:WU00:FS01:0x15:Completed  14400000 out of 40000000 steps (36%).
15:21:01:WU02:FS02:0xa4:Completed 1065000 out of 1500000 steps  (71%)
15:21:52:WU03:FS00:0x15:Completed  26000000 out of 40000000 steps (65%).
15:27:37:WU03:FS00:0x15:Completed  26400000 out of 40000000 steps (66%).
15:33:22:WU03:FS00:0x15:Completed  26800000 out of 40000000 steps (67%).
15:35:42:WU00:FS01:0x15:Completed  14800000 out of 40000000 steps (37%).
15:39:06:WU03:FS00:0x15:Completed  27200000 out of 40000000 steps (68%).
15:40:12:WU02:FS02:0xa4:Completed 1080000 out of 1500000 steps  (72%)
15:44:51:WU03:FS00:0x15:Completed  27600000 out of 40000000 steps (69%).
15:50:36:WU03:FS00:0x15:Completed  28000000 out of 40000000 steps (70%).
15:52:02:WU00:FS01:0x15:Completed  15200000 out of 40000000 steps (38%).
15:56:20:WU03:FS00:0x15:Completed  28400000 out of 40000000 steps (71%).
15:59:36:WU02:FS02:0xa4:Completed 1095000 out of 1500000 steps  (73%)
16:02:05:WU03:FS00:0x15:Completed  28800000 out of 40000000 steps (72%).
16:05:18:WU03:FS00:0x15:Run: exception thrown in GuardedRun -- cannot continue further.
16:05:19:WU03:FS00:0x15:Going to send back what have done -- stepsTotalG=40000000
16:05:19:WU03:FS00:0x15:Work fraction=0.7255 steps=40000000.
16:05:22:WU03:FS00:0x15:logfile size=19754 infoLength=19754 edr=0 trr=23
16:05:22:WU03:FS00:0x15:+ Opened results file
16:05:22:WU03:FS00:0x15:- Writing 20290 bytes of core data to disk...
16:05:22:WU03:FS00:0x15:Done: 19778 -> 5402 (compressed to 27.3 percent)
16:05:22:WU03:FS00:0x15:  ... Done.
16:05:22:WU03:FS00:0x15:DeleteFrameFiles: successfully deleted file=03/wudata_01.ckp
16:05:23:WU03:FS00:0x15:
16:05:23:WU03:FS00:0x15:Folding@home Core Shutdown: UNSTABLE_MACHINE
16:05:23:WARNING:WU03:FS00:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
16:05:23:WU03:FS00:Sending unit results: id:03 state:SEND error:FAULTY project:7624 run:318 clone:0 gen:68 core:0x15 unit:0x0000005e664f2dd14fe6125c8e4eb7ce
16:05:23:WU03:FS00:Uploading 5.78KiB to 171.64.65.105
16:05:23:WU03:FS00:Connecting to 171.64.65.105:8080
16:05:23:WU01:FS00:Connecting to assign-GPU.stanford.edu:80
16:05:23:WU03:FS00:Upload complete
16:05:23:WU03:FS00:Server responded WORK_ACK (400)
16:05:23:WU03:FS00:Cleaning up
16:05:24:WU01:FS00:News: Welcome to Folding@Home
16:05:24:WU01:FS00:Assigned to work server 171.64.65.105
16:05:24:WU01:FS00:Requesting new work unit for slot 00: READY gpu:0:"GK104 [GeForce GTX 660 Ti]" from 171.64.65.105
16:05:24:WU01:FS00:Connecting to 171.64.65.105:8080
16:05:24:WU01:FS00:Downloading 124.31KiB
16:05:25:WU01:FS00:Download complete
16:05:25:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:7626 run:442 clone:0 gen:35 core:0x15 unit:0x00000042664f2dd14fe61c4b3bb45697
16:05:25:WU01:FS00:Starting
16:05:25:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/aoeu/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 01 -suffix 01 -version 702 -lifeline 3844 -checkpoint 15 -gpu 0
16:05:25:WU01:FS00:Started FahCore on PID 17220
16:05:25:WU01:FS00:Core PID:17000
16:05:25:WU01:FS00:FahCore 0x15 started
16:05:25:WU01:FS00:0x15:
16:05:25:WU01:FS00:0x15:*------------------------------*
16:05:25:WU01:FS00:0x15:Folding@Home GPU Core
16:05:25:WU01:FS00:0x15:Version                2.25 (Wed May 9 17:03:01 EDT 2012)
16:05:25:WU01:FS00:0x15:Build host             AmoebaRemote
16:05:25:WU01:FS00:0x15:Board Type             NVIDIA/CUDA
16:05:25:WU01:FS00:0x15:Core                   15
16:05:25:WU01:FS00:0x15:
16:05:25:WU01:FS00:0x15:Window's signal control handler registered.
16:05:25:WU01:FS00:0x15:Preparing to commence simulation
16:05:25:WU01:FS00:0x15:- Looking at optimizations...
16:05:25:WU01:FS00:0x15:DeleteFrameFiles: successfully deleted file=01/wudata_01.ckp
16:05:25:WU01:FS00:0x15:- Created dyn
16:05:25:WU01:FS00:0x15:- Files status OK
16:05:25:WU01:FS00:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
16:05:25:WU01:FS00:0x15:- Expanded 126786 -> 507182 (decompressed 400.0 percent)
16:05:25:WU01:FS00:0x15:Called DecompressByteArray: compressed_data_size=126786 data_size=507182, decompressed_data_size=507182 diff=0
16:05:25:WU01:FS00:0x15:- Digital signature verified
16:05:25:WU01:FS00:0x15:
16:05:25:WU01:FS00:0x15:Project: 7626 (Run 442, Clone 0, Gen 35)
16:05:25:WU01:FS00:0x15:
16:05:25:WU01:FS00:0x15:Assembly optimizations on if available.
16:05:25:WU01:FS00:0x15:Entering M.D.
16:05:27:WU01:FS00:0x15:Tpr hash 01/wudata_01.tpr:  2066300321 2075131559 4194280837 2111105817 2050253927
16:05:27:WU01:FS00:0x15:GPU device id=0
16:05:27:WU01:FS00:0x15:Working on Protein
16:05:27:WU01:FS00:0x15:Client config unavailable.
16:05:27:WU01:FS00:0x15:Starting GUI Server
16:06:34:WU01:FS00:0x15:Setting checkpoint frequency: 400000
16:06:34:WU01:FS00:0x15:Completed         3 out of 40000000 steps (0%).
16:08:28:WU00:FS01:0x15:Completed  15600000 out of 40000000 steps (39%).
16:13:27:FS00:Paused
16:13:27:FS00:Shutting core down
16:13:35:WU01:FS00:0x15:Client no longer detected. Shutting down core
16:13:35:WU01:FS00:0x15:
16:13:35:WU01:FS00:0x15:Folding@home Core Shutdown: CLIENT_DIED
16:13:35:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
16:18:11:WU02:FS02:0xa4:Completed 1110000 out of 1500000 steps  (74%)


[GreyWhiskers]

Three things come to mind:

- What are the drivers you are using? I see the report of CUDA Driver 5000, but that doesn't tell whether you have loaded the absolutely newest driver (which wouldn't be surprising with the installation of a new GTX660Ti high performance card) or if you have an older one like the older but reported stable 306.97 drivers.

- How is your power supply? the Nvidia specs on the GT450 shows a TDP of 106 watts. The GTX660 TI-DC2O-2GD5 (which is probably the SKU you have) has a TDP of up to 225 watts. You could be experiencing some power spikes that may be an issue if you don't have a big enough power supply.

- How is your cooling? the ASUS DC ii is an excellent two fan cooler for the new card but how's the environment in the case with everything running?

You may want to set up to monitor with afterburner. The GTX660Ti has a complex algorithm internal to it to manage the GPU voltage and boosted core clock according to the GPU temperature and power %age - even with no manual overclocking.

Just an FYI, here is a screen shot of my routine monitoring. I think I go overboard, but the stuff shown here gives me information that helps me troubleshoot situations. Briefly, this is a i7 2600k Sandy Bridge with two GPUs - GTX560Ti and GTX660Ti - 750 watt ps - Win 7 home premium - V7.1.52 and NVIDIA 306.97 drivers.

Upper right corner - real time readout from the UPS that the computer is plugged in - shows 459 watts wall-plug power in the computer case at the moment.
Afterburner screen - GPU1 at the top is the GTX660Ti showing the 1241 automatically boosted core clock and 1.175 voltage based on the 62 deg C temp and 75% power. Lower readout for GPU2 is the good temps and manually OC clock of 932 GHz.

Of course, the system power and GPU behavior is of use only in the context of exactly what work units are being processed, so I show the part of the HFM.net display just for this computer.

I apologize if this is in the "TMI" realm, but it is useful to see how all the parts fit together. The devil is often in the details.

[aoeu]

Power consumption is under 400 watts at the wall and the power supply is a Cooler Master (750?).

Driver is 310.90 and installed before buying the new card. The one on the disk was older and I didn't install any part of it.

I need to look into cooling specs but the case has a lower intake fan as well as the PS fan.

[GreyWhiskers]

What are the temps running on the GPUs? Afterburner will show that, as will GPU-z.

And per drivers, if you haven't seen it, here's the most recent post from the latest working nvidia driver thread. I'm successfully running my Sandybridge with the 550Ti and 660Ti under 306.97. I am running a Sandybridge laptop with GTX560M using 310.90, and it seems OK, with some peculiar modulation on the clocks as shown in my Afterburner graphs. But, there are a lot of negative reports.

You might try to fall back to 306.97. My desktop with my new EVGA GTX660Ti is humming along fine with these.

There continue to be many reports at the NVidia GeForce Forums, from users who have problems with the 310.70 and 310.90 drivers. Five of the top 7 "Popular Topics" there are about problems with the 310.xx drivers, especially the 310.90 driver.

Many of these users are able to restore normal operation of their system by going back to version 306.97. A typical comment, one of many similar comments in the 39 pages of the Official NVIDIA R310.70 WHQL Candidate Display Driver Feedback Thread, is this:

Upgraded from 306.97 and made my system unstable. BSOD 30-60 seconds after boot into Windows. Tried both 310.70 and 310.90 both are the same. Had to go back to 306.97, now all is peachy.

and this:

Same issues with 310.70 as I've had with each 310.xx driver.
Once agained rolled back to 306.97. I moved from an AMD 6950 that gave me zero problems driver wise to a GTX 670 expecting to take advantage of the "superior nvidia drivers". At first with 306.97 it was great, but every release since has been bad. I don't understand what's going on with this, but I've NEVER had issues like this with AMD. If it doesn't get straightened out, I might be switching back next year.

So I would suggest that the answer to the initial question about the "latest working nvidia driver", is that you should use version 306.97. A newer version may work for you, or it may not. It's just the luck of the draw.

[aoeu]

Thank you GreyWhiskers.

310.90 was working fine for me but it looks as though I will be rolling back after lunch. If that doesn't do the trick then better thermal monitoring is next.

Peace?
aoeu

[bruce]

Before concluding that the GPU might be bad, pay attention to temperature (etc.). You might also try "underclocking" (reverting to standard clocks) to confirm your supplier didn't overclock too much.

Also it's important to establish a pattern. Has that same GPU completed other WUs under similar conditions? Are others able to complete the same WU successfully? (Forum mods can check the status of project:7624 run:318 clone:0 gen:68 for you.) In this case, you were the first to return that WU and we'll have to give others time to complete the reassignment of the same WU.

[art_l_j_PlanetAMD64]

Driver is 310.90 and installed before buying the new card. The one on the disk was older and I didn't install any part of it.

The NVidia 310.90 driver has had numerous reports of causing exactly the type of problem you are experiencing. Please see the information about it here.

I have dual Gigabyte GTX 660 Ti OC (GV-N66TOC-2GD) cards in my #6 system, please see the specifications here. They self-overclock as high as 1228MHz with rock-solid stability, using the 306.97 driver, and getting as much as 39152 PPD per card. So I would suggest trying the 306.97 driver, as many users on the NVidia GeForce Forums have reported that it fixes the problems they were having with the 310.70 and 310.90 drivers.

[aoeu]

306.97 driver installed and folding.
Both GPU WUs either started over or were dropped.
GTX660TI temp is 78C, 35k ppd
GTS450 temp is 81C, 7.4k ppd which is anomalously low by a factor of nearly 2

[art_l_j_PlanetAMD64]

306.97 driver installed and folding.
Both GPU WUs either started over or were dropped.
GTX660TI temp is 78C, 35k ppd
GTS450 temp is 81C, 7.4k ppd which is anomalously low by a factor of nearly 2

Those are very high GPU temps, which will severely shorten the lifetime of your GPUs. This may also explain the low PPD on the GTS450, if it backs off on frequency due to high temperature. You should download and install a program like this:
EVGA Precision X
and use it to increase the fan speeds on your GPUs. I keep all of my GPU's temperatures below 65C.

You may also need to do something like I have shown here, to get the temperatures down to acceptable values (ie remove the side of the case and have a fan blowing on the motherboard/CPU/GPUs). Better to be safe than sorry.

[aoeu]

After three frames the GTS450 is up to 12.6k ppd which is about where it has been for a long time.

Unless there are additional failures I consider this problem solved.

Thanks for the help everyone who answered.
aoeu

[bruce]

Stock cooling was (usually) never designed for the 100% usage for both CPU and GPU, that running FAH causes. Better to be safe than sorry.

Stock cooling was designed to dissipate all of the heat the computer could generate running at 100% in a very hot habitable environment. As soon as you add or upgrade a GPU, getting the extra heat out of the case becomes your responsibility, not the responsibility of Dell/HP/Sony/Compaq/etc.

[art_l_j_PlanetAMD64]

Stock cooling was (usually) never designed for the 100% usage for both CPU and GPU, that running FAH causes. Better to be safe than sorry.

Stock cooling was designed to dissipate all of the heat the computer could generate running at 100% in a very hot habitable environment. As soon as you add or upgrade a GPU, getting the extra heat out of the case becomes your responsibility, not the responsibility of Dell/HP/Sony/Compaq/etc.

The top brand names like HP, Dell, Compaq, in a totally stock configuration, yes. Lower consumer-grade computers, maybe. And as you said, adding one or more high-performance GPUs and/or overclocking, definitely not.

[bollix47]

Work unit was returned successfully by another donor.

Hi xxxxx (team xxxx),
Your WU (P7624 R318 C0 G68) was added to the stats database on 2013-01-25 03:04:31 for 14093 points of credit.

[JimF]

Old video configuration was 2 non SLIed GeForce GTS450sc cards and ppd was about 25k.
I replaced the primary card with the one in the title. I have never done anything to overclock anything in the computer. ppd on the new card with 76xx projects is about 35k but it drops them occasionally and it just happened again. Screens go black for a couple of seconds and then it starts over. Occasionally a project will run much slower on the new card and a reboot is required to fix it.

This may be another variation on the old theme that Folding does not work well on dissimilar cards.
viewtopic.php?f=38&t=20895&p=208915

Sometimes V7 works better, sometimes V6. But it is said to be a driver issue at heart, and nothing (by PG) can apparently be done for it.

[bruce]

The "screen goes black for a couple of seconds" symptom is probably accompanied by a tiny message saying the GPU had a problem and was reset. This will also be recorded in the Event Log.

GPUs should not hang and need to be reset. This can be caused by overheating, by some impending hardware failure, by insufficient 12v power, or by bad drivers. We've had a rash of reports of bad drivers so the first thing I'd try is reverting to older drivers.