EUE: FAULTY project:7623 run:573 clone:5 gen:1 (Fermi GPU)

[GreyWhiskers]

Just noticed that my laptop GPU was running awfully quiet - looked in log and found the previous work unit (see post subject) had a thrown exception leading to an EUE.

The next work unit - Project: 7623 (Run 350, Clone 5, Gen 3) - seems to be stuck at 0% although FAHControl GUI seems to indicate the WU is clocking along (showing 2.24% of WU completed after 33 minutes since start when expected TPF is about 22 minutes. Neither the log or HFM.net are showing any progression on the next work unit beyond 0%.

I just rebooted the computer - the GPU seems to be behaving normally - Afterburner shows correct clock and expected GPU temps. It restarted the same WU from the archive at zero.

FWIW, the HFM.net Work Unit History shows 854 successful work units over the last 14 months for this GPU in essentially the same configuration.

Config:
d) 2011 lappy-15.6"-1920x1080;i7-2860QM,2.5;IC Diamond Thermal Compound;GTX 560M 1,536MB;16GB-1333MHz RAM;HDD:500GBHyb w/ 4GB SSD;Win7HomePrem64;285.30 drivers FAH 7.2.9

Core 15 version 2.25 is active. The GPU is underclocked to a core clock of 700 MHz (vs stock [EDIT] 775 MHz) for heat management.


07:51:20:WU00:FS01:0x15:Completed 15200000 out of 40000000 steps (38%).
07:59:52:WU00:FS01:0x15:Run: exception thrown in GuardedRun -- cannot continue further.
07:59:52:WU00:FS01:0x15:Going to send back what have done -- stepsTotalG=40000000
07:59:52:WU00:FS01:0x15:Work fraction=0.3839 steps=40000000.

SNIP

07:59:56:WU00:FS01:0x15:Folding@home Core Shutdown: EARLY_UNIT_END
07:59:56:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:59:56:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:7623 run:573 clone:5 gen:1 core:0x15 unit:0x00000001664f2dd14fe4fba906d787d1

Code: Select all

*********************** Log Started 2012-12-22T11:20:04Z ***********************
11:20:04:************************* Folding@home Client *************************
11:20:04:      Website: http://folding.stanford.edu/
11:20:04:    Copyright: (c) 2009-2012 Stanford University
11:20:04:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
11:20:04:         Args: --lifeline 6488 --command-port=36330
11:20:04:       Config: C:/Users/USER/AppData/Roaming/FAHClient/config.xml
11:20:04:******************************** Build ********************************
11:20:04:      Version: 7.2.9
11:20:04:         Date: Oct 3 2012
11:20:04:         Time: 18:05:48
11:20:04:      SVN Rev: 3578
11:20:04:       Branch: fah/trunk/client
11:20:04:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
11:20:04:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
11:20:04:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
11:20:04:     Platform: win32 XP
11:20:04:         Bits: 32
11:20:04:         Mode: Release
11:20:04:******************************* System ********************************
11:20:04:          CPU: Intel(R) Core(TM) i7-2860QM CPU @ 2.50GHz
11:20:04:       CPU ID: GenuineIntel Family 6 Model 42 Stepping 7
11:20:04:         CPUs: 8
11:20:04:       Memory: 15.98GiB
11:20:04:  Free Memory: 13.12GiB
11:20:04:      Threads: WINDOWS_THREADS
11:20:04:   On Battery: false
11:20:04:   UTC offset: -8
11:20:04:          PID: 9180
11:20:04:          CWD: C:/Users/USER/AppData/Roaming/FAHClient
11:20:04:           OS: Windows 7 Home Premium
11:20:04:      OS Arch: AMD64
11:20:04:         GPUs: 1
11:20:04:        GPU 0: NVIDIA:2 GF116 [GeForce GTX 560M]
11:20:04:         CUDA: 2.1
11:20:04:  CUDA Driver: 4010
11:20:04:Win32 Service: false
11:20:04:***********************************************************************
11:20:04:<config>
11:20:04:  <!-- Logging -->
11:20:04:  <log-rotate-max v='60'/>
11:20:04:
11:20:04:  <!-- Network -->
11:20:04:  <proxy v=':8080'/>
11:20:04:
11:20:04:  <!-- Remote Command Server -->
11:20:04:  <password v='********************************'/>
11:20:04:
11:20:04:  <!-- Slot Control -->
11:20:04:  <pause-on-battery v='true'/>
11:20:04:
11:20:04:  <!-- User Information -->
11:20:04:  <passkey v='********************************'/>
11:20:04:  <user v='GreyWhiskers'/>
11:20:04:
11:20:04:  <!-- Work Unit Control -->
11:20:04:  <next-unit-percentage v='100'/>
11:20:04:
11:20:04:  <!-- Folding Slots -->
11:20:04:  <slot id='0' type='SMP'>
11:20:04:    <client-type v='beta'/>
11:20:04:    <cpus v='-1'/>
11:20:04:  </slot>
11:20:04:  <slot id='1' type='GPU'>
11:20:04:    <client-type v='beta'/>
11:20:04:    <gpu-usage v='75'/>
11:20:04:  </slot>
11:20:04:</config>

START OF EUE WORK UNIT:

16:59:58:WU00:FS01:Starting
16:59:58:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:/Users/USER/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe" -dir 00 -suffix 01 -version 702 -lifeline 9180 -checkpoint 15 -gpu 0
16:59:58:WU00:FS01:Started FahCore on PID 1216
16:59:58:WU00:FS01:Core PID:4880
16:59:58:WU00:FS01:FahCore 0x15 started
16:59:58:WU00:FS01:0x15:
16:59:58:WU00:FS01:0x15:*------------------------------*
16:59:58:WU00:FS01:0x15:Folding@Home GPU Core
16:59:58:WU00:FS01:0x15:Version                2.25 (Wed May 9 17:03:01 EDT 2012)
16:59:58:WU00:FS01:0x15:Build host             AmoebaRemote
16:59:58:WU00:FS01:0x15:Board Type             NVIDIA/CUDA
16:59:58:WU00:FS01:0x15:Core                   15
16:59:58:WU00:FS01:0x15:
16:59:58:WU00:FS01:0x15:Window's signal control handler registered.
16:59:58:WU00:FS01:0x15:Preparing to commence simulation
16:59:58:WU00:FS01:0x15:- Looking at optimizations...
16:59:58:WU00:FS01:0x15:DeleteFrameFiles: successfully deleted file=00/wudata_01.ckp
16:59:58:WU00:FS01:0x15:- Created dyn
16:59:58:WU00:FS01:0x15:- Files status OK
16:59:58:WU00:FS01:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
16:59:58:WU00:FS01:0x15:- Expanded 125014 -> 501826 (decompressed 401.4 percent)
16:59:58:WU00:FS01:0x15:Called DecompressByteArray: compressed_data_size=125014 data_size=501826, decompressed_data_size=501826 diff=0
16:59:59:WU00:FS01:0x15:- Digital signature verified
16:59:59:WU00:FS01:0x15:
16:59:59:WU00:FS01:0x15:Project: 7623 (Run 573, Clone 5, Gen 1)
16:59:59:WU00:FS01:0x15:
16:59:59:WU00:FS01:0x15:Assembly optimizations on if available.
16:59:59:WU00:FS01:0x15:Entering M.D.
17:00:00:WU01:FS01:Upload complete
17:00:00:WU00:FS01:0x15:Tpr hash 00/wudata_01.tpr:  2919284196 2353584001 2776102465 3694188980 1050966890
17:00:00:WU01:FS01:Server responded WORK_ACK (400)
17:00:00:WU00:FS01:0x15:GPU device id=0
17:00:00:WU01:FS01:Final credit estimate, 14093.00 points
17:00:00:WU01:FS01:Cleaning up
17:00:01:WU00:FS01:0x15:Working on Protein
17:00:01:WU00:FS01:0x15:Client config unavailable.
17:00:01:WU00:FS01:0x15:Starting GUI Server
17:01:09:WU00:FS01:0x15:Setting checkpoint frequency: 400000
17:01:09:WU00:FS01:0x15:Completed         3 out of 40000000 steps (0%).
******************************** Date: 22/12/12 ********************************
17:25:36:WU00:FS01:0x15:Completed    400000 out of 40000000 steps (1%).

SNIP

07:07:20:WU00:FS01:0x15:Completed  14400000 out of 40000000 steps (36%).
07:29:26:WU00:FS01:0x15:Completed  14800000 out of 40000000 steps (37%).
07:51:20:WU00:FS01:0x15:Completed  15200000 out of 40000000 steps (38%).
07:59:52:WU00:FS01:0x15:Run: exception thrown in GuardedRun -- cannot continue further.
07:59:52:WU00:FS01:0x15:Going to send back what have done -- stepsTotalG=40000000
07:59:52:WU00:FS01:0x15:Work fraction=0.3839 steps=40000000.
07:59:56:WU00:FS01:0x15:logfile size=21082 infoLength=21082 edr=0 trr=23
07:59:56:WU00:FS01:0x15:+ Opened results file
07:59:56:WU00:FS01:0x15:- Writing 21618 bytes of core data to disk...
07:59:56:WU00:FS01:0x15:Done: 21106 -> 5483 (compressed to 25.9 percent)
07:59:56:WU00:FS01:0x15:  ... Done.
07:59:56:WU00:FS01:0x15:DeleteFrameFiles: successfully deleted file=00/wudata_01.ckp
07:59:56:WU00:FS01:0x15:
07:59:56:WU00:FS01:0x15:Folding@home Core Shutdown: EARLY_UNIT_END
07:59:56:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:59:56:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:7623 run:573 clone:5 gen:1 core:0x15 unit:0x00000001664f2dd14fe4fba906d787d1
07:59:56:WU00:FS01:Uploading 5.85KiB to 171.64.65.105
07:59:56:WU00:FS01:Connecting to 171.64.65.105:8080
07:59:56:WU00:FS01:Upload complete
07:59:56:WU00:FS01:Server responded WORK_ACK (400)
07:59:56:WU00:FS01:Cleaning up
07:59:56:WU02:FS01:Connecting to assign-GPU.stanford.edu:80
07:59:56:WU02:FS01:News: Welcome to Folding@Home
07:59:56:WU02:FS01:Assigned to work server 171.64.65.105
07:59:56:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:"GF116 [GeForce GTX 560M]" from 171.64.65.105
07:59:56:WU02:FS01:Connecting to 171.64.65.105:8080
07:59:57:WU02:FS01:Downloading 121.88KiB
07:59:57:WU02:FS01:Download complete
07:59:57:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:7623 run:350 clone:5 gen:3 core:0x15 unit:0x00000004664f2dd14fe4fa6d97222356
07:59:57:WU02:FS01:Starting
07:59:57:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:/Users/USER/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe" -dir 02 -suffix 01 -version 702 -lifeline 9180 -checkpoint 15 -gpu 0
07:59:57:WU02:FS01:Started FahCore on PID 5012
07:59:57:WU02:FS01:Core PID:3332
07:59:57:WU02:FS01:FahCore 0x15 started
07:59:58:WU02:FS01:0x15:
07:59:58:WU02:FS01:0x15:*------------------------------*
07:59:58:WU02:FS01:0x15:Folding@Home GPU Core
07:59:58:WU02:FS01:0x15:Version                2.25 (Wed May 9 17:03:01 EDT 2012)
07:59:58:WU02:FS01:0x15:Build host             AmoebaRemote
07:59:58:WU02:FS01:0x15:Board Type             NVIDIA/CUDA
07:59:58:WU02:FS01:0x15:Core                   15
07:59:58:WU02:FS01:0x15:
07:59:58:WU02:FS01:0x15:Window's signal control handler registered.
07:59:58:WU02:FS01:0x15:Preparing to commence simulation
07:59:58:WU02:FS01:0x15:- Looking at optimizations...
07:59:58:WU02:FS01:0x15:DeleteFrameFiles: successfully deleted file=02/wudata_01.ckp
07:59:58:WU02:FS01:0x15:- Created dyn
07:59:58:WU02:FS01:0x15:- Files status OK
07:59:58:WU02:FS01:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
07:59:58:WU02:FS01:0x15:- Expanded 124298 -> 501826 (decompressed 403.7 percent)
07:59:58:WU02:FS01:0x15:Called DecompressByteArray: compressed_data_size=124298 data_size=501826, decompressed_data_size=501826 diff=0
07:59:58:WU02:FS01:0x15:- Digital signature verified
07:59:58:WU02:FS01:0x15:
07:59:58:WU02:FS01:0x15:Project: 7623 (Run 350, Clone 5, Gen 3)
07:59:58:WU02:FS01:0x15:
07:59:58:WU02:FS01:0x15:Assembly optimizations on if available.
07:59:58:WU02:FS01:0x15:Entering M.D.
07:59:59:WU02:FS01:0x15:Tpr hash 02/wudata_01.tpr:  3133075595 4278542008 2330310594 2433961574 1992594013
07:59:59:WU02:FS01:0x15:GPU device id=0
07:59:59:WU02:FS01:0x15:Working on Protein
07:59:59:WU02:FS01:0x15:Client config unavailable.
08:00:00:WU02:FS01:0x15:Starting GUI Server
08:01:11:WU02:FS01:0x15:Setting checkpoint frequency: 400000
08:01:11:WU02:FS01:0x15:Completed         3 out of 40000000 steps (0%).

[NOTE AT 08:43 LOG STILL SHOWS 0% ON NEXT WORK UNIT]

[bollix47]

Thanks for the report.

Hi GreyWhiskers (team 0),
Your WU (P7623 R573 C5 G1) was added to the stats database on 2012-12-23 00:04:24 for 0 points of credit.

Your return is the only one so far in the database so I've marked it for a follow-up.

[GreyWhiskers]

@bollix47 - thanks for the update. I think that this is my first EUE on the 7623-24-25-26 class since October 2012 --- I've successfully finished some 36 of them on 4 different GPUs (GT430, this laptop's GTX560M, GTX560TI, and GTX660Ti). They are stressful on the hardware though. A

[art_l_j_PlanetAMD64]

@bollix47 - thanks for the update. I think that this is my first EUE on the 7623-24-25-26 class since October 2012 --- I've successfully finished some 36 of them on 4 different GPUs (GT430, this laptop's GTX560M, GTX560TI, and GTX660Ti). They are stressful on the hardware though.

Good stress though, I like getting these Projects on my two GTX460s:
1) PRCG 7626 (614, 1, 2); TPF 11 mins 01 secs; 18421 PPD
2) PRCG 7623 (229, 0, 41); TPF 10 mins 52 secs; 18675 PPD

[GreyWhiskers]

I found one of the projects where I have stats from all 3 of my active GPU folders - 7625. All the members of the 7623-26 family act similarly.

All of these are on Core 15 v2.25. Some of the earliest of the 762x in October were on 2.22, which had significantly faster performance on the GT430 and GTX560M. For example, on the GTX560M, there is one Proj 7625 (that HFM didn't apparently include in its benchmark set) on 2.22 with a TPF of 17:09. The next 7625 on the 560M under 2.25 slowed down to over 25 minutes TPF.

These projects seem ideally suited for the GTX660Ti Kepler - even though the WUs put a lot of stress on the system, the card keeps under critical temps and runs at about 93% of the power profile on the Kepler, so the firmware maintains maximum voltage and therefore maximum boosted clock rate. The fans run at about 75% plus/minus. The firmware always tweaks the fan rate, so the temps stay at a steady 54-56 deg C.

I have some good Afterburner screenshots on the Kepler boost profile that I will post as a tutorial with my observations of how the Kepler boost works when I have the time.

Code: Select all

 HFM.net Benchmarks Viewer stats

Project ID: 7625
 Core: OPENMMGPU
 Credit: 14093
 
 Name: Sager8150 Slot 01 [GTx560M laptop underclocked to 700 MHz from stock 775 for heat control]
 Number of Frames Observed: 100

 Min. Time / Frame : 00:20:06 - 10,096.5 PPD
 Avg. Time / Frame : 00:24:02 - 8,444.1 PPD


 Name: DigiStorm Slot 00 [Kepler GTX660Ti at boosted clock ~ 1251 MHz - stock EVGA FTW Signature 2 - no manual tweaks]
 Number of Frames Observed: 199

 Min. Time / Frame : 00:05:09 - 39,405.7 PPD
 Avg. Time / Frame : 00:05:12 - 39,026.8 PPD


 Name: HP a475c Slot 01 [GT430 - PCI non-Express bus - no built-in fan - OC to 900 MHz core from stock 700 MHz; OC 850 MHz Memory clocks]
 Number of Frames Observed: 70

 Min. Time / Frame : 00:31:17 - 6,487.1 PPD
 Avg. Time / Frame : 00:32:05 - 6,325.4 PPD
 Cur. Time / Frame : 00:32:48 - 6,187.2 PPD
 R3F. Time / Frame : 00:32:48 - 6,187.2 PPD
 All  Time / Frame : 00:32:42 - 6,206.1 PPD
 Eff. Time / Frame : 00:36:33 - 5,552.4 PPD

[bollix47]

Another folder was able to complete the WU for full credit:

Hi xxxxxxxx (team xxxxxx),
Your WU (P7623 R573 C5 G1) was added to the stats database on 2012-12-25 09:04:55 for 14093 points of credit.

[mmonnin]

It's been mentioned here somewhere that the 762x WUs run hotter than other WUs for nV cards.

[artoar_11]

It's been mentioned here somewhere that the 762x WUs run hotter than other WUs for nV cards.

Yes, 726x WUs are hotter than others.