G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED Loop

Moderators: Site Moderators, FAHC Science Team

Re: G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED L

Postby bollix47 » Sat Dec 08, 2012 8:02 pm

Welcome to the folding support forum smackiethefrog.

You appear to be using very recent drivers so that eliminates one of my concerns in the post above yours. The more info we can supply, the closer the solution. Thanks.
Image
bollix47
 
Posts: 2868
Joined: Sun Dec 02, 2007 6:04 am
Location: Canada

Re: G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED L

Postby SomeStones » Sat Dec 08, 2012 8:11 pm

@Sliced

Ugh... there goes the 762X vs. 807X WU theory.

I don't know where to start. It sounds like we have similar cards. Mine is an EVGA 512-P3-N872-AR, 512MB. The person I got it from called "Superclocked" and it does run at 800MHz (GTX+) as opposed to 675MHz for the regular GTX. Mine is in Slot 0. CUDA is 1.1. CUDA Driver is 4000. I am using no User Variables.

You might try slowing it down and see if it passes the mem test then, but as you say, there has never been a problem before. Perhaps the checking module is looking for a different memory size.

Good luck.
SomeStones
 
Posts: 53
Joined: Wed Aug 26, 2009 1:10 am

Re: G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED L

Postby art_l_j_PlanetAMD64 » Sat Dec 08, 2012 8:16 pm

The 2 GPUs that were failing are working OK now:
1) WinXP/GTX275: PRCG 8054 (0, 1179, 49); FahCore 0x15; Estimated TPF 4 mins 12 secs
2) Win7Pro/GTX285: PRCG 8054 (0, 2078, 40); FahCore 0x15; Estimated TPF 4 mins 30 secs
Thanks!
art_l_j_PlanetAMD64
Over 1.04 Billion Total Points
Over 185,000 Work Units
Over 3,800,000 PPD
Overall rank (if points are combined) 20 of 1721690
In memory of my Mother May 12th 1923 - February 10th 2012
art_l_j_PlanetAMD64
 
Posts: 472
Joined: Sun May 30, 2010 3:28 pm

Re: G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED L

Postby Sliced » Sat Dec 08, 2012 8:26 pm

@SomeStones
It's ok, I appreciate your help.
Mine is also 512MB , Stock clocks are 778 / 1836 / 1123 .
I have had it overclocked to 810 / 1998 / 1160 for a few years and all has been fine.
It is a EVGA 9800GTX+ (6 pin power).
I am currently running driver version 306.97 (I do not like using beta drivers for anything.)
Sliced
 
Posts: 11
Joined: Sat Dec 08, 2012 1:29 pm

Re: G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED L

Postby AETiglathPZ » Sat Dec 08, 2012 8:59 pm

This error loop is comming up in 7.2.9 and GPU XP_XP-641 on my Asus GTX275 running XP at stock clocks.

Started getting it on with the GTX275 with GPU XP_XP-641 using the Nvidia 256.00 (12/22/2009) XP driver.
(Would run it with FAH6.34win32-SMP at the time before I updated to 7.2.9.)

Updated to XP driver 306.81, deleted work unit, GPU XP_XP-641 still a nogo.
Updated to Folding client 7.2.9 and still a nogo.

00 Ready GPU: 0: GT200b (GeForce GTX 275)
Status Update_Core
Base and Est credit of 14093.00
project 7624
Fahcore 0x15
Waiting On FahCore Update
Attempts 0
Next Attempt 36min 14 secs
Work Server 171.64.65.105
Collection Server 171.65.103.160

My 560ti is running pretty nicely, no problem at all.
Last edited by AETiglathPZ on Sat Dec 08, 2012 9:44 pm, edited 1 time in total.
AETiglathPZ
 
Posts: 1
Joined: Sat Dec 08, 2012 8:37 pm

Re: G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED L

Postby Ravage7779 » Sat Dec 08, 2012 10:06 pm

No joy, client fails instantly, no reason given. Have new issues tho. core_15 hits the uac, and requires manual intervention to run. Upon running, forces a screen resolution change to 640x480 (whee!). core fails, but is left running, when FAH tries again, it spawn a new instance of core_15.
Code: Select all
*********************** Log Started 2012-12-08T20:54:12Z ***********************
20:54:13:WU01:FS01:Cleaning up
20:54:13:WU01:FS01:Connecting to assign-GPU.stanford.edu:80
20:54:15:WU01:FS01:News: Welcome to Folding@Home
20:54:15:WU01:FS01:Assigned to work server 171.67.108.143
20:54:15:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:"G92 [GeForce 9800 GT]" from 171.67.108.143
20:54:15:WU01:FS01:Connecting to 171.67.108.143:8080
20:54:15:WU01:FS01:Downloading 59.29KiB
20:54:15:WU01:FS01:Download complete
20:54:15:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:8054 run:0 clone:2530 gen:50 core:0x15 unit:0x0000003a6953ee2f50626b5570b209fd
20:54:15:WU01:FS01:Downloading core from http://www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah
20:54:15:WU01:FS01:Connecting to www.stanford.edu:80
20:54:16:WU01:FS01:FahCore 15: Downloading 1.88MiB
20:54:21:WU01:FS01:FahCore 15: Download complete
20:54:21:WU01:FS01:Valid core signature
20:54:21:WU01:FS01:Unpacked 7.71MiB to cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah/FahCore_15.exe
20:54:21:WU01:FS01:Starting
20:54:21:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nate/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah/FahCore_15.exe -dir 01 -suffix 01 -version 702 -lifeline 3476 -checkpoint 15 -gpu 0
20:54:21:WU01:FS01:Started FahCore on PID 2620
20:54:27:WU01:FS01:Core PID:1840
20:54:27:WU01:FS01:FahCore 0x15 started
20:54:27:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
20:54:27:WU01:FS01:Starting
20:54:27:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nate/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah/FahCore_15.exe -dir 01 -suffix 01 -version 702 -lifeline 3476 -checkpoint 15 -gpu 0
20:54:27:WU01:FS01:Started FahCore on PID 440
20:54:28:WU01:FS01:Core PID:3208
20:54:28:WU01:FS01:FahCore 0x15 started
20:54:28:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
20:55:27:WU01:FS01:Starting
20:55:27:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nate/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah/FahCore_15.exe -dir 01 -suffix 01 -version 702 -lifeline 3476 -checkpoint 15 -gpu 0
20:55:27:WU01:FS01:Started FahCore on PID 2332
*********************** Log Started 2012-12-08T20:54:12Z ***********************
20:54:13:WU01:FS01:Cleaning up
20:54:13:WU01:FS01:Connecting to assign-GPU.stanford.edu:80
20:54:15:WU01:FS01:News: Welcome to Folding@Home
20:54:15:WU01:FS01:Assigned to work server 171.67.108.143
20:54:15:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:"G92 [GeForce 9800 GT]" from 171.67.108.143
20:54:15:WU01:FS01:Connecting to 171.67.108.143:8080
20:54:15:WU01:FS01:Downloading 59.29KiB
20:54:15:WU01:FS01:Download complete
20:54:15:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:8054 run:0 clone:2530 gen:50 core:0x15 unit:0x0000003a6953ee2f50626b5570b209fd
20:54:15:WU01:FS01:Downloading core from http://www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah
20:54:15:WU01:FS01:Connecting to www.stanford.edu:80
20:54:16:WU01:FS01:FahCore 15: Downloading 1.88MiB
20:54:21:WU01:FS01:FahCore 15: Download complete
20:54:21:WU01:FS01:Valid core signature
20:54:21:WU01:FS01:Unpacked 7.71MiB to cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah/FahCore_15.exe
20:54:21:WU01:FS01:Starting
20:54:21:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nate/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah/FahCore_15.exe -dir 01 -suffix 01 -version 702 -lifeline 3476 -checkpoint 15 -gpu 0
20:54:21:WU01:FS01:Started FahCore on PID 2620
20:54:27:WU01:FS01:Core PID:1840
20:54:27:WU01:FS01:FahCore 0x15 started
20:54:27:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
20:54:27:WU01:FS01:Starting
20:54:27:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nate/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah/FahCore_15.exe -dir 01 -suffix 01 -version 702 -lifeline 3476 -checkpoint 15 -gpu 0
20:54:27:WU01:FS01:Started FahCore on PID 440
20:54:28:WU01:FS01:Core PID:3208
20:54:28:WU01:FS01:FahCore 0x15 started
20:54:28:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
20:55:27:WU01:FS01:Starting
20:55:27:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nate/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah/FahCore_15.exe -dir 01 -suffix 01 -version 702 -lifeline 3476 -checkpoint 15 -gpu 0
20:55:27:WU01:FS01:Started FahCore on PID 2332
20:56:00:WU01:FS01:Core PID:3888
20:56:00:WU01:FS01:FahCore 0x15 started
20:56:01:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
20:57:04:WU01:FS01:Starting
20:57:04:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nate/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah/FahCore_15.exe -dir 01 -suffix 01 -version 702 -lifeline 3476 -checkpoint 15 -gpu 0
20:57:04:WU01:FS01:Started FahCore on PID 896
20:57:11:WU01:FS01:Core PID:4004
20:57:11:WU01:FS01:FahCore 0x15 started
20:57:11:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
20:59:41:WU01:FS01:Starting
20:59:41:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nate/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah/FahCore_15.exe -dir 01 -suffix 01 -version 702 -lifeline 3476 -checkpoint 15 -gpu 0
20:59:41:WU01:FS01:Started FahCore on PID 3916
20:59:43:WU01:FS01:Core PID:2260
20:59:43:WU01:FS01:FahCore 0x15 started
20:59:43:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
20:59:43:WARNING:WU01:FS01:Too many errors, failing
20:59:43:WU01:FS01:Sending unit results: id:01 state:SEND error:FAILED project:8054 run:0 clone:2530 gen:50 core:0x15 unit:0x0000003a6953ee2f50626b5570b209fd
20:59:43:WU01:FS01:Connecting to 171.67.108.143:8080
20:59:43:WU02:FS01:Connecting to assign-GPU.stanford.edu:80
20:59:44:WU02:FS01:News: Welcome to Folding@Home
20:59:44:WU02:FS01:Assigned to work server 171.67.108.143
20:59:44:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:"G92 [GeForce 9800 GT]" from 171.67.108.143
20:59:44:WU02:FS01:Connecting to 171.67.108.143:8080
20:59:44:WU01:FS01:Server responded WORK_QUIT (404)
20:59:44:WARNING:WU01:FS01:Server did not like results, dumping
20:59:44:WU01:FS01:Cleaning up
20:59:44:ERROR:WU01:FS01:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The directory is not empty: ".\work\01"
20:59:44:WU01:FS01:Cleaning up
20:59:44:ERROR:WU01:FS01:Exception: Failed to remove directory './work/01': boost::filesystem:remove_all: Access is denied: ".\work\01"
20:59:44:WU02:FS01:Downloading 59.22KiB
20:59:44:WU02:FS01:Download complete
20:59:44:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:8054 run:0 clone:2778 gen:22 core:0x15 unit:0x0000001e6953ee2f50626b6abb9bb350
20:59:45:WU02:FS01:Starting
20:59:45:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nate/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah/FahCore_15.exe -dir 02 -suffix 01 -version 702 -lifeline 3476 -checkpoint 15 -gpu 0
20:59:45:WU02:FS01:Started FahCore on PID 428
20:59:48:WU02:FS01:Core PID:584
20:59:48:WU02:FS01:FahCore 0x15 started
20:59:48:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
20:59:48:WU02:FS01:Starting
20:59:48:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nate/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah/FahCore_15.exe -dir 02 -suffix 01 -version 702 -lifeline 3476 -checkpoint 15 -gpu 0
20:59:48:WU02:FS01:Started FahCore on PID 3468
20:59:50:WU02:FS01:Core PID:3336
20:59:50:WU02:FS01:FahCore 0x15 started
20:59:50:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
21:00:44:WU01:FS01:Cleaning up
21:00:48:WU02:FS01:Starting
21:00:48:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nate/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_15.fah/FahCore_15.exe -dir 02 -suffix 01 -version 702 -lifeline 3476 -checkpoint 15 -gpu 0
21:00:48:WU02:FS01:Started FahCore on PID 3404
21:00:51:WU02:FS01:Core PID:2932
21:00:51:WU02:FS01:FahCore 0x15 started
21:00:51:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
21:01:41:FS01:Paused
Get more Mac OS X folding help from the friendly experts @ http://www.foldingathome.org/index.html
Image
User avatar
Ravage7779
 
Posts: 144
Joined: Sun Jun 21, 2009 11:42 pm

Re: G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED L

Postby Mactin » Sat Dec 08, 2012 10:37 pm

I to am having the GPU_MEMTEST_ERROR.
Stock GTS250 that has successfully folded almost 14k WUs mostly Core_11, but a few Core_14 and Core_15.
I have even tried reverting to the good old v6 client. It downloaded a fresh 2.25 Core_15, but failed with the same GPU_MEMTEST_ERROR.
Image
Mactin
 
Posts: 222
Joined: Sun Dec 02, 2007 2:08 pm
Location: Côte-des-Neiges, Montréal, Québec

Re: G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED L

Postby VijayPande » Sat Dec 08, 2012 10:48 pm

thanks for the reports. Regarding the GPU_MEMTEST_ERROR, the old core was really lax about testing and let a lot of things go that it shouldn't. I'm open to the possiblility that the new core is too strict, but we have done a lot of testing of this part. For now, we'll investigate what's up with the MEMTEST_ERROR issue, but this one won't be an easy fix (i.e. will take a new core, so it won't be ready this weekend).

I'll also look to see if we can get some of the old projects that use core_11 back up, but those were pretty much finished, although a bit more statistics is never a bad thing.
Prof. Vijay Pande, PhD
Departments of Chemistry, Structural Biology, and Computer Science
Chair, Biophysics
Director, Folding@home Distributed Computing Project
Stanford University
VijayPande
Pande Group Member
 
Posts: 2058
Joined: Fri Nov 30, 2007 7:25 am
Location: Stanford

Re: G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED L

Postby VijayPande » Sat Dec 08, 2012 11:09 pm

I've started a new thread for the GPU_MEMTEST_ERROR issues.
viewtopic.php?f=85&t=23200
and in particular asked for help with fixing that there (in particular what WS are you getting sent to?). TIA.
VijayPande
Pande Group Member
 
Posts: 2058
Joined: Fri Nov 30, 2007 7:25 am
Location: Stanford

Re: G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED L

Postby Sliced » Sat Dec 08, 2012 11:09 pm

VijayPande wrote:thanks for the reports. Regarding the GPU_MEMTEST_ERROR, the old core was really lax about testing and let a lot of things go that it shouldn't. I'm open to the possiblility that the new core is too strict, but we have done a lot of testing of this part. For now, we'll investigate what's up with the MEMTEST_ERROR issue, but this one won't be an easy fix (i.e. will take a new core, so it won't be ready this weekend).

I'll also look to see if we can get some of the old projects that use core_11 back up, but those were pretty much finished, although a bit more statistics is never a bad thing.


HI Vijay,
So what you'r saying is, there is a very good chance that my perfectly good 9800gtx+ will probably never fold again when it has been folding for a few years without a problem?
Sliced
 
Posts: 11
Joined: Sat Dec 08, 2012 1:29 pm

Re: G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED L

Postby dnamechanic » Sat Dec 08, 2012 11:13 pm

If it is a hardware failure, it is an unusual one.

Three 8800GT have folded for about three years, they all came down with this symptom last night.

The three 8800GT were all tested with a G80 version of MemTest many times, tho not recently.

None of the GPU's are overclocked.

----------------------------------------------------

Wonder if this relates to amount of memory available?

Although others that have reported failure, undoubtedly they have more available memory.

These particular 8800's here, were chosen to have only 256MB RAM.

Do not have other GPU's here, to compare with.
dnamechanic
 
Posts: 16
Joined: Tue Dec 04, 2007 11:42 pm
Location: Dallas, Texas

Re: G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED L

Postby VijayPande » Sat Dec 08, 2012 11:15 pm

PS Let's keep this thread for the core_outdated issue and use the new thread.

With all the reports, it seems like the memtest issue is something on our side (nothing wrong with the GPUs). We're looking into it right now.
VijayPande
Pande Group Member
 
Posts: 2058
Joined: Fri Nov 30, 2007 7:25 am
Location: Stanford

Re: FahCore Update CORE_OUTDATED Loop

Postby JCM3500 » Sun Dec 09, 2012 4:44 am

bollix47 wrote:There is nothing anyone but PG can do to fix the assignment problem.

Please see the following link for instructions to pause and delete the work unit:

viewtopic.php?p=231253#p231253

Leave your GPU slots paused and do not restart the client until PG has signified that the assignment problem has been fixed. Otherwise, you'll just end up dumping more work units which is not good for the project.


Done and waiting. WGG tasks (CPU & GPU) being processed on three PCs. This FAH failure is a disapointment.... :?
JCM3500
 
Posts: 21
Joined: Thu Oct 25, 2012 3:52 pm
Location: Whitby, ON, Canada

Re: FahCore Update CORE_OUTDATED Loop

Postby k1wi » Sun Dec 09, 2012 4:53 am

JCM3500 wrote:Done and waiting. WGG tasks (CPU & GPU) being processed on three PCs. This FAH failure is a disapointment.... :?
It's an unfortunate situation. No one wants technical issues to prevent people folding for even the shortest period of time. Unfortunately with such a large and complex operation as PG is, perfect all the time is nigh on impossible.

I don't think anyone would be more frustrated about the issue than PG themselves.
k1wi
 
Posts: 910
Joined: Tue Sep 22, 2009 11:48 pm

Re: G80 GPUs being assigned Fermi/Kepler WUs CORE_OUTDATED L

Postby VijayPande » Sun Dec 09, 2012 7:06 am

We've put in a new patch on the AS. We think this should fix the problem, but time will tell.
VijayPande
Pande Group Member
 
Posts: 2058
Joined: Fri Nov 30, 2007 7:25 am
Location: Stanford

PreviousNext

Return to V7.2.x -- Windows/Linux Release & OSX Beta

Who is online

Users browsing this forum: No registered users and 1 guest

cron