Core Crash?

Alan C. Lawhon · Post by **Alan C. Lawhon** » Tue Jan 22, 2013 6:40 am

Not sure what to make of this ... A friend informed me that core crashes are “exceedingly rare,” so I should go ahead and post this although it appears (to my inexperienced eye) that my FAH client went on to another work unit – or continued processing the one that crashed. All seems to be working fine now, so I don’t know if this is a big deal or not. (In over a month of continuous 24/7 processing, this is the first time I’ve encountered an error of this nature.) The [Windows?] error message dialogue I received concurrent with this anomaly (or whatever it is) requested that I send an error report to Microsoft - which I did. (System information immediately follows code tags.)

Code: Select all

20:25:46:WU02:FS00:0x11:Completed 99%
20:25:47:WU00:FS00:Connecting to assign-GPU.stanford.edu:80
20:25:48:WU00:FS00:News: Welcome to Folding@Home
20:25:48:WU00:FS00:Assigned to work server 171.67.108.21
20:25:48:WU00:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:"G94 [GeForce 9600 GT]" from 171.67.108.21
20:25:48:WU00:FS00:Connecting to 171.67.108.21:8080
20:25:48:WU
00:FS00:Downloading 61.85KiB
20:25:49:WU00:FS00:Download complete
20:25:49:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:10504 run:76 clone:1 gen:1241 core:0x11 unit:0x00000b076652eda54b75aeed000029f6
20:25:49:WU00:FS00:Downloading project 10504 description
20:25:49:WU00:FS00:Connecting to fah-web.stanford.edu:80
20:25:49:WU00:FS00:Project 10504 description downloaded successfully
20:28:16:WU02:FS00:0x11:Completed 100%
20:28:16:WU02:FS00:0x11:Successful run
20:28:16:WU02:FS00:0x11:DynamicWrapper: Finished Work Unit: sleep=10000
20:28:26:WU02:FS00:0x11:Reserved 109504 bytes for xtc file; Cosm status=0
20:28:26:WU02:FS00:0x11:Allocated 109504 bytes for xtc file
20:28:26:WU02:FS00:0x11:- Reading up to 109504 from "02/wudata_01.xtc": Read 109504
20:28:26:WU02:FS00:0x11:Read 109504 bytes from xtc file; available packet space=786320960
20:28:26:WU02:FS00:0x11:xtc file hash check passed.
20:28:26:WU02:FS00:0x11:Reserved 21912 21912 786320960 bytes for arc file=<02/wudata_01.trr> Cosm status=0
20:28:26:WU02:FS00:0x11:Allocated 21912 bytes for arc file
20:28:26:WU02:FS00:0x11:- Reading up to 21912 from "02/wudata_01.trr": Read 21912
20:28:26:WU02:FS00:0x11:Read 21912 bytes from arc file; available packet space=786299048
20:28:26:WU02:FS00:0x11:trr file hash check passed.
20:28:26:WU02:FS00:0x11:Allocated 560 bytes for edr file
20:28:26:WU02:FS00:0x11:Read bedfile
20:28:26:WU02:FS00:0x11:edr file hash check passed.
20:28:26:WU02:FS00:0x11:Logfile not read.
20:28:26:WU02:FS00:0x11:GuardedRun: success in DynamicWrapper
20:28:26:WU02:FS00:0x11:GuardedRun: done
20:28:26:WU02:FS00:0x11:Run: GuardedRun completed.
20:28:29:WU02:FS00:0x11:+ Opened results file
20:28:29:WU02:FS00:0x11:- Writing 132488 bytes of core data to disk...
20:28:29:WU02:FS00:0x11:Done: 131976 -> 130958 (compressed to 99.2 percent)
20:28:29:WU02:FS00:0x11:  ... Done.
20:28:29:WU02:FS00:0x11:DeleteFrameFiles: successfully deleted file=02/wudata_01.ckp
20:28:29:WU02:FS00:0x11:Shutting down core 
20:28:29:WU02:FS00:0x11:
20:28:29:WU02:FS00:0x11:Folding@home Core Shutdown: FINISHED_UNIT
20:28:29:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
20:28:29:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:10502 run:111 clone:0 gen:968 core:0x11 unit:0x000007e36652eda54b6f3a03000032f4
20:28:29:WU02:FS00:Uploading 128.39KiB to 171.67.108.21
20:28:29:WU00:FS00:Starting
20:28:29:WU02:FS00:Connecting to 171.67.108.21:8080
20:28:29:WU00:FS00:Running FahCore: "C:\Program Files\FAHClient/FAHCoreWrapper.exe" "C:/Documents and Settings/Alan/Application Data/FAHClient/cores/www.stanford.edu/~pande/Win32/x86/NVIDIA/G80/Core_11.fah/FahCore_11.exe" -dir 00 -suffix 01 -version 702 -lifeline 2212 -checkpoint 15 -gpu 0
20:28:29:WU00:FS00:Started FahCore on PID 9116
20:28:29:WU00:FS00:Core PID:7016
20:28:29:WU00:FS00:FahCore 0x11 started
20:28:30:WU00:FS00:0x11:
20:28:30:WU00:FS00:0x11:*------------------------------*
20:28:30:WU00:FS00:0x11:Folding@Home GPU Core
20:28:30:WU00:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
20:28:30:WU00:FS00:0x11:
20:28:30:WU00:FS00:0x11:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
20:28:30:WU00:FS00:0x11:Build host: amoeba
20:28:30:WU00:FS00:0x11:Board Type: Nvidia
20:28:30:WU00:FS00:0x11:Core      : 
20:28:30:WU00:FS00:0x11:Preparing to commence simulation
20:28:30:WU00:FS00:0x11:- Looking at optimizations...
20:28:30:WU00:FS00:0x11:DeleteFrameFiles: successfully deleted file=00/wudata_01.ckp
20:28:30:WU00:FS00:0x11:- Created dyn
20:28:30:WU00:FS00:0x11:- Files status OK
20:28:30:WU00:FS00:0x11:- Expanded 62826 -> 336799 (decompressed 536.0 percent)
20:28:30:WU00:FS00:0x11:Called DecompressByteArray: compressed_data_size=62826 data_size=336799, decompressed_data_size=336799 diff=0
20:28:30:WU00:FS00:0x11:- Digital signature verified
20:28:30:WU00:FS00:0x11:
20:28:30:WU00:FS00:0x11:Project: 10504 (Run 76, Clone 1, Gen 1241)
20:28:30:WU00:FS00:0x11:
20:28:30:WU00:FS00:0x11:Assembly optimizations on if available.
20:28:30:WU00:FS00:0x11:Entering M.D.
20:28:34:WU02:FS00:Upload complete
20:28:34:WU02:FS00:Server responded WORK_ACK (400)
20:28:34:WU02:FS00:Cleaning up
20:28:36:WU00:FS00:0x11:Tpr hash 00/wudata_01.tpr:  2497685400 3989437958 953309500 2151444769 2051623843
20:28:36:WU00:FS00:0x11:
20:28:36:WU00:FS00:0x11:Calling fah_main args: 14 usage=100
20:28:36:WU00:FS00:0x11:
20:28:36:WU00:FS00:0x11:Working on Protein
20:28:37:WU00:FS00:0x11:Client config unavailable.
20:28:37:WU00:FS00:0x11:Starting GUI Server
20:31:07:WU00:FS00:0x11:Completed 1%
20:33:38:WU00:FS00:0x11:Completed 2%
20:36:08:WU00:FS00:0x11:Completed 3%
20:38:38:WU00:FS00:0x11:Completed 4%
20:41:08:WU00:FS00:0x11:Completed 5%
20:43:39:WU00:FS00:0x11:Completed 6%
20:46:09:WU00:FS00:0x11:Completed 7%
20:48:39:WU00:FS00:0x11:Completed 8%
20:51:09:WU00:FS00:0x11:Completed 9%
20:53:40:WU00:FS00:0x11:Completed 10%
20:56:10:WU00:FS00:0x11:Completed 11%
20:58:40:WU00:FS00:0x11:Completed 12%
21:01:11:WU00:FS00:0x11:Completed 13%
21:03:41:WU00:FS00:0x11:Completed 14%
21:06:11:WU00:FS00:0x11:Completed 15%
21:08:41:WU00:FS00:0x11:Completed 16%
21:11:12:WU00:FS00:0x11:Completed 17%
21:13:42:WU00:FS00:0x11:Completed 18%
21:16:12:WU00:FS00:0x11:Completed 19%
21:18:42:WU00:FS00:0x11:Completed 20%
21:21:12:WU00:FS00:0x11:Completed 21%
21:23:43:WU00:FS00:0x11:Completed 22%
21:26:13:WU00:FS00:0x11:Completed 23%
21:28:43:WU00:FS00:0x11:Completed 24%
21:31:13:WU00:FS00:0x11:Completed 25%
21:33:44:WU00:FS00:0x11:Completed 26%
21:36:14:WU00:FS00:0x11:Completed 27%
21:37:00:WARNING:WU01:FS01:FahCore returned an unknown error code which probably indicates that it crashed
21:37:00:WARNING:WU01:FS01:FahCore returned: UNKNOWN_ENUM (-1073741819 = 0xc0000005)
21:37:00:WU01:FS01:Starting
21:37:00:WU01:FS01:Running FahCore: "C:\Program Files\FAHClient/FAHCoreWrapper.exe" "C:/Documents and Settings/Alan/Application Data/FAHClient/cores/www.stanford.edu/~pande/Win32/x86/Core_a4.fah/FahCore_a4.exe" -dir 01 -suffix 01 -version 702 -lifeline 2212 -checkpoint 15 -np 2
21:37:00:WU01:FS01:Started FahCore on PID 5796
21:37:00:WU01:FS01:Core PID:4344
21:37:00:WU01:FS01:FahCore 0xa4 started
21:37:01:WU01:FS01:0xa4:
21:37:01:WU01:FS01:0xa4:*------------------------------*
21:37:01:WU01:FS01:0xa4:Folding@Home Gromacs GB Core
21:37:01:WU01:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
21:37:01:WU01:FS01:0xa4:
21:37:01:WU01:FS01:0xa4:Preparing to commence simulation
21:37:01:WU01:FS01:0xa4:- Ensuring status. Please wait.
21:37:10:WU01:FS01:0xa4:- Looking at optimizations...
21:37:10:WU01:FS01:0xa4:- Working with standard loops on this execution.
21:37:10:WU01:FS01:0xa4:- Previous termination of core was improper.
21:37:10:WU01:FS01:0xa4:- Files status OK
21:37:10:WU01:FS01:0xa4:- Expanded 16680 -> 55804 (decompressed 334.5 percent)
21:37:10:WU01:FS01:0xa4:Called DecompressByteArray: compressed_data_size=16680 data_size=55804, decompressed_data_size=55804 diff=0
21:37:10:WU01:FS01:0xa4:- Digital signature verified
21:37:10:WU01:FS01:0xa4:
21:37:10:WU01:FS01:0xa4:Project: 6338 (Run 25, Clone 14, Gen 7)
21:37:10:WU01:FS01:0xa4:
21:37:10:WU01:FS01:0xa4:Entering M.D.
21:37:16:WU01:FS01:0xa4:Mapping NT from 2 to 2 
21:37:16:WU01:FS01:0xa4:Completed 0 out of 50000000 steps  (0%)
21:38:45:WU00:FS00:0x11:Completed 28%
21:41:15:WU00:FS00:0x11:Completed 29%
21:41:36:WU01:FS01:0xa4:Completed 500000 out of 50000000 steps  (1%)
21:43:46:WU00:FS00:0x11:Completed 30%
21:45:56:WU01:FS01:0xa4:Completed 1000000 out of 50000000 steps  (2%)
21:46:17:WU00:FS00:0x11:Completed 31%
21:48:51:WU00:FS00:0x11:Completed 32%
21:50:22:WU01:FS01:0xa4:Completed 1500000 out of 50000000 steps  (3%)
21:51:27:WU00:FS00:0x11:Completed 33%
21:54:03:WU00:FS00:0x11:Completed 34%

*********************** Log Started 2013-01-15T09:07:29Z ***********************
09:07:29:************************* Folding@home Client *************************
09:07:29: Website: http://folding.stanford.edu/
09:07:29: Copyright: (c) 2009-2012 Stanford University
09:07:29: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
09:07:29: Args: --lifeline 3840 --command-port=36330
09:07:29: Config: C:/Documents and Settings/Alan/Application
09:07:29: Data/FAHClient/config.xml
09:07:29:******************************** Build ********************************
09:07:29: Version: 7.2.9
09:07:29: Date: Oct 3 2012
09:07:29: Time: 18:05:48
09:07:29: SVN Rev: 3578
09:07:29: Branch: fah/trunk/client
09:07:29: Compiler: Intel(R) C++ MSVC 1500 mode 1200
09:07:29: Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
09:07:29: /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
09:07:29: Platform: win32 XP
09:07:29: Bits: 32
09:07:29: Mode: Release
09:07:29:******************************* System ********************************
09:07:29: CPU: Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
09:07:29: CPU ID: GenuineIntel Family 6 Model 23 Stepping 6
09:07:29: CPUs: 2
09:07:29: Memory: 3.24GiB
09:07:29: Free Memory: 2.53GiB
09:07:29: Threads: WINDOWS_THREADS
09:07:29: On Battery: false
09:07:29: UTC offset: -6
09:07:29: PID: 2212
09:07:29: CWD: C:/Documents and Settings/Alan/Application Data/FAHClient
09:07:29: OS: Microsoft Windows XP Service Pack 3
09:07:29: OS Arch: X86
09:07:29: GPUs: 1
09:07:29: GPU 0: NVIDIA:1 G94 [GeForce 9600 GT]
09:07:29: CUDA: Not detected
09:07:29:Win32 Service: false
09:07:29:***********************************************************************

Napoleon · Post by **Napoleon** » Tue Jan 22, 2013 7:01 am

vvoelz wrote:I'm releasing project 6345 for beta testing. This is a clone of p6338 (just new runs at multiple temperatures) so the same points, benchmarks, everything should apply.

As for P6338,

vvoelz wrote:FYI -- I've released this to fah. Thanks for all the beta testing! Hopefully you'll start to see 6339-41 next (still no assigns for these! see other thread) --Vince

and back to P6345:

vvoelz wrote:Thanks everyone for the detailed bug reports. We too are getting errors including the FahCore_a4.exe crashing and getting the v7 client "stuck" into a "Updating..." mode. A similar project still in internal testing (6346) has the same problems. So, I've pulled this project back to internal testing so we can figure out what the problem might be. --Vince

I'll post a note about this in the P6338 beta testing thread... done and done, see new SMP Project 6338 released for beta testing.

art_l_j_PlanetAMD64 · Post by **art_l_j_PlanetAMD64** » Tue Jan 22, 2013 7:17 am

This is the key part of the log:

Code: Select all

21:37:00:WARNING:WU01:FS01:FahCore returned an unknown error code which probably indicates that it crashed
21:37:00:WARNING:WU01:FS01:FahCore returned: UNKNOWN_ENUM (-1073741819 = 0xc0000005)

As P5-133XL said here:

P5-133XL wrote:0xc0000005 is a Windows error implying that an application is trying to access memory that it is not entitled to. It is commonly a sign that you have some bad RAM or your memory controller is having an issue.

Napoleon · Post by **Napoleon** » Tue Jan 22, 2013 7:22 am

I am assuming the troublesome WU got restarted immediately after the (A4) core got stuck and then crashed for the first time:

21:37:00:WU01:FS01:Started FahCore on PID 5796
21:37:00:WU01:FS01:Core PID:4344
21:37:00:WU01:FS01:FahCore 0xa4 started
21:37:01:WU01:FS01:0xa4:
21:37:01:WU01:FS01:0xa4:*------------------------------*
21:37:01:WU01:FS01:0xa4:Folding@Home Gromacs GB Core
21:37:01:WU01:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
21:37:01:WU01:FS01:0xa4:
21:37:01:WU01:FS01:0xa4:Preparing to commence simulation
21:37:01:WU01:FS01:0xa4:- Ensuring status. Please wait.
21:37:10:WU01:FS01:0xa4:- Looking at optimizations...
21:37:10:WU01:FS01:0xa4:- Working with standard loops on this execution.
21:37:10:WU01:FS01:0xa4:- Previous termination of core was improper.
21:37:10:WU01:FS01:0xa4:- Files status OK
21:37:10:WU01:FS01:0xa4:- Expanded 16680 -> 55804 (decompressed 334.5 percent)
21:37:10:WU01:FS01:0xa4:Called DecompressByteArray: compressed_data_size=16680 data_size=55804, decompressed_data_size=55804 diff=0
21:37:10:WU01:FS01:0xa4:- Digital signature verified
21:37:10:WU01:FS01:0xa4:
21:37:10:WU01:FS01:0xa4:Project: 6338 (Run 25, Clone 14, Gen 7)

art_l_j_PlanetAMD64 · Post by **art_l_j_PlanetAMD64** » Tue Jan 22, 2013 7:31 am

OK, gotcha now, but the 0xc0000005 error code still means this:

P5-133XL wrote:0xc0000005 is a Windows error implying that an application is trying to access memory that it is not entitled to. It is commonly a sign that you have some bad RAM or your memory controller is having an issue.

Napoleon · Post by **Napoleon** » Tue Jan 22, 2013 7:36 am

If I had to guess, the A4 core got stuck in some loop which leaked memory, and eventually... 0xc0000005. Bad memory (controller) is one thing that could cause 0xc0000005, but there are others too. Could be an actual bug in FahCore_a4 (we already know it isn't perfect)... or just plaing old GIGO thing. So I linked back to the P6338 beta thread just in case it is the latter. Considering the "choice quotes" in my first response, I'd say I've got the "probable cause" thingy covered. Napoleon proposing armistice...

art_l_j_PlanetAMD64 · Post by **art_l_j_PlanetAMD64** » Tue Jan 22, 2013 8:04 am

Agreed, I edited my comments to agree with yours.

Alan C. Lawhon · Post by **Alan C. Lawhon** » Tue Jan 22, 2013 8:35 am

Art, Napoleon:

The closest I ever got to being a real honest-to-god "computer scientist" was database programming (which probably doesn't count), so what you guys are discussing is a little over my head. I just hope the problem is not with my machine's RAM or the [disk?] controller or anything to do with my hardware/software setup. My machine has been running like a champ for nearly six years, so I hope it's not going south on me.

The best I can tell from what you guys have indicated is that this is more likely a problem with the FAH a4 "core" software, although it will be interesting to see what the final verdict is from Dr. Voelz and his team. (Maybe I've stumbled across an actual "bug" - which makes me an official "debugger" - Ha! Ha!) If it turns out to be a problem with my box, I suppose I'll get more of these type errors in the future. (Boo on that!)

I want my baby to run for at least one full year crunching work units so I can chalk up over 1.5 million points and over 3,000 completed WUs!

P.S. I owe a tip of the hat to Jesse_V without whose advice and assistance I would have had no idea on how to go about posting this. Thanks Jesse!

Napoleon · Post by **Napoleon** » Tue Jan 22, 2013 9:28 am

art_l_j_PlanetAMD64 wrote:OK, gotcha now, but the 0xc0000005 error code still means this:
P5-133XL wrote:0xc0000005 is a Windows error implying that an application is trying to access memory that it is not entitled to. It is commonly a sign that you have some bad RAM or your memory controller is having an issue.

Agreed, as far as FAH is concerned. But I did provide a very crude example of a programming error causing a 0xC0000005, merely to state my case that "commonly" isn't quite the same thing as "always".
viewtopic.php?f=58&t=22431#p234492

IIRC, Alan is running his setup at stock. Furthermore, the project/WU which caused the problem is a bit suspicious, in my book at least. So I'm tentatively suggesting that we've ran into a very rare case of "uncommon" where a project got released to full FAH a bit hastily. I daresay there's no need for Alan to start MemTesting... just yet.

art_l_j_PlanetAMD64 · Tue Jan 22, 2013 11:20 am

Alan C. Lawhon wrote:The best I can tell from what you guys have indicated is that this is more likely a problem with the FAH a4 "core" software, although it will be interesting to see what the final verdict is from Dr. Voelz and his team.

Yes, that's correct, there have been a bunch of other reports of bugs/errors traced to this series of WUs (P6338).

Napoleon wrote:Agreed, as far as FAH is concerned. But I did provide a very crude example of a programming error causing a 0xC0000005, merely to state my case that "commonly" isn't quite the same thing as "always".

Yes, I agree. This type of fault (memory access error) can also be caused by using an uninitialized or NULL pointer,

Dereferencing the NULL pointer typically results in an attempted read or write from memory that is not mapped - triggering segmentation fault or access violation. This may represent itself to the developer as a program crash, or be transformed into an exception that can be caught.

or somehow accessing a block of memory after it has been free()'d.

The malloc statement will ask for an amount of memory with the size of an integer (32 bits or 4 bytes). If there is not enough memory available, the malloc function will return a NULL.
If you have allocated a memory block with the functions malloc(), calloc() or realloc() then you need to free() the previously allocated memory.

Napoleon wrote:IIRC, Alan is running his setup at stock. Furthermore, the project/WU which caused the problem is a bit suspicious, in my book at least. So I'm tentatively suggesting that we've ran into a very rare case of "uncommon" where a project got released to full FAH a bit hastily. I daresay there's no need for Alan to start MemTesting... just yet.

I agree 100%, please see my comment to Alan above.

7im · Post by **7im** » Tue Jan 22, 2013 1:33 pm

It wouldn't hurt to run memtest or a similar program to eliminate the possibility of a memory problem.

art_l_j_PlanetAMD64 · Post by **art_l_j_PlanetAMD64** » Tue Jan 22, 2013 2:55 pm

7im wrote:It wouldn't hurt to run memtest or a similar program to eliminate the possibility of a memory problem.

Alan, what he means is this:
Memtest86 - Download Page

Post by **Joe_H** » Sat Jan 26, 2013 5:31 pm

Just as a followup to this thread, the second run of the WU after the core crash did complete successfully:

Hi Lawhonac (team 0),
Your WU (P6338 R25 C14 G7) was added to the stats database on 2013-01-21 21:08:57 for 527.14 points of credit.

So there may have been a single error that caused the core to fault the previous attempt.

Folding Forum

Core Crash?

Core Crash?

Re: Core Crash?

Re: Core Crash?

Re: Core Crash?

Re: Core Crash?

Re: Core Crash?

Re: Core Crash?

Re: Core Crash?

Re: Core Crash?

Re: Core Crash?

Re: Core Crash?

Re: Core Crash?

Re: Core Crash?