Project:6892 run:788 clone:0 gen:101

[Napoleon]

Code: Select all

12:13:23:WU01:FS03:Running FahCore: C:\FAH\Client/FAHCoreWrapper.exe C:/FAH/Data/cores/www.stanford.edu/~pande/Win32/AMD64/beta/Core_78.fah/FahCore_78.exe -dir 01 -suffix 01 -version 701 -lifeline 72632 -checkpoint 30 -verbose
12:13:23:WU01:FS03:Started FahCore on PID 59864
12:13:23:WU01:FS03:Core PID:60776
12:13:23:WU01:FS03:FahCore 0x78 started
12:13:24:WU01:FS03:0x78:
12:13:24:WU01:FS03:0x78:*------------------------------*
12:13:24:WU01:FS03:0x78:Folding@Home Gromacs Core
12:13:24:WU01:FS03:0x78:Version 1.90 (March 8, 2006)
12:13:24:WU01:FS03:0x78:
12:13:24:WU01:FS03:0x78:Preparing to commence simulation
12:13:24:WU01:FS03:0x78:- Looking at optimizations...
12:13:24:WU01:FS03:0x78:- Created dyn
12:13:24:WU01:FS03:0x78:- Files status OK
12:13:24:WU01:FS03:0x78:- Expanded 662374 -> 3332352 (decompressed 503.0 percent)
12:13:24:WU01:FS03:0x78:- Starting from initial work packet
12:13:24:WU01:FS03:0x78:
12:13:24:WU01:FS03:0x78:Project: 6892 (Run 788, Clone 0, Gen 101)
12:13:24:WU01:FS03:0x78:
12:13:24:WU01:FS03:0x78:Assembly optimizations on if available.
12:13:24:WU01:FS03:0x78:Entering M.D.
12:13:31:WU01:FS03:0x78:Protein: ALZHEIMER DISEASE AMYLOID
12:13:31:WU01:FS03:0x78:
12:13:31:WU01:FS03:0x78:Writing local files
12:16:35:WU01:FS03:0x78:Extra SSE boost OK.
12:16:35:WU01:FS03:0x78:Writing local files
12:16:36:WU01:FS03:0x78:Completed 0 out of 250000 steps  (0%)
12:44:57:WU01:FS03:0x78:Writing local files
12:44:57:WU01:FS03:0x78:Completed 2500 out of 250000 steps  (1%)
13:13:20:WU01:FS03:0x78:Writing local files
13:13:20:WU01:FS03:0x78:Completed 5000 out of 250000 steps  (2%)
13:41:41:WU01:FS03:0x78:Writing local files
13:41:41:WU01:FS03:0x78:Completed 7500 out of 250000 steps  (3%)
14:10:01:WU01:FS03:0x78:Writing local files
14:10:01:WU01:FS03:0x78:Completed 10000 out of 250000 steps  (4%)
14:38:24:WU01:FS03:0x78:Writing local files
14:38:24:WU01:FS03:0x78:Completed 12500 out of 250000 steps  (5%)
15:06:48:WU01:FS03:0x78:Writing local files
15:06:49:WU01:FS03:0x78:Completed 15000 out of 250000 steps  (6%)
15:35:12:WU01:FS03:0x78:Writing local files
15:35:12:WU01:FS03:0x78:Completed 17500 out of 250000 steps  (7%)
16:03:37:WU01:FS03:0x78:Writing local files
16:03:37:WU01:FS03:0x78:Completed 20000 out of 250000 steps  (8%)
******************************** Date: 30/08/12 ********************************
16:32:00:WU01:FS03:0x78:Writing local files
16:32:00:WU01:FS03:0x78:Completed 22500 out of 250000 steps  (9%)
17:00:24:WU01:FS03:0x78:Writing local files
17:00:24:WU01:FS03:0x78:Completed 25000 out of 250000 steps  (10%)
17:28:51:WU01:FS03:0x78:Writing local files
17:28:51:WU01:FS03:0x78:Completed 27500 out of 250000 steps  (11%)
17:57:14:WU01:FS03:0x78:Writing local files
17:57:15:WU01:FS03:0x78:Completed 30000 out of 250000 steps  (12%)
18:25:42:WU01:FS03:0x78:Writing local files
18:25:42:WU01:FS03:0x78:Completed 32500 out of 250000 steps  (13%)
18:54:05:WU01:FS03:0x78:Writing local files
18:54:05:WU01:FS03:0x78:Completed 35000 out of 250000 steps  (14%)
19:22:31:WU01:FS03:0x78:Writing local files
19:22:31:WU01:FS03:0x78:Completed 37500 out of 250000 steps  (15%)
19:50:56:WU01:FS03:0x78:Writing local files
19:50:56:WU01:FS03:0x78:Completed 40000 out of 250000 steps  (16%)
20:19:21:WU01:FS03:0x78:Writing local files
20:19:21:WU01:FS03:0x78:Completed 42500 out of 250000 steps  (17%)
20:47:44:WU01:FS03:0x78:Writing local files
20:47:44:WU01:FS03:0x78:Completed 45000 out of 250000 steps  (18%)
21:16:07:WU01:FS03:0x78:Writing local files
21:16:07:WU01:FS03:0x78:Completed 47500 out of 250000 steps  (19%)
21:44:32:WU01:FS03:0x78:Writing local files
21:44:32:WU01:FS03:0x78:Completed 50000 out of 250000 steps  (20%)
******************************** Date: 30/08/12 ********************************
22:12:57:WU01:FS03:0x78:Writing local files
22:12:57:WU01:FS03:0x78:Completed 52500 out of 250000 steps  (21%)
22:41:25:WU01:FS03:0x78:Writing local files
22:41:25:WU01:FS03:0x78:Completed 55000 out of 250000 steps  (22%)
23:09:54:WU01:FS03:0x78:Writing local files
23:09:55:WU01:FS03:0x78:Completed 57500 out of 250000 steps  (23%)
23:38:25:WU01:FS03:0x78:Writing local files
23:38:25:WU01:FS03:0x78:Completed 60000 out of 250000 steps  (24%)
00:07:06:WU01:FS03:0x78:Writing local files
00:07:06:WU01:FS03:0x78:Completed 62500 out of 250000 steps  (25%)
00:35:36:WU01:FS03:0x78:Writing local files
00:35:36:WU01:FS03:0x78:Completed 65000 out of 250000 steps  (26%)
01:04:11:WU01:FS03:0x78:Writing local files
01:04:11:WU01:FS03:0x78:Completed 67500 out of 250000 steps  (27%)
01:32:34:WU01:FS03:0x78:Writing local files
01:32:34:WU01:FS03:0x78:Completed 70000 out of 250000 steps  (28%)
02:02:33:WU01:FS03:0x78:Timered checkpoint triggered.
02:03:58:WU01:FS03:0x78:Writing local files
02:03:58:WU01:FS03:0x78:Completed 72500 out of 250000 steps  (29%)
02:33:56:WU01:FS03:0x78:Writing local files
02:33:56:WU01:FS03:0x78:Completed 75000 out of 250000 steps  (30%)
03:03:56:WU01:FS03:0x78:Timered checkpoint triggered.
03:06:22:WU01:FS03:0x78:Gromacs cannot continue further.
03:06:22:WU01:FS03:0x78:Going to send back what have done.
03:06:22:WU01:FS03:0x78:logfile size: 0
03:06:22:WU01:FS03:0x78:Warning: Core could not open logfile.
03:06:22:WU01:FS03:0x78:- Writing 536 bytes of core data to disk...
03:06:22:WU01:FS03:0x78:Done: 24 -> 69 (compressed to 287.5 percent)
03:06:22:WU01:FS03:0x78:  ... Done.
03:06:22:WU01:FS03:0x78:
03:06:22:WU01:FS03:0x78:Folding@home Core Shutdown: EARLY_UNIT_END
03:06:24:WU01:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:06:24:WU01:FS03:Sending unit results: id:01 state:SEND error:FAULTY project:6892 run:788 clone:0 gen:101 core:0x78 unit:0x000000856652edc54e25d65222d2cc45
03:06:24:WU01:FS03:Uploading 581B to 171.67.108.53
03:06:24:WU01:FS03:Connecting to 171.67.108.53:8080
03:06:24:WU00:FS03:Connecting to assign3.stanford.edu:8080
03:06:25:WU01:FS03:Upload complete
03:06:25:WU01:FS03:Server responded WORK_ACK (400)
03:06:25:WU01:FS03:Cleaning up

[bollix47]

Another user was able to complete this work unit:

Hi xxxx (team xxxx),
Your WU (P6892 R788 C0 G101) was added to the stats database on 2012-08-31 13:04:51 for 136 points of credit.

Since it is not a bad WU, even though the log says it is, I'm changing the title.

[Napoleon]

This one line in the log left me a bit puzzled:

Code: Select all

03:06:22:WU01:FS03:0x78:Warning: Core could not open logfile.

The permissions are correct, and FAH data directory is excluded from Avast. I had HFM running at the time, though, but I quit it because of this. Is there any chance that 3rd party apps such as HFM could interfere with the client on some extremely rare occasions? I presume it's a long shot, but then again, various antivirus apps have been known to interfere with the client's files on some rare occasions. How about other apps?

I'm following up on this because I'm a bit skeptical when it comes to calling my rig (or OC) marginally stable. As far as temps go, the rig runs just fine even if I stress test it beyond FAH levels. Furthermore, I only reboot when SW updates demand it - current uptime is 18 days and counting. If my setup were a borderline case, I'd expect problems to creep in sooner or later somewhere else as well, resulting in a BSOD or some other clear indicator of HW instability. The setup isn't purpose-built for folding, I run other apps on it as well, although under normal circumstances FAH is by far the toughest for the HW.

[bollix47]

Code: Select all

03:06:22:WU01:FS03:0x78:logfile size: 0

I believe that's referring to the logfile in work/01.

Since you had already completed 30% the logfile should have had data in it.

I don't see how Avast could have caused the problem if the Data Directory has been excluded from it's checking. Is there a log or something else in Avast that you can check to see if it did anything to the logfile in work/01? Personally, I use Microsoft Security Essentials and have never seen a problem.

I wouldn't think there's anything HFM could do that would interfere with the client. AFAIK all the client files that HFM needs would be accessed as read-only. I've been running HFM 24/7 for years and have never had a problem like the one you've described.

[bruce]

The FahCore would need to open a log file for Writing and/or Appending. When you open a file for Read/Only, if it's already open for Write, it gets open as Shared and HFM would know it might be changed by another application. If you open it for R/O first, you probably can't open it correctly for Write. (It has been a long time since I used this information and I didn't look it up to confirm the old memories. Let's just say "It's possible that HFM caused the error.")

[Napoleon]

According to timestamps, it seems more or less likely that the core was trying to write a checkpoint at 31% (about 2min 26s after the timered checkpoint) and something went wrong with it. It was quite a while back on different HW IIRC, but there was a similar mysterious failure with fahcore_78. Back then, it was suspected that the timered checkpoint occurred at the same time as the regular checkpoint and they clashed. That WU was also completed succesfully by another donor, I think.

I looked up my earlier "bad WU" reports for this setup and P6892 to see if there's any historical pattern:

1. 14 Dec 2011 (marked bad 04 Mar 2012 as there is still nothing in the database )
2. 27 Dec 2011 (other donor completed successfully)
3. 23 Feb 2012 (bad WU)
4. 09 Mar 2012 (bad WU)

09-Mar-2012 case looks eeriely similar to this current one. But looking at timestamps, it doesn't look like a checkpoint issue. I didn't use HFM back then, for obvious reasons. Not so sure if I had remembered to add FAH data directory to Avast exclusion list, though. And that one was a bad WU, after all. Go figure.

I suppose this track record isn't too bad considering that #1, #3 and #4 were deemed actual bad WUs and possibly #2 was my bad, due to a less-than-optimal OC and a couple of other things. But this mysterious "completed successfully by other donor" WU gives me The Willies anyway.

OK, I'm more relaxed about folding with less-than-mature GPGPUs - ION1 in particular - but with the CPU it's got to be stable or else...

[bollix47]

What is your checkpoint setting? Are you using the default of 15 minutes or have you entered something different?

[Napoleon]

I've set it to the max 30min because I keep this setup (only the GT430 is fully dedicated to FAH) running 24/7. Like the log with timestamps in my original post indicates:

Code: Select all

01:32:34:WU01:FS03:0x78:Completed 70000 out of 250000 steps  (28%)
02:02:33:WU01:FS03:0x78:Timered checkpoint triggered.
02:03:58:WU01:FS03:0x78:Writing local files
02:03:58:WU01:FS03:0x78:Completed 72500 out of 250000 steps  (29%)
02:33:56:WU01:FS03:0x78:Writing local files
02:33:56:WU01:FS03:0x78:Completed 75000 out of 250000 steps  (30%)
03:03:56:WU01:FS03:0x78:Timered checkpoint triggered.
03:06:22:WU01:FS03:0x78:Gromacs cannot continue further.

[bollix47]

Back then, it was suspected that the timered checkpoint occurred at the same time as the regular checkpoint and they clashed.

From that extract it looks like "Writing local files" and "Timered checkpoint triggered" should have occurred at the same time. You could try changing your checkpoint to something lower like 25-29 but it's such a rare occurrence you might have to wait a long time to determine if it helps.

[bruce]

I've seen a message reporting that a checkpoint was already in progress and reporting the fact that one checkpoint was skipped, so the code is supposed to handle this. As bollix47 said it's hard to test since it happens very, very rarely, no matter what checkpoint interval you select. It's also complicated by the differences between various FahCore's

Anyway, I though this problem didn't exist, but FahCore_78 sees very limited use.

[Napoleon]

If I had been smart, I would have checked old logs before ranting quite so confidently about stability.
Simple enough find "error:FAULTY" *.txt shows quite a few CPU folding failures:

Code: Select all

---------- LOG-20120609-084938.TXT
project:6896 run:868 clone:1 gen:49 core:0x78 unit:0x000000426652edc44e28a7ce78a85bf4

---------- LOG-20120709-175110.TXT
project:6891 run:231 clone:5 gen:69 core:0x78 unit:0x0000004f6652edc44e2456824d4f258c
project:6897 run:1055 clone:0 gen:57 core:0x78 unit:0x0000004b6652edc44e2f0813777a1ccd
project:6897 run:377 clone:0 gen:101 core:0x78 unit:0x0000006f6652edc44e2efabc4c175683

---------- LOG-20120804-143047.TXT
project:6897 run:714 clone:0 gen:79 core:0x78 unit:0x000000576652edc44e2f015f83203a8b

---------- LOG-20120826-153727.TXT
project:6892 run:551 clone:5 gen:157 core:0x78 unit:0x000000ae6652edc54e25d1d2cd3d17cb

---------- LOG-20120906-091638.TXT
project:6892 run:788 clone:0 gen:101 core:0x78 unit:0x000000856652edc54e25d65222d2cc45
project:6892 run:330 clone:14 gen:225 core:0x78 unit:0x000000ed6652edc54e25cd9a063e8879

It'd be really great if a mod could check up on these WUs and let me know just how many of them have been completed successfully by someone else. Looking at this list, I'm less confident about the stability of my CPU OC. Having some trouble with FAHWatch, but it listed only 94% success rate for CPU WUs. Sounds pretty bad for full FAH classic WUs, eh?

[bollix47]

Hi xxxxxx (team xx),
Your WU (P6896 R868 C1 G49) was added to the stats database on 2012-06-01 14:02:23 for 135 oints of credit.


project:6891 run:231 clone:5 gen:69 - 6 results - no credits

project:6897 run:1055 clone:0 gen:57 - 7 results - no credits

project:6897 run:377 clone:0 gen:101 - 6 results - no credits


Hi xxxxx (team xxxxx),
Your WU (P6897 R714 C0 G79) was added to the stats database on 2012-07-27 08:02:07 for 135 points of credit.

Hi xxxx (team x),
Your WU (P6892 R551 C5 G157) was added to the stats database on 2012-08-19 05:03:33 for 136 points of credit.

Hi xxx (team x),
Your WU (P6892 R788 C0 G101) was added to the stats database on 2012-08-31 13:04:51 for 136 points of credit.

project:6892 run:330 clone:14 gen:225 - 3 results - no credits

I did mark the last one as bad but the others with no credit have not been assigned in over a month so they may have already been reported. I will investigate them further.

The WU (P6892,R330,C14,G225) has been reported as a bad WU. Note that the list of reported WUs are stopped daily at 8am pacific time.

[Napoleon]

Thanks, that less than stellar track record convinced me to rethink my OC, and I reduced the CPU clock considerably (2055MHz to 1710MHz). I put it in place mainly because of some ancient GPU3 WUs anyway. The 8000+ PPD glory days of GT430 are long gone and CPU utilization is really small with the newer NVidia GPU3 WUs, so actually I have no real reason to keep an extreme CPU OC anymore. Pretty much down to stock clocks now. Still trying to optimize memory speed (within spec) because of the integrated ION GPU, hence the ~7% FSB OC. I will be monitoring my folding more closely from now on, just in case I'm pushing the memory subsystem too hard:

• CPU FSB-overclocked from 1600MHz (12x133MHz) to 1710MHz (12x142.5MHz). Core voltage increased to 1.15V, CPU spec limits are 0.900V - 1.1625V
• 2x2GB Kingston KVR1333D3N9K2/4G memory kit @1368MHz. Memory voltage increased to 1.57V, memory spec limits are 1.425V - 1.575V. BIOS autoconfigure applies 1370MHz (2x685MHz) JEDEC #4 timings from SPD, so nominally the memories are ever so slightly underclocked (0.15% ). FSB:DRAM == 5:24, so 142.5MHz x 24 / 5 == 684MHz 2x684MHz == 1368MHz effective memory clock

EDIT: I'm folding with all 4 CPU threads now, so I get to keep the 30min setting and the timered vs regular checkpoints typically are further apart. I'd say it compensates the reduced OC nicely, and it seems all the CPU WUs I'm getting are for fahcore_78 anyway. Considering its "limited use", I'm guessing things have changed so that it's better to optimize for throughput instead of returning individual WUs ASAP.

[P5-133XL]

Fahcore_78's are uniprocessor, not multicore. Do you have 5 slots configured (4x UNI + GPU)? You might find it better to run GPU+SMP.

[Napoleon]

Like my updated sig states "1x GPU3, 1x GPU2, 4x uniprocessor slots (too slow for SMP)". With a 1.6GHZ ==> 2.1GHz OC it looked like SMP-only would've been barely possible, but the SMP WUs I received ran very close to the preferred deadline. Just adding one GPU would've delayed SMP WUs too much. Atom330 is really weak CPU but low wattage too. Back when I built this thingy, I just wanted to enjoy a completely fanless setup for a while. Then I got bored, added a quiet 120mm case fan and a (passively cooled) GT430 to do some "serious" folding.
I wouldn't mind checking how the A4 uniprocessor WUs would fare on this setup, but I can't recall ever getting even one. Looks like it's core78 ad nauseaum for me.