P9625-9643 and bad states- some observations

[billford]

With my experience (GTX 970) the linux driver version 346.96 for folding is the fastest and most stable version (Linux mint mate 17.2).

I'd go along with that- 2x GTX 980's, same OS, a few % faster in terms of TPF, no noticeable change in BS frequency.

FWIW I'm only having problems with WUs from a few projects:

P9629/31/36/38: yet to have one run to completion (too many retries)

P9630/33/34/37: will sometimes run to completion, sometimes won't

All other Core_21's have run to completion (sometimes with one or two BS's), and no failures (or BS's afaict) on a GTX 780 Ti.

So far

(One reservation about the failures- I can only find those by checking the client logs, and for various reasons (unconnected with folding) I've had an abnormally high number of system restarts over the last few weeks so not as many log files as I might like.)

[billford]

As I said in my OP, there appears to be a periodic/cumulative effect involved. From a WU that has just failed:

Code: Select all

09:26:38:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9634 run:1 clone:25 gen:31 core:0x21 unit:0x00000029ab436c9b5609bee39c568421
09:26:38:WU01:FS01:Starting
09:26:38:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1351 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
09:26:38:WU01:FS01:Started FahCore on PID 17515
09:26:38:WU01:FS01:Core PID:17519
09:26:38:WU01:FS01:FahCore 0x21 started
09:26:38:WU01:FS01:0x21:*********************** Log Started 2015-11-04T09:26:38Z ***********************
09:26:38:WU01:FS01:0x21:Project: 9634 (Run 1, Clone 25, Gen 31)
09:26:38:WU01:FS01:0x21:Unit: 0x00000029ab436c9b5609bee39c568421
09:26:38:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
09:26:38:WU01:FS01:0x21:Machine: 1
09:26:38:WU01:FS01:0x21:Reading tar file core.xml
09:26:38:WU01:FS01:0x21:Reading tar file integrator.xml
09:26:38:WU01:FS01:0x21:Reading tar file state.xml
09:26:38:WU01:FS01:0x21:Reading tar file system.xml
09:26:38:WU01:FS01:0x21:Digital signatures verified
09:26:38:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
09:26:38:WU01:FS01:0x21:Version 0.0.12
09:27:01:WU01:FS01:0x21:Completed 0 out of 2000000 steps (0%)
09:27:01:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
09:28:36:WU01:FS01:0x21:Completed 20000 out of 2000000 steps (1%)
09:30:06:WU01:FS01:0x21:Completed 40000 out of 2000000 steps (2%)
09:31:36:WU01:FS01:0x21:Completed 60000 out of 2000000 steps (3%)
09:33:06:WU01:FS01:0x21:Completed 80000 out of 2000000 steps (4%)
09:34:36:WU01:FS01:0x21:Completed 100000 out of 2000000 steps (5%)
09:36:12:WU01:FS01:0x21:Completed 120000 out of 2000000 steps (6%)
09:37:42:WU01:FS01:0x21:Completed 140000 out of 2000000 steps (7%)
09:39:12:WU01:FS01:0x21:Completed 160000 out of 2000000 steps (8%)
09:40:42:WU01:FS01:0x21:Completed 180000 out of 2000000 steps (9%)
09:42:12:WU01:FS01:0x21:Completed 200000 out of 2000000 steps (10%)
09:42:17:WU01:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
09:43:47:WU01:FS01:0x21:Completed 120000 out of 2000000 steps (6%)
09:45:17:WU01:FS01:0x21:Completed 140000 out of 2000000 steps (7%)
09:46:47:WU01:FS01:0x21:Completed 160000 out of 2000000 steps (8%)
09:48:17:WU01:FS01:0x21:Completed 180000 out of 2000000 steps (9%)
09:49:47:WU01:FS01:0x21:Completed 200000 out of 2000000 steps (10%)
09:51:23:WU01:FS01:0x21:Completed 220000 out of 2000000 steps (11%)
09:52:53:WU01:FS01:0x21:Completed 240000 out of 2000000 steps (12%)
09:54:23:WU01:FS01:0x21:Completed 260000 out of 2000000 steps (13%)
09:55:53:WU01:FS01:0x21:Completed 280000 out of 2000000 steps (14%)
09:57:23:WU01:FS01:0x21:Completed 300000 out of 2000000 steps (15%)
09:57:28:WU01:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
09:58:58:WU01:FS01:0x21:Completed 220000 out of 2000000 steps (11%)
10:00:28:WU01:FS01:0x21:Completed 240000 out of 2000000 steps (12%)
10:01:58:WU01:FS01:0x21:Completed 260000 out of 2000000 steps (13%)
10:03:28:WU01:FS01:0x21:Completed 280000 out of 2000000 steps (14%)
10:04:58:WU01:FS01:0x21:Completed 300000 out of 2000000 steps (15%)
10:06:34:WU01:FS01:0x21:Completed 320000 out of 2000000 steps (16%)
10:08:03:WU01:FS01:0x21:Completed 340000 out of 2000000 steps (17%)
0:09:33:WU01:FS01:0x21:Completed 360000 out of 2000000 steps (18%)
10:11:03:WU01:FS01:0x21:Completed 380000 out of 2000000 steps (19%)
10:12:33:WU01:FS01:0x21:Completed 400000 out of 2000000 steps (20%)
10:12:37:WU01:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint
10:12:37:WU01:FS01:0x21:Max number of retries reached. Aborting.
10:12:37:WU01:FS01:0x21:ERROR:Max Retries Reached
10:12:37:WU01:FS01:0x21:Saving result file logfile_01.txt
10:12:37:WU01:FS01:0x21:Saving result file log.txt
10:12:37:WU01:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
\x1b[93m10:12:38:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)\x1b[0m
10:12:38:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9634 run:1 clone:25 gen:31 core:0x21 unit:0x00000029ab436c9b5609bee39c568421
10:12:38:WU01:FS01:Uploading 10.50KiB to 171.67.108.155
10:12:38:WU01:FS01:Connecting to 171.67.108.155:8080
10:12:38:WU01:FS01:Upload complete
10:12:38:WU01:FS01:Server responded WORK_ACK (400)
10:12:38:WU01:FS01:Cleaning up

This is not untypical of all the BS's I see with Core_21- if the first BS is later than ~40% it will probably complete, less than ~20% it won't. In between it's a maybe.

Does it mean anything? No idea, but if the client was a bit more adaptive to BS's I'm pretty sure it would eventually complete.

Easy to implement- when the client writes a checkpoint, decrement (or clear) the BS counter. Worst case (assuming <3 BS's between checkpoints) it would take 2-3 times as long as it should.

[Grandpa_01]

Did you change clock for the right P state ? 7010 MHz (1752 MHz real) is the clock for P0, but most cards are folding in P2 state, which default clock is 6000 MHz (1500 MHz real).

Yes, I created a modified bios so that p2 runs at 7010mhz same as p0. I then modified that bios so that p2 runs at 5500mhz to test Grandpa_01 theory in linux.

Just curious as to how you measured the running memory speed after you created the bios ?

[bigblock990]

Did you change clock for the right P state ? 7010 MHz (1752 MHz real) is the clock for P0, but most cards are folding in P2 state, which default clock is 6000 MHz (1500 MHz real).

Yes, I created a modified bios so that p2 runs at 7010mhz same as p0. I then modified that bios so that p2 runs at 5500mhz to test Grandpa_01 theory in linux.

Just curious as to how you measured the running memory speed after you created the bios ?

In xserver settings GUI if you click on "powermizer" tab for each gpu it shows current core and mem clocks right at the top.

[Grandpa_01]

And it showed something other than 6008Mhz available for P2 state which in x server is Performance level 2 ?

That is odd because from my understanding that is set by the drivers and there is only 1 option for P2 state in any of the drivers that I know of for the Maxwell's and that is 6008Mhz , I will see if I can get one of the NVidia developers if it is possible to get the P2 memory speed to change by changing the bios. I have a feeling it may be limited by the driver options but there is a good chance I misunderstood what I was told before.

[mattifolder]

Version 0.0.14 of Core_21 is running with client-type=beta, problems have gone. I copies the core (under linux) to standard core path, switched client to advanced and no bad states at the time with moderate OC (some pieces lower than core_18).

[bruce]

Version 0.0.14 of Core_21 is running with client-type=beta, problems have gone. I copies the core (under linux) to standard core path, switched client to advanced and no bad states at the time with moderate OC (some pieces lower than core_18).

Good to hear.

Overclocking might cause the same error if there's insufficient margin to assure stability under worst-case conditions, but a good Factory Overclock tends to leave enough margin.

[mattifolder]

Overclocking might cause the same error if there's insufficient margin to assure stability under worst-case conditions, but a good Factory Overclock tends to leave enough margin.

The extra over factory OC for my GTX 970 ist about 170 to 180 MHz, that's not too much. I've a script watching folding logs by inotifyd, which decreases the value at every "Bat State" for actually project and overwrites my script configuration for the following projects. The script has an configuration table with values for every project type matching by numbers. The initially OC values are stability tested by other applications and the already finished projects. New projects will be added to the configuration manually after watching the script created extra log file.

[bruce]

A good factory overclock is supposed to be able to remain stable under extreme conditions. Maybe "it's not much" seems easy to believe but even a small "I can get away with 170 to 180 MHz" without ever testing it in under extreme conditions might very well push it into marginal stability (i.e.- unstable). This is especially true if you're overclocking VRAM.

The Developers are tracking this issue very carefully. The clincher will be if your system develops internal NaNs which are not reproduced when the identical WU is assigned to another Donor.