Folding Forum

by **billford** » Sun Oct 25, 2015 1:46 pm

I haven't had enough of these for this to be conclusive, but it has happened several times and I haven't seen it with other projects. I put it forward as perhaps suggesting something that a developer may recognise. Or not, as the case may be.

It relates to my GTX 980's, NVidia 355.11 drivers (Linux), factory overclocked by 90MHz, another 125MHz overclock added by me, set to "Prefer Maximum performance". (This last seemed to improve matters a little).

First, only WUs that run the GPU close to maximum power (~190W wrt 196W) seem to cause issues. Those (most) that run at a more usual 160-180W seem OK.

If the WU is going to have issues, the first bad state usually occurs at around 35-40 frames, it backs up to the last checkpoint and continues, throws another bad state after another 35-40 frames, backs up again. At this point it will either complete the WU (occasionally with a 3rd bad state during the sanity check after 100%) or throw the third, fatal, bad state around 95%.

In other words, it doesn't seem to be random. It's as though there's a small, but cumulative, error somewhere which eventually forces it "out of spec", and is reset when a checkpoint file is loaded. (Which could imply that it's not an inherent part of the WU data but related to the fahcore or drivers) Whether it reaches three bad states or not depends how fast it accumulates. (Pure speculation on my part)

This could be supported by the observation that, if one of these WUs does fail due to too many bad states, the next WU (whatever project it's from) is quite likely to throw a bad state within seconds of starting, back up to the previous non-existent checkpoint (ie start again) and then complete without further issue.

Note- I have no intention of resetting the clock to factory values; with the current spread of projects I'm getting the loss of overall PPD (hence science done) that this would entail is about three times the loss due to binning the occasional WU.

by **Grandpa_01** » Mon Oct 26, 2015 4:16 am

I can tell you how to fix this but you probably are not going to be real happy with the solution.

This is not a core21 problem it is a Nvidia problem that affects the upper end Maxwells mainly GTX 980's and some of the 970's, It may also affect 980 TI's but I do not know that since I do not have a 980 TI to test. I have a feeling Nvidia knows about it since they took steps to reduce the problem. When you run compute software on the Maxwells it defaults to P2 state which has the same core clock but has a reduced memory speed it gets lowered from 7000Mhz to 6000Mhz no other generation of Nvidia GPU's does this that I know of.

Anyway I started having a lot of the bad states on my GPU's both in Windows and Linux with this series of WU's and some others and I started testing things out with the help of a few others from here in the forum. And one of the thing's I did was lower the Memory speed using Nvidia Inspector in P2 state a little. The first adjustment was 300Mhz and the errors stopped completely and I was able to achieve a higher stable OC. I ended up between 5750Mhz and 5755Mhz on all 3 of the cards.

The part you are not going to like is that there is no way at this time to lower the P2 state memory speeds in Linux. I have put in a request for help over at the Nvidia Developers forum so hopefully they will enable that ability in x server soon. But as of yet nobody has replied to the post but it is Sunday. Until then I would recommend putting the 980 in a Windows box, which I know is slower but in the long run it will pay off, I have moved 3 of mine from Linux to Windows. Or you can yell at Nvidia for selling us a GPU with what I am thinking is either faulty memory or faulty memory controller, most likely the latter.

https://devtalk.nvidia.com/default/topic/888789/linux/problem-with-memory-speed-on-gtx-9xx-when-running-compute-software/

by **billford** » Mon Oct 26, 2015 7:30 am

Thanks for that, very interesting.

Must admit I was under the impression that the memory clock could be adjusted just like the graphics clock, but closer examination (plus experiment) shows you are perfectly correct

I'm reluctant to run the cards in a Windows box- apart from any other reason I haven't got any spare Windows licences!

(edit- I assume that expecting Nvidia Inspector to work under WINE would be a touch optimistic? Even if I plucked up the courage to try it...)

For the moment I'll leave it as it is, as I indicated I'm only losing an occasional WU. If that changes then I'll have to re-think.

FWIW I've added a reply to your post on the NVidia forum, but from their response to the Maxwell bug some time ago (ie there wasn't any) I'm not going to hold my breath.

Further edit- if NVidia show no signs of fixing it, a workaround might be the suggestion elsewhere to increase the tolerance of the Core_21 code to bad state errors. Maybe in v0.0.13.

by **bigblock990** » Thu Oct 29, 2015 7:29 pm

Grandpa_01 wrote:I can tell you how to fix this but you probably are not going to be real happy with the solution.

This is not a core21 problem it is a Nvidia problem that affects the upper end Maxwells mainly GTX 980's and some of the 970's, It may also affect 980 TI's but I do not know that since I do not have a 980 TI to test. I have a feeling Nvidia knows about it since they took steps to reduce the problem. When you run compute software on the Maxwells it defaults to P2 state which has the same core clock but has a reduced memory speed it gets lowered from 7000Mhz to 6000Mhz no other generation of Nvidia GPU's does this that I know of.

Anyway I started having a lot of the bad states on my GPU's both in Windows and Linux with this series of WU's and some others and I started testing things out with the help of a few others from here in the forum. And one of the thing's I did was lower the Memory speed using Nvidia Inspector in P2 state a little. The first adjustment was 300Mhz and the errors stopped completely and I was able to achieve a higher stable OC. I ended up between 5750Mhz and 5755Mhz on all 3 of the cards.

The part you are not going to like is that there is no way at this time to lower the P2 state memory speeds in Linux. I have put in a request for help over at the Nvidia Developers forum so hopefully they will enable that ability in x server soon. But as of yet nobody has replied to the post but it is Sunday. Until then I would recommend putting the 980 in a Windows box, which I know is slower but in the long run it will pay off, I have moved 3 of mine from Linux to Windows. Or you can yell at Nvidia for selling us a GPU with what I am thinking is either faulty memory or faulty memory controller, most likely the latter.

https://devtalk.nvidia.com/default/topic/888789/linux/problem-with-memory-speed-on-gtx-9xx-when-running-compute-software/

You really might be on to something here. I have titan x, 980ti, and two 970's. I have modded the 970 bios to run at default 7010mhz for p2. The titan x and 980 ti both run at 6608mhz p2. The titan x is the worse offender, with 980ti next, then 970's when it comes to bad state errors for core21 units. More memory on titan x means memory controller works harder? I will edit my bios down to 5750mhz for the 970's and if that eliminates the errors, I will make a mod bios for the other two.

Any idea why core21 causes problems, but core18 runs perfectly fine?

by **Grandpa_01** » Thu Oct 29, 2015 8:01 pm

Just a wild guess but I would say core 21 is more efficient and pushes the memory / memory controller a little harder than previous cores. But that really should not be a problem if Nvidias hardware was up to Industry standards, No other generation of Nvidia graphics have a problem with core 21 WU's. In addition to my 980's and 970 I am running 1 - GTX 770, 2 - GTX 680's and a GTX 580 all of which do not have any problem with core 21 projects on Windows or Linux.

by **mattifolder** » Thu Oct 29, 2015 11:42 pm

It seems to be a way to dynamically adjust gpu memory clock under linux for folding. It works for me with nvidia driver 346.96. I think, it's more a bug than a feature, but it is reproducible.
Short description for my setup with nvidia gtx970, all commands have to execute under grafic desktop in terminal with root user (or sudo).
This is initially situation:

Code: Select all: nvidia-settings -q [gpu:0]/GPUCurrentPerfLevel -q [gpu:0]/GPUCurrentClockFreqs Attribute 'GPUCurrentPerfLevel' (mslinuxmint:0[gpu:0]): 2. 'GPUCurrentPerfLevel' is an integer attribute. 'GPUCurrentPerfLevel' is a read-only attribute. 'GPUCurrentPerfLevel' can use the following target types: X Screen, GPU. Attribute 'GPUCurrentClockFreqs' (mslinuxmint:0[gpu:0]): 1514,3004. 'GPUCurrentClockFreqs' is a packed integer attribute. 'GPUCurrentClockFreqs' is a read-only attribute. 'GPUCurrentClockFreqs' can use the following target types: X Screen, GPU. nvidia-smi --query-gpu=pstate --format=csv pstate P2

Next steps to prepare for changing memory clock for folding are:
- get actual application clocks with nvidia-smi

Code: Select all: nvidia-smi --query-gpu=clocks.applications.graphics --format=csv clocks.applications.graphics [MHz] 1113 MHz nvidia-smi --query-gpu=clocks.applications.memory --format=csv clocks.applications.memory [MHz] 3505 MHz

- get supported application clocks with nvidia-smi

Code: Select all: nvidia-smi -q -d SUPPORTED_CLOCKS ==============NVSMI LOG============== Timestamp : Thu Oct 29 23:12:38 2015 Driver Version : 346.96 Attached GPUs : 1 GPU 0000:01:00.0 Supported Clocks Memory : 3505 MHz Graphics : 1641 MHz Graphics : 1628 MHz Graphics : 1616 MHz Graphics : 1603 MHz Graphics : 1590 MHz Graphics : 1578 MHz Graphics : 1565 MHz Graphics : 1552 MHz Graphics : 1540 MHz Graphics : 1527 MHz Graphics : 1514 MHz Graphics : 1502 MHz Graphics : 1489 MHz Graphics : 1476 MHz Graphics : 1464 MHz Graphics : 1451 MHz Graphics : 1438 MHz Graphics : 1426 MHz Graphics : 1413 MHz Graphics : 1400 MHz Graphics : 1388 MHz Graphics : 1375 MHz Graphics : 1362 MHz Graphics : 1350 MHz Graphics : 1337 MHz Graphics : 1324 MHz Graphics : 1312 MHz Graphics : 1299 MHz Graphics : 1286 MHz Graphics : 1274 MHz Graphics : 1261 MHz Graphics : 1249 MHz Graphics : 1236 MHz Graphics : 1223 MHz Graphics : 1211 MHz Graphics : 1198 MHz Graphics : 1185 MHz Graphics : 1173 MHz Graphics : 1160 MHz Graphics : 1147 MHz Graphics : 1135 MHz Graphics : 1122 MHz Graphics : 1109 MHz Graphics : 1097 MHz Graphics : 1085 MHz Graphics : 1072 MHz Graphics : 1071 MHz Graphics : 1059 MHz Graphics : 1046 MHz Graphics : 1033 MHz Graphics : 1021 MHz Graphics : 1008 MHz Graphics : 995 MHz Graphics : 983 MHz Graphics : 970 MHz Graphics : 957 MHz Graphics : 945 MHz Graphics : 932 MHz Graphics : 919 MHz Graphics : 907 MHz Graphics : 894 MHz Graphics : 881 MHz Graphics : 869 MHz Graphics : 856 MHz Graphics : 844 MHz Graphics : 831 MHz Graphics : 818 MHz Graphics : 806 MHz Graphics : 793 MHz Graphics : 780 MHz Graphics : 768 MHz Graphics : 755 MHz Graphics : 742 MHz Graphics : 730 MHz Graphics : 717 MHz Graphics : 704 MHz Graphics : 692 MHz Graphics : 680 MHz Graphics : 667 MHz Graphics : 655 MHz Graphics : 642 MHz Graphics : 630 MHz Graphics : 617 MHz Graphics : 605 MHz Graphics : 592 MHz Graphics : 590 MHz Graphics : 509 MHz Graphics : 455 MHz Graphics : 388 MHz Graphics : 349 MHz Graphics : 320 MHz Memory : 3004 MHz Graphics : 1641 MHz Graphics : 1628 MHz Graphics : 1616 MHz Graphics : 1603 MHz Graphics : 1590 MHz Graphics : 1578 MHz Graphics : 1565 MHz Graphics : 1552 MHz Graphics : 1540 MHz Graphics : 1527 MHz Graphics : 1514 MHz Graphics : 1502 MHz Graphics : 1489 MHz Graphics : 1476 MHz Graphics : 1464 MHz Graphics : 1451 MHz Graphics : 1438 MHz Graphics : 1426 MHz Graphics : 1413 MHz Graphics : 1400 MHz Graphics : 1388 MHz Graphics : 1375 MHz Graphics : 1362 MHz Graphics : 1350 MHz Graphics : 1337 MHz Graphics : 1324 MHz Graphics : 1312 MHz Graphics : 1299 MHz Graphics : 1286 MHz Graphics : 1274 MHz Graphics : 1261 MHz Graphics : 1249 MHz Graphics : 1236 MHz Graphics : 1223 MHz Graphics : 1211 MHz Graphics : 1198 MHz Graphics : 1185 MHz Graphics : 1173 MHz Graphics : 1160 MHz Graphics : 1147 MHz Graphics : 1135 MHz Graphics : 1122 MHz Graphics : 1109 MHz Graphics : 1097 MHz Graphics : 1085 MHz Graphics : 1072 MHz Graphics : 1071 MHz Graphics : 1059 MHz Graphics : 1046 MHz Graphics : 1033 MHz Graphics : 1021 MHz Graphics : 1008 MHz Graphics : 995 MHz Graphics : 983 MHz Graphics : 970 MHz Graphics : 957 MHz Graphics : 945 MHz Graphics : 932 MHz Graphics : 919 MHz Graphics : 907 MHz Graphics : 894 MHz Graphics : 881 MHz Graphics : 869 MHz Graphics : 856 MHz Graphics : 844 MHz Graphics : 831 MHz Graphics : 818 MHz Graphics : 806 MHz Graphics : 793 MHz Graphics : 780 MHz Graphics : 768 MHz Graphics : 755 MHz Graphics : 742 MHz Graphics : 730 MHz Graphics : 717 MHz Graphics : 704 MHz Graphics : 692 MHz Graphics : 680 MHz Graphics : 667 MHz Graphics : 655 MHz Graphics : 642 MHz Graphics : 630 MHz Graphics : 617 MHz Graphics : 605 MHz Graphics : 592 MHz Graphics : 590 MHz Graphics : 509 MHz Graphics : 455 MHz Graphics : 388 MHz Graphics : 349 MHz Graphics : 320 MHz Memory : 810 MHz Graphics : 1455 MHz Graphics : 1442 MHz Graphics : 1430 MHz Graphics : 1417 MHz Graphics : 1404 MHz Graphics : 1392 MHz Graphics : 1379 MHz Graphics : 1366 MHz Graphics : 1354 MHz Graphics : 1341 MHz Graphics : 1328 MHz Graphics : 1316 MHz Graphics : 1303 MHz Graphics : 1290 MHz Graphics : 1278 MHz Graphics : 1265 MHz Graphics : 1252 MHz Graphics : 1240 MHz Graphics : 1227 MHz Graphics : 1215 MHz Graphics : 1202 MHz Graphics : 1189 MHz Graphics : 1177 MHz Graphics : 1164 MHz Graphics : 1151 MHz Graphics : 1139 MHz Graphics : 1126 MHz Graphics : 1113 MHz Graphics : 1101 MHz Graphics : 1088 MHz Graphics : 1075 MHz Graphics : 1063 MHz Graphics : 1050 MHz Graphics : 1037 MHz Graphics : 1025 MHz Graphics : 1012 MHz Graphics : 999 MHz Graphics : 987 MHz Graphics : 974 MHz Graphics : 961 MHz Graphics : 949 MHz Graphics : 936 MHz Graphics : 923 MHz Graphics : 911 MHz Graphics : 899 MHz Graphics : 886 MHz Graphics : 885 MHz Graphics : 873 MHz Graphics : 860 MHz Graphics : 847 MHz Graphics : 835 MHz Graphics : 822 MHz Graphics : 810 MHz Graphics : 797 MHz Graphics : 784 MHz Graphics : 772 MHz Graphics : 759 MHz Graphics : 746 MHz Graphics : 734 MHz Graphics : 721 MHz Graphics : 708 MHz Graphics : 696 MHz Graphics : 683 MHz Graphics : 670 MHz Graphics : 658 MHz Graphics : 645 MHz Graphics : 632 MHz Graphics : 620 MHz Graphics : 607 MHz Graphics : 594 MHz Graphics : 582 MHz Graphics : 569 MHz Graphics : 556 MHz Graphics : 544 MHz Graphics : 531 MHz Graphics : 519 MHz Graphics : 506 MHz Graphics : 494 MHz Graphics : 481 MHz Graphics : 469 MHz Graphics : 456 MHz Graphics : 444 MHz Graphics : 431 MHz Graphics : 419 MHz Graphics : 406 MHz Graphics : 405 MHz Graphics : 324 MHz Graphics : 270 MHz Graphics : 202 MHz Graphics : 162 MHz Graphics : 135 MHz Memory : 324 MHz Graphics : 405 MHz Graphics : 324 MHz Graphics : 270 MHz Graphics : 202 MHz Graphics : 162 MHz Graphics : 135 MHz

- select one combination from supported application clocks near actual clocks (without gpu-boost)
(I selected Memory 3004 (from P2 State) and Graphics 1109)
- change actual application clocks (it's secondary for our intention) and incidentally switch to full grafic performance state
(in nvidia-smi it names P0, in nvidia-settings P3)

Code: Select all: nvidia-smi -ac 3004,1109 Applications clocks set to "(MEM 3004, SM 1109)" for GPU 0000:01:00.0 Warning: persistence mode is disabled on this device. This settings will go back to default as soon as driver unloads (e.g. last application like nvidia-smi or cuda application terminates). Run with [--help | -h] switch to get more information on how to enable persistence mode. All done.

That are the actually settings:

Code: Select all: nvidia-settings -q [gpu:0]/GPUCurrentPerfLevel -q [gpu:0]/GPUCurrentClockFreqs Attribute 'GPUCurrentPerfLevel' (mslinuxmint:0[gpu:0]): 3. 'GPUCurrentPerfLevel' is an integer attribute. 'GPUCurrentPerfLevel' is a read-only attribute. 'GPUCurrentPerfLevel' can use the following target types: X Screen, GPU. Attribute 'GPUCurrentClockFreqs' (mslinuxmint:0[gpu:0]): 1514,3505. 'GPUCurrentClockFreqs' is a packed integer attribute. 'GPUCurrentClockFreqs' is a read-only attribute. 'GPUCurrentClockFreqs' can use the following target types: X Screen, GPU. nvidia-smi --query-gpu=pstate --format=csv pstate P0

The graphics clock is the same as before, memory clock and performance state are changed to full graphics values.
Because performance state now changed to full graphics (3 / P0) after this steps the change of memory clocks is possible on the well known way by nvidia-settings.

Code: Select all: nvidia-settings --assign [gpu:0]/GPUMemoryTransferRateOffset[3]=...

Resetting application clocks and performance level is done by "nvidia-smi -rac" and all values are as before.

Code: Select all: nvidia-smi -rac All done. nvidia-settings -q [gpu:0]/GPUCurrentPerfLevel -q [gpu:0]/GPUCurrentClockFreqs Attribute 'GPUCurrentPerfLevel' (mslinuxmint:0[gpu:0]): 2. 'GPUCurrentPerfLevel' is an integer attribute. 'GPUCurrentPerfLevel' is a read-only attribute. 'GPUCurrentPerfLevel' can use the following target types: X Screen, GPU. Attribute 'GPUCurrentClockFreqs' (mslinuxmint:0[gpu:0]): 1514,3004. 'GPUCurrentClockFreqs' is a packed integer attribute. 'GPUCurrentClockFreqs' is a read-only attribute. 'GPUCurrentClockFreqs' can use the following target types: X Screen, GPU. nvidia-smi --query-gpu=clocks.applications.graphics --format=csv clocks.applications.graphics [MHz] 1113 MHz nvidia-smi --query-gpu=clocks.applications.memory --format=csv clocks.applications.memory [MHz] 3505 MHz nvidia-smi --query-gpu=pstate --format=csv pstate P2

EDIT:
There is one marginal problem: the minimum vram clock offset at P3 (P0) for "nvidia-settings --assign [gpu:0]/GPUMemoryTransferRateOffset[3]=..." because of higher base vram clock of P3 results in the same base clock as P2.

So nothing is won. I'll have a lock to other performance options of nvidia-settings and nvidia-smi, may be another solution. The Windows tools may increase / degrease the limits, why not under linux ?

by **Grandpa_01** » Sat Oct 31, 2015 4:12 am

Yeah we need NVidia to add some more memoryclock speed options in their drivers it would be a simple fix if they would.

by **bigblock990** » Mon Nov 02, 2015 2:36 pm

Changed my custom bios from 7010mhz to 5500mhz mem clocks for my 970's. The frequency of bad state errors, and failed WU's remained the same for openmm_21 projects. Also dropped PPD by ~50k on those projects. Core18 still work great with same PPD.

So dropping mem clocks may help in windows, but it doesn't in Linux. I'll be going back to 7010mhz

by **toTOW** » Mon Nov 02, 2015 5:01 pm

Did you change clock for the right P state ? 7010 MHz (1752 MHz real) is the clock for P0, but most cards are folding in P2 state, which default clock is 6000 MHz (1500 MHz real).

by **bigblock990** » Mon Nov 02, 2015 5:22 pm

toTOW wrote:Did you change clock for the right P state ? 7010 MHz (1752 MHz real) is the clock for P0, but most cards are folding in P2 state, which default clock is 6000 MHz (1500 MHz real).

Yes, I created a modified bios so that p2 runs at 7010mhz same as p0. I then modified that bios so that p2 runs at 5500mhz to test Grandpa_01 theory in linux.

by **toTOW** » Mon Nov 02, 2015 7:17 pm

So maybe Linux core is able to push something harder than Windows one ?

Or we're completely mistaken, and the root cause still has to be found ...

by **bigblock990** » Mon Nov 02, 2015 8:30 pm

Im not sure what the problem is. I don't have any issues with core18, and core21 9704/9712 work just fine.

I only have problems with core21 9205/06 and 9625-9643.

I have tried returning cards to stock (factory OC), and also to reference stock clocks and still have bad state errors and failed units. Now I have tried low mem clocks which also didn't help. I would say the problem lies within the fahcore, except that grandpa_01 claims his kepler cards work fine, and its only maxwell that has issues.

by **jimerickson** » Tue Nov 03, 2015 12:02 am

9205 & 9206 are problematic on titan X. very few run properly. between slowdowns and bad states and hanging at 99.99% few if any run to completion without intervention. i have to babysit my titan X's and reboot several times a day. its getting old. this is on linux with 355.11 driver.

by **bigblock990** » Tue Nov 03, 2015 8:50 pm

Completed 9205 today with ZERO bad state errors, first time thats ever happened for me. EVGA 980 classy KPE, which I just got folding yesterday evening. The KPE gpu's come with samsung memory, whereas pretty much all others come with either hynix or elpida. I have two of them folding, I will keep an eye on them over the next couple days to see if other core21 projects complete without issues.

by **mattifolder** » Wed Nov 04, 2015 7:57 am

jimerickson wrote:this is on linux with 355.11 driver.

With my experience (GTX 970) the linux driver version 346.96 for folding is the fastest and most stable version (Linux mint mate 17.2). With reduced OC the 96[234]x-er projects also produce bad states, but not all are faulty.

Folding Forum

P9625-9643 and bad states- some observations

P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Re: P9625-9643 and bad states- some observations

Who is online