85% GPU Usage

[{RaW}Eagle1]

Hi,

Firstly, let me thank you for all the effort that you guys are putting in to keep folding at home alive and kicking. Its really good to be a part of something this spectacular. However I have a problem. (Not so much a problem as a minor inconvenience).

My setup:
Windows 7 Home Premium 64 bit SP1
AMD Phenom 1090T o/clocked 3850 MHz
8GB o/clocked G.Skill RAM @ 1920 MHz (9/9/9/24)
ASUS ROG Crosshair V AMD 990FX/SB950 mainboard

• - I am running GPU folding on two NVidia GTX480s in SLI and have set them up correctly. (different machine IDs etc).
  - Both cards are underclocked slightly and have higher settings for fan speed. I did this using MSI Afterburner (650MHz from 700MHz) due to power draw and heat. (Maximum load on both cards at factory clock is enough to cause the power protection on my 1200W PSU to trip out).
  - I am also running the SMP client, folding on 3 cores of the CPU. (Keeps the temperatures down and enables me to use the PC whilst folding).
  - I have recently updated the graphics drivers to NVidia's 296.1 driver having had problems with the 295.x driver not picking up new work units when the monitor went into "sleep" mode. I have also disabled the "sleep" mode for the monitor due to this problem.
  - All overclocks have been extensivley stability tested using prime95, memtest etc.

New problem:
Both GPUs now seem to be on a new project (7643) and are only running at 86% load in their underclocked state. I can't seem to narrow down the reason for this as I have changed too many variables recently to ascertain the source of the issue.

What I've done that may have caused it, or my best guesses as to the cause:

• 1. Updated the graphics driver?
  2. New project doesn't use as much load? (Unlikely I'd have thought).
  3. Changed the properties of the shortcuts to disable visual themes? (Thought this might give a slight performance boost but I'll change it back if it's the problem).
  4. Set the graphics to SLI performance mode? (Never made a difference before. . . Does it now?)
  5. I have also just noticed that a newer version of fahcore_15.exe was aquired before undertaking these new projects as the previous on I had was "out of date". Is this a problem with the design of the new core or possibly a measure taken to ensure stability during testing of the new core?

Any assistance would be gratefuly recieved. If none is forthcoming I guess I'll restore the cards to their factory clock settings and just leave the folding running at a reduced load state. This hasn't made much impact to the scores I am recieving from the clients as the new projects are worth more. . . I guess I'm just a stickler for performance and I'd love to be contributing as much as possible.

Thanks all.

{RaW}Eagle1 of The {RaW}Gaming team (208841)

[tjlane]

Hi RawEagle,

I believe that this might be an unfortunate an common issue/feature of the p764x series going forward. The limitations are likely not software related, but hardware imposed. Specifically, these projects need to do a lot more file I/O than previous GPU projects, and the PCI bus that connects most GPUs to their motherboards are terribly slow. Since the calculations on the GPU are so fast, the PCI bus can't always keep up, and I suspect that this might be a limiting factor for these projects.

That said, the GPU performance still beats CPUs for these guys, so even though you're running at 85%, there is a significant speedup from the science end. So thanks for folding!

If you figure out a tweak to get to 100% though let us know . I'm not optimistic, however...

Best,

TJ

[7im]

763x and 764x projects use an updated fahcore, so that is the cause to the effect.

[k1wi]

Specifically, these projects need to do a lot more file I/O than previous GPU projects, and the PCI bus that connects most GPUs to their motherboards are terribly slow. Since the calculations on the GPU are so fast, the PCI bus can't always keep up, and I suspect that this might be a limiting factor for these projects.

Does that mean PCIe 3 could improve utilisation?

When you say is it I/O, is that I/O to disk drives (in which case, would a RAM drive improve things by decreasing latency and increasing throughput?)

[bruce]

Welcome to foldingforum.org, "{RaW}Eagle1"

{quote]- I have recently updated the graphics drivers to NVidia's 296.1 driver having had problems with the 295.x driver not picking up new work units when the monitor went into "sleep" mode. I have also disabled the "sleep" mode for the monitor due to this problem.[/quote] This seems to be a consistent problem ... and you've found the only known solution. There's a strong suspicion that it's only true for the 64-bit drivers.

- All overclocks have been extensively stability tested using prime95, memtest etc.

memtest is always a good idea, but prime95 is inadequate when it comes to testing stability of FAH's heavy use of SSE on your CPUs. I personally recommend StressCPU2 because it's based on the actual GROMACS code that FAH uses so it matches FAH's pattern of use more closely than any other test, but others recommend OCCT or IntelBurn Test.

If you overclock your GPU, you might also consider MemtestG80 for CUDA-based GPUs or MemtestCL for any device that supports OpenCL.

[{RaW}Eagle1]

Hi RawEagle,

I believe that this might be an unfortunate an common issue/feature of the p764x series going forward. The limitations are likely not software related, but hardware imposed. Specifically, these projects need to do a lot more file I/O than previous GPU projects, and the PCI bus that connects most GPUs to their motherboards are terribly slow. Since the calculations on the GPU are so fast, the PCI bus can't always keep up, and I suspect that this might be a limiting factor for these projects.

That said, the GPU performance still beats CPUs for these guys, so even though you're running at 85%, there is a significant speedup from the science end. So thanks for folding!

If you figure out a tweak to get to 100% though let us know . I'm not optimistic, however...

Best,

TJ

Thanks for the quick response.

I believe I am in a position to confirm that it is not the PCI bus speed that is the problem on my system for a number of reasons:

• - The mainboard I am using is PCIe 2 with dual 16x slots (not 8x/8x).
  - I have overclocked my graphics to take up the headroom due to lost load and have seen proportional time per frame decrease. If there was a bottleneck due to PCI bus speed, I wouldn't expect to see a substantial decrease from having done so.
  - If I run folding at home on one card I get the same 86% load as I do if I run folding on both. In theory; if there was a problem with the overall bus speed one card should surely manage to run at 100% load?

I'm going to try rolling back the NVidia driver. . . Its about the only thing left on my list that I haven't tried yet. I'll post back here with the results.

{RaW}Eagle1

PS. Thanks also to Bruce for the suggestions on stability testing. I have already run these with the exception of OCCT which I have added to my library of testing, overclocking and benchmark applications. I'm sure my overclocks are air-tight and have not recieved any "unstable machine" errors.

[tjlane]

Hmm my guess was wrong then. Let us know what you find out, RaWEagle.

[{RaW}Eagle1]

I apologise for having discounted the possibility of PCI bus speed so quickly. . . It is now the only reason I can find for the reduction in percentage load on the GPUs.

I don't believe the PCIe bandwidth to be the problem, but rather; the speed of the communication or latency there of. That being the case, a simple overclock is sufficient to reclaim lost time per frame and therefore points production; as the increase in data transference isn't the issue, but rather by the speed at which new data is swapped out of the graphics memory.

The clinching evidence of this will be to hear from someone else folding project 764x on an NVidia card with the same or less total graphics memory in a PCIe 1 or PCIe 2 slot (8x or 16x), who is having similar issues. . .Otherwise I must surmise that it's a local problem with my client or PC and not a wider issue relating to the hardware or software that I am using. If I don't hear from anyone else having similar issues then I will proceed to test every possible factor, now having ruled out all the variables in my original post that I am in control of.

I have run out of time to do more thorough testing before the weekend, as I am away. I will begin again in earnest when I return on Monday.

Thanks again.

{RaW}Eagle1

PS. Just to be clear; I have now ruled out all of the factors in my initial post other than the following:

• 2. New project doesn't or can't use as much load.
  5. Newer version of fahcore_15.exe doesn't or can't use as much load.

[{RaW}Eagle1]

Update
Having searched the web, I have come across a thread in the EVGA folding forums in which several other people are relating to the same issues i am having with this project. I must therefore conclude that the limiting factor is indeed PCI bus speed or latency. I have gone through all the release notes from previous NVidia drivers and rolled back to a previous driver to no avail. . . Bring on PCIe 3!

Another problem exists, all be it from a research point of view: The solution found on EVGA's forum for this problem is to remove the -advmethods flag from the GPU client, thus preventing project 764x from being downloaded. I believe many will take this option due to the drop in "points per day" from undertaking one of these projects over any other available work units and due to either not having the available overhead on their GPUs to overclock, or not having the stomach for it. The only options I can see, from Stanford's point of view would be either to distribute project 764x to those without the -advmethods flag specified in their client configuration (which would seem rather unfair), or to review the points awarded to completed simulations of project 764x in favour of their completion.

I will continue to fold project 764x in the spirit of charity and also due to the fact that I have the headroom for overclocking, thus picking up the productivity lost due to reduced load. In return, please let me know when project 764x is coming to the end of it's availability so I can reset my graphics clock settings. I currently have them set too high to run under 100% load due to power draw. . . I don't want to come home from holiday to find a black screen and many lost folds (and points), or at worst; the pungent aroma of burned microchips coming from my PC!

Thanks

{RaW}Eagle1

PS. I would give you the link to the EVGA folding forum thread on this issue but your anti-spam filter has prevented me from doing so. . . as did theirs when I tried to post a link to this thread. . .Anti-spam filters everywhere are joining force to create an anti-social dis-networking pact. . . lol.

[7im]

As those projects leave beta testing status, they will eventually be assigned to all clients, not just the clients with -advmethods. problem 1 solved.

They'll review the feedback on points, and re-evaluate if a change is needed or not. problem 2 solved.

[{RaW}Eagle1]

Please do not mistake my brevity as dispassion or condescension. I recognize the time you spend reading the forum is time you could use elsewhere, so my short responses save you time. Please do not hesitate to ask for clarification if I was too terse.

Colour me slighted by the lack of effort on your part in replying to this thread, in which I have invested a lot of time, however I feel I must ask for the clarification your signature offers as to your motive in being so condescending and abrupt? Please don't attempt to disguise your obnoxious demeanor by suggesting it is any effort on your part to "save the time" of the reader. . . If I spend a lot of time on something I expect those who feel they can offer something worthy of comment to take similar time in consideration, reflection and recourse.

Your lack of attention to the detail of this thread is evident in your answer to a thread started recently by "Mactin" on the same topic, in which you quoted PCIe bandwidth to be the issue. Had you taken the time to digest the information I painstakingly accrued and subsequently divulged, you would have noticed that latency is the issue, rather than bandwidth and would not have mis-advised our friend.

Your signature seems little more than an apology for your posts. Your comment in this thread is of low quality content and adds very little of intelectual value. Your post doesn't come off as helpful, intelligent or endearing and infact comes off as narcissistic and egotistical.

Eagle1

[7im]

Everyone is entitled to their own initial impressions and beliefs, right or wrong.

I believe I am in a position to confirm that it is not the PCI bus speed that is the problem on my system...

I apologise for having discounted the possibility of PCI bus speed so quickly. . .

To clarifiy, in many years of posting in the folding forum, I've found that long drawn out molly-coddling answers come off as condescending more often than short answers. And shorter answers allow me to help more people while offending the sensibilities of fewer people. I follow the better odds... nothing personal.

[bruce]

Let's ASSUME that when benchmarked on a GTX 460, one warp (unit of GPU work) takes one unit of time to process and almost exactly the same amount of time to swap data through a 16x bus. The 460 will be 100% utilized because the next warp arrives just as the previous warp is completed.

Now let's assume we substitute a faster GPU. It will still take the same amount of time to transfer the data so the faster GPU will finish the warp sooner but the next one will not arrive any sooner so the GPU utilization will be lower.

Now if the words "almost exactly" are replaced by either "somewhat faster" or "somewhat slower" the percent utilization of the faster GPU will change to other values but in all cases, the folding speed will be limited by whichever is longer, the time to transfer the warp or the time to process the warp, and the utilization will be whatever it will be.

To test this theory, replace the 16x slot with an 8x slot. Transferring the data will essentially be half as fast. GPU utilization might be as little as half what it was with 16x or might be the same (if the GPU manages to overlap all I/O with processing).

As you've already concluded: Bring on PCIe 3! which will help in some cases and not in others.

[Off topic] I wonder how much of the 'screen lag' problem comes from PCIe saturation and how much comes from GPU shader saturation.

[{RaW}Eagle1]

Thanks once again for your time Bruce.

I follow the explanaition given, however my experience differs:

. . . Now let's assume we substitute a faster GPU. It will still take the same amount of time to transfer the data so the faster GPU will finish the warp sooner but the next one will not arrive any sooner so the GPU utilization will be lower. . .

I had originally underclocked my SLI GTX480s due to power draw under load. (700MHz to 650MHz). At this point I saw 86% utilization of each card. I then overclocked to 730MHz to pick up the lost utilization from project 764x. I saw a proportional decrease in time per frame (and increase in points production), but saw no change in the load, which remained constant at 86%. In your post you suggested that you would expect to see a utilization decrease? This leads me to believe the bandwidth is not the issue, but rather the latency. . . or something else that may be at play?

I have an enterprise edition of Windows somewhere and an older mobo with 8x PCIe slots. If I can build a test rig then I'll fire it up and see what happens. . . If any of the 764x projects are still available?

Cheers

Eagle1

[k1wi]

You should be able to adjust the PCI-e slots on your current box and limit them to 8x. You might also be able to adjust the frequency of it (lower, as going higher easily increases instability).