Benchmarking of GPU3 - Prof Pande?

If you think it might be a driver problem, see viewforum.php?f=79

Moderators: slegrand, Site Moderators, PandeGroup

Benchmarking of GPU3 - Prof Pande?

Postby Mitsimonsta » Tue May 25, 2010 4:32 am

Well, this one comes out every time there is a new client, so I might as well get it out of the way early. Since Professor Pande is watching, Sir may I ask you to inform us of the benchmarking that is applied to GPU3 units?

At the moment, on my GTX275's, there seems to be a 30% loss in points running GPU3 against GPU2. I have received two P10626's and these give 610 points credit, with a frame time of 1:21 gives 6588ppd. The same cards on P10504 do 9392ppd (884 points credit, 0:54 per frame).

We all know that benchmarking is a major issue amongst donators. In this case it appears to me that the units are more complex, have a larger upload footprint (approx 3.2mb compared to <1MB) and have a longer frame time, yet the credit applied is lower than comparable GPU2 units. Based on the very consistent benchmarking on nVidia GPU2 units of late, I would have expected a credit of approximately 800-850 points for the P10504 units.

This is not a complaint, I would merely like to know how the unit credits are worked out. However I would make the point that I doubt that I will keep any of my GPU3 clients once I am satisfied that I have no real issues with it. To take a 30% hit in PPD is a bit unpalatable, especially when donators are being asked to run what is possibly an unstable client. SMP clients were given a bonus in the early days as it was inherently unstable - I would be interested to know if any 'beta' bonus has been incorporated into these unit benchmarks.

Over to you Professor.
Image
Mitsimonsta
 
Posts: 54
Joined: Tue Feb 12, 2008 1:53 am

Re: Benchmarking of GPU3 - Prof Pande?

Postby ArVee » Tue May 25, 2010 5:06 am

At the other end of the spectrum I'm seeing the equivalent percentage loss with 8800gt's running GPU3's. Unless they're realigned I'll also be reverting to GPU2's due to this after the test phase, which I think seems counter-productive, the points alignment I mean.
ArVee
 
Posts: 209
Joined: Sun Dec 02, 2007 9:25 am

Re: Benchmarking of GPU3 - Prof Pande?

Postby Evil Penguin » Tue May 25, 2010 5:16 am

It's just that, a test phase. Please give Pande group some time to work out the kinks and any input is appreciated.
User avatar
Evil Penguin
 
Posts: 380
Joined: Sun Apr 13, 2008 4:34 am
Location: Texas, United States

Re: Benchmarking of GPU3 - Prof Pande?

Postby Mitsimonsta » Tue May 25, 2010 6:14 am

SolidSteel144 wrote:It's just that, a test phase. Please give Pande group some time to work out the kinks and any input is appreciated.

Absolutely. I certainly did not accuse them of being tight with points, just laid out my evidence and asked what the benchmarking method was. I am more than happy to listen to the official explanation.

Of course there will be some bugs, both with the software, the cores, and the projects. My point is that unless the output at least matches GPU2, then there is no real incentive to run it and find those bugs. It runs on my GTX275's fine, and I am about to switch back to GPU2 as it is costing me 3000ppd per card.

They may just be trying to finish the original closed beta units, then the new projects will be benchmarked properly. If that is the case, that is fine.
Last edited by Mitsimonsta on Tue May 25, 2010 6:27 am, edited 1 time in total.
Mitsimonsta
 
Posts: 54
Joined: Tue Feb 12, 2008 1:53 am

Re: Benchmarking of GPU3 - Prof Pande?

Postby 7im » Tue May 25, 2010 6:26 am

Why would the benchmarking change for GPU3? It would appear to be the same as always, according to the new FAQ they put out in that announcement. See also: GPU3 FAQ: How do you decide the credit value of GPU work units?

There is no beta bonus, clearly, considering the PPD is a bit lower on these work units.

Also, it's been stated since day one that the PPD would likely go down on the NV cards as the size of the work units increased. Could be part of the drop.

And this News post may indicate part of the drop. http://folding.typepad.com/news/2009/10/updates-on-new-cores-protomol-and-gpu3-core.html

OpenMM was based on our FAH GPU MD code to start, but has had several enhancements and additions. In particular, it should be much more stable than the previous FAH GPU MD cores. However, this stability does come at a mild cost in performance.


See also: GPU3 FAQ: My points per day (PPD) varies significantly from project to project

So drawing any conclusion about points this early is very premature.
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
User avatar
7im
 
Posts: 14648
Joined: Thu Nov 29, 2007 4:30 pm
Location: Arizona

Re: Benchmarking of GPU3 - Prof Pande?

Postby RAH » Tue May 25, 2010 11:56 am

And another point, once the GPU 3 passes beta to main stream, there won't be a GPU 2. Or for very long.
So your stuck with it anyhow.
Image
User avatar
RAH
 
Posts: 614
Joined: Sun Dec 02, 2007 6:29 am
Location: Florida

Re: Benchmarking of GPU3 - Prof Pande?

Postby Viper666 » Tue May 25, 2010 4:04 pm

7im wrote:Why would the benchmarking change for GPU3? It would appear to be the same as always, according to the new FAQ they put out in that announcement. See also: GPU3 FAQ: How do you decide the credit value of GPU work units?

There is no beta bonus, clearly, considering the PPD is a bit lower on these work units.

Also, it's been stated since day one that the PPD would likely go down on the NV cards as the size of the work units increased. Could be part of the drop.

And this News post may indicate part of the drop. http://folding.typepad.com/news/2009/10/updates-on-new-cores-protomol-and-gpu3-core.html

OpenMM was based on our FAH GPU MD code to start, but has had several enhancements and additions. In particular, it should be much more stable than the previous FAH GPU MD cores. However, this stability does come at a mild cost in performance.


See also: GPU3 FAQ: My points per day (PPD) varies significantly from project to project

So drawing any conclusion about points this early is very premature.


So true it was also stated at that time it would go up On ATI,but it does even run on ATI,so what part do we believe?
Waves to 7im !
ImageImage
Powered by AMD and Open SUSe 11.2 ----The New servers have nothing to do with fixing the stats problem !
User avatar
Viper666
 
Posts: 223
Joined: Fri Dec 14, 2007 9:57 pm

Re: Benchmarking of GPU3 - Prof Pande?

Postby VijayPande » Tue May 25, 2010 4:09 pm

It's early in the beta test and some issues may need tweaking, including points. We'll keep your comments in mind.
Prof. Vijay Pande, PhD
Departments of Chemistry, Structural Biology, and Computer Science
Chair, Biophysics
Director, Folding@home Distributed Computing Project
Stanford University
User avatar
VijayPande
Pande Group Member
 
Posts: 2651
Joined: Fri Nov 30, 2007 6:25 am
Location: Stanford

Re: Benchmarking of GPU3 - Prof Pande?

Postby 7im » Tue May 25, 2010 4:15 pm

Viper666 wrote:
7im wrote:So drawing any conclusion about points this early is very premature.


So true it was also stated at that time it would go up On ATI,but it does even run on ATI,so what part do we believe?
Waves to 7im !


So I'll reiterate that first part of my post. Again, we're premature for NV, and since GPU3 for ATI isn't even released, it's going to be difficult to see if the ATI points go up as was stated.

You can't say the points were supposed to go up at the same time complaining about the client not running. If it doesn't run, how can you judge the points, and vice versa. Catch-22, circular logic, whatever. Besided, NV and ATI are not on the same development schedule. So saying one thing about NV has no correlation to another thing about ATI. It just means it is yet to happen.

Waves back! ;)
User avatar
7im
 
Posts: 14648
Joined: Thu Nov 29, 2007 4:30 pm
Location: Arizona

Re: Benchmarking of GPU3 - Prof Pande?

Postby bollix47 » Tue May 25, 2010 4:19 pm

Mitsimonsta wrote: The same cards on P10504 do 9392ppd (884 points credit, 0:54 per frame).



Where did you see 884 point credit for P10504?

According to PSummary it's 587.
Image
bollix47
 
Posts: 3413
Joined: Sun Dec 02, 2007 5:04 am
Location: Canada

Re: Benchmarking of GPU3 - Prof Pande?

Postby Mitsimonsta » Wed May 26, 2010 1:42 am

VijayPande wrote:It's early in the beta test and some issues may need tweaking, including points. We'll keep your comments in mind.

Thanks Professor. Again, it was not a harsh criticism, just a heads up that some people will feel that there is a disadvantage to helping with the open beta. If donators feel that way, you may find that there is not enough results coming through to make a realistic decision to go live.

I also note that the GPU Benchmark is based on a 3870 and as such is unable to produce a viable benchmark for the new OpenMM based core. Until there is a working ATi core, then it is purely my opinion that the benchmarking system is broken as far as GPU3 is concerned. Hopefully this will be fixed with an ATi implementation that actually uses the full potential of these cards - I have two 4870's hobbl;ed by the lack of a proper GPU2 implementation.

bollix47 wrote:Where did you see 884 point credit for P10504?
According to PSummary it's 587.

Whoops, wrong copy/paste. It is indeed. The PPD figure is correct however, and so is the thrust of my argument.
Mitsimonsta
 
Posts: 54
Joined: Tue Feb 12, 2008 1:53 am

Re: Benchmarking of GPU3 - Prof Pande?

Postby 7im » Wed May 26, 2010 6:29 am

Good question about how to benchmark core_15 WUs that are NV only when the benchmark is ATI, and there is no core_15 for ATI.

And there is a good viable answer, that will have to wait until tomorrow. It's been a long day. Good night. :twisted:





Okay, just to show you I'm not BS'ing, (and IIRC) NV and ATI designations are applied after the benchmarking. So there doesn't need to be a Core 15 for ATI to be able to benchmark NV WUs.

How else do you think they benchmarked the NV only projects prior to GPU3? :twisted:
Last edited by 7im on Wed May 26, 2010 6:38 am, edited 1 time in total.
User avatar
7im
 
Posts: 14648
Joined: Thu Nov 29, 2007 4:30 pm
Location: Arizona

Re: Benchmarking of GPU3 - Prof Pande?

Postby Mitsimonsta » Wed May 26, 2010 7:10 am

7im, I know how they benchmark the GPU units, there is some sort of factoring going on. And yes, I know that the GPU designations go on after the projects are generated. As you can see from my stats, I have been here for a fair while.

My point is that they cannot benchmark a unit which has no core to run on that hardware. The OpenMM core is quite different to the GPU2 core. There is probably enough error-catching and verification code in the current core to slow it down by 30% or thereabouts, but this has not been noted yet by PandeLab. While it has been mentioned that CUDA levels (and hence functions available to the core, aka SSE2 on CPU's) will be detected and the AS will not hand out a unit that the client cannot process, I doubt that is happening yet. I also highly doubt that the core has the higher levels of CUDA switched on yet anyway.

What I was hoping for was a real explanation of how they decided that P10626 is worth 610 points and not more like 840 which keeps it on a roughly even ppd as the current GPU2 projects. Now, assuming that a 3870 running these GPU3 projects would (theoretically) get ~1500ppd as per the currently published benchmark since we have not been told any differently, how did they apply a factor to this unit? Generate the project for GPU2, do a quick bench, and then regenerate it for GPU3?

And the original GPU2 projects were the same across both ATi and nVidia - however because the ATi core was so poorly optimised and they were using a 3870 as the benchmark card, the nVidia cards wiped the floor with them on a PPD basis. The nVidia cards were also much better at the smaller units too. As the unit size and complexity grew, the performance lead enjoyed by the green team did narrow significantly but is still a big gap today.
Mitsimonsta
 
Posts: 54
Joined: Tue Feb 12, 2008 1:53 am

Re: Benchmarking of GPU3 - Prof Pande?

Postby Wrish » Wed May 26, 2010 12:28 pm

Perhaps they're still benchmarking with Brook, and the expected slowdown from OpenMM isn't being reflected. VP did say before that they'd account for the slowdown, and VP did again state here that they'll take our comments into consideration, so I'd leave this as just a beta anomaly.

When people report ATI cards being underutilized, what they have to realize is that since the benchmark is an ATI machine, correcting for that underutilization may just have the effect of reducing Nvidia-derived points while leaving ATI- and CPU-derived points the same. Without the comparative factor of points, people wouldn't be aware of the issue.
Wrish
 
Posts: 319
Joined: Thu Jan 28, 2010 5:09 am

Re: Benchmarking of GPU3 - Prof Pande?

Postby stevehat1 » Wed May 26, 2010 12:36 pm

I would like to add a little more info into the discussion. One of my systems consisting of a gtx 275, gtx 260 216SP, and i7 showed similar GPU3 results to those posted by Mitsimonsta, quite discourgaging for upper end cards. Another two systems with 2x GTS 250's show little or no points difference from GPU2 WU's. So possibly this is not a benchmarking issue but a utilization/optimization issue with GTX 260/275 cards.

On a side note all of these systems also run SMP2, the systems running GPU3 show a marked improvement in SMP2 A3 WU performance.
ImageImage
stevehat1
 
Posts: 85
Joined: Fri Jun 06, 2008 12:33 pm

Next

Return to V6 GPU3 beta (including Fermi) OpenMM

Who is online

Users browsing this forum: No registered users and 1 guest

cron