GPU Benchmarking concerns

Moderators: Site Moderators, PandeGroup

GPU Benchmarking concerns

Postby Karamiekos » Mon Sep 15, 2008 2:55 am

To consolidate valid concerns on how projects are benchmarked, please post here. Having them all in one spot will provide better data on your case/suggestion how to accurately account for PPD. Please Realize that Standford reserves the right to benchmark using the current procedures they have. Keep it clean.
Zakk Wylde, "Then you start firing back some cocktails."
Rigs
Phenom II 965 With 2 4850s Running BOINC
Quad 8356s Running BOINC
User avatar
Karamiekos
 
Posts: 177
Joined: Tue Jul 15, 2008 1:27 am

Re: GPU Benchmarking concerns

Postby Foxery » Mon Sep 15, 2008 3:00 am

Why?

The process is clearly explained and available to all: All GPU WUs, for both architectures, are benchmarked on a Radeon 3850, fed by an Athlon X2 4400+. However much faster or slower your machine is than a 3850, that's how many PPD you get.

edit:
This fluctuates as the project matures, since companies are still improving their respective code. That's the nature of new technology, we all knew it when we signed up, and making a new complaint thread every week is getting sooo tiring.
Core2 Quad/Q9300, Radeon 3850/512MB (WinXP SP2)
User avatar
Foxery
 
Posts: 322
Joined: Mon Mar 03, 2008 4:11 am
Location: Syracuse, NY

Re: GPU Benchmarking concerns

Postby Karamiekos » Mon Sep 15, 2008 3:17 am

Foxery wrote:Why?


So people don't litter the other threads with it, they can feel free to vent here, and important info on other threads won't get clogged. I have been guilty of this myself over here:
http://foldingforum.org/viewtopic.php?f=52&t=5452&start=210
I personally have no problems with the current system. Many people seem to have something to say. If they have a valid concern, and an improvement suggestion to go along with it, this would be the place to put it.
User avatar
Karamiekos
 
Posts: 177
Joined: Tue Jul 15, 2008 1:27 am

Re: GPU Benchmarking concerns

Postby TPetey » Mon Sep 15, 2008 3:46 am

I think it's not quite as simple as "if your machine is faster or slower than the benchmark machine, you'll get more or less points". We've already seen that certain types of work units fold faster on cards with more shaders and other work units are a better fit for machines with fewer, but faster shaders. It's not a question of machines being faster or slower than the benchmark; the issue is simply "which configuration is better suited to the task at hand".

I'm not fully informed on what direction the project needs to go -- but if there is still much to be learned from work units that are good fits for both kinds of GPU clients, then it makes sense (from the science perspective and from the donor points perspective) to tune the distribution of work assignments such that each flavor of donor client gets the work it does best. Why send large proteins to NVIDIA machines if there are ATI machines available to do it faster? Why send small proteins to be processed by ATI cards when NVIDIA cards can better utilize their faster shaders on those work units?

I know it's not a trivial exercise to categorize donor machines in terms of which work units fit their personalities and to configure the assignment system to match, but aren't there large gains in productivity to be made?
TPetey
 
Posts: 27
Joined: Mon Aug 11, 2008 4:26 pm

Re: GPU Benchmarking concerns

Postby Karamiekos » Mon Sep 15, 2008 4:10 am

I see what your saying, for example, this is some convo from myself and shatteredsilicon:
Karamiekos wrote:
shatteredsilicon wrote:
Karamiekos wrote:The reference GPU card does make a static 1500ppd no matter what WU it is given.


Precisely - and this is WRONG. It skews the value of one "point".


How is this wrong. The new work units are eventually meant to replace the old ones.... so they become the new benchmark. We still have old work units that need to be complaeted though, Stanford just can't dump them. Some variation in PPD is to be expected as they try new things. Sometimes they make steps forward, but as with any research sometimes they try something new and it may not work out.

You also can't compare the new GTX 260/280 "cores" to older generations as they are different. As far as I know the ATI "cores", or streams processors are pretty much the same, just more of them. Just because the work unit is bigger doesn't neccaserily mean there is more room for optimizations. I still like my thought process on, Nvidia cards may have faster SPs but the speed cannot make up for the fact that it has less SPs, in a given situation. The Nvidia card do have less Gflops compared to the ATI ones.

I do see your point, and understand the frustration. There are going to be more of these growing pains in the future I guarentee, it's just part of the process.


So how can we improve this process. I mean the system in place works, but it doesn't work especially well. If we attack this constructively, some progress might be made. In the mean time, the Pande Group and Stanford in general needn't be spending their brainpower on worrying about this insignificant subject, they need to be reviewing folding data, and making observations!!!!

I'm tired right now, but I will think about this. I know someone said iterations/second cannot be looked at as an accurate amount of science done.
Or someone also suggested dropping the Benchmark GPU down to something maybe not as cpu bound.
User avatar
Karamiekos
 
Posts: 177
Joined: Tue Jul 15, 2008 1:27 am

Re: GPU Benchmarking concerns

Postby Karamiekos » Mon Sep 15, 2008 4:29 am

I can't readily think of anything better than the current method PG uses to get a reference. But I'm brainstorming and thought I would share. Maybe they could use the Most powerful GPU to banchmark. Let's think radical. Run the Project in question on a 4870 and a GTX280. Whichever card runs it the fastest, gets to set the bar at 5000ppd. Say the GTX280 was faster at three versus four hours. The GTX280 would make that unit worth 625 points. GTX280 would get 5000ppd, and the 4870 would get 3750ppd.

I really need to get to bed, but I was throwing this out there as a brainstorm mind you. Please feel free.
User avatar
Karamiekos
 
Posts: 177
Joined: Tue Jul 15, 2008 1:27 am

Re: GPU Benchmarking concerns

Postby Karamiekos » Mon Sep 15, 2008 4:39 am

Or let the slower one set the bar @ 1500 PPD lets say.
User avatar
Karamiekos
 
Posts: 177
Joined: Tue Jul 15, 2008 1:27 am

Re: GPU Benchmarking concerns

Postby Mattus » Mon Sep 15, 2008 5:35 am

But that won't really make any difference - relative to one another, the points awarded to two different GPUs will remain in the same proportion. The figures would just change in absolute terms.
folding proudly for team 10, OcUK ¦ what am I folding?
Mattus
 
Posts: 74
Joined: Sat Mar 29, 2008 4:18 am
Location: Leighton Buzzard / Oxford, UK

Re: GPU Benchmarking concerns

Postby Karamiekos » Mon Sep 15, 2008 11:29 am

TPetey wrote:I think it's not quite as simple as "if your machine is faster or slower than the benchmark machine, you'll get more or less points". We've already seen that certain types of work units fold faster on cards with more shaders and other work units are a better fit for machines with fewer, but faster shaders. It's not a question of machines being faster or slower than the benchmark; the issue is simply "which configuration is better suited to the task at hand".

I'm not fully informed on what direction the project needs to go -- but if there is still much to be learned from work units that are good fits for both kinds of GPU clients, then it makes sense (from the science perspective and from the donor points perspective) to tune the distribution of work assignments such that each flavor of donor client gets the work it does best. Why send large proteins to NVIDIA machines if there are ATI machines available to do it faster? Why send small proteins to be processed by ATI cards when NVIDIA cards can better utilize their faster shaders on those work units?

I know it's not a trivial exercise to categorize donor machines in terms of which work units fit their personalities and to configure the assignment system to match, but aren't there large gains in productivity to be made?

It shouldn't really be dificult to do I would think seeing as how they already do that basically. There was a mistake once where an ATI project got assigned to an Nvidia card. Make all the Nvidia projects small proteins, and the ATIs can handle the larger ones. It would seem to reason this might be a more efficient use of the hardware being donated. There would at least have to be an Nvidia and ATI reference card. Comparing CPUs together might work, since the are a bit more similar in nature. The GPUs seem way way different and since the disparity is much more. It seems comparing an AMD to Intel CPU might be comparing a Royal Gala apple to a Red delicious, but AMD to nvidia might be more apples to oranges.
User avatar
Karamiekos
 
Posts: 177
Joined: Tue Jul 15, 2008 1:27 am

Re: GPU Benchmarking concerns

Postby Karamiekos » Mon Sep 15, 2008 11:31 am

I don't think I am reaching, when I say the inflexibility of GPUS comes with the high computational power right now. They are so specialized that even smaller differences in hardware would create bigger differences in performance.
User avatar
Karamiekos
 
Posts: 177
Joined: Tue Jul 15, 2008 1:27 am

Re: GPU Benchmarking concerns

Postby 7im » Mon Sep 15, 2008 3:11 pm

Yes, but your title and your opening post make it sound like there is an actual problem with the benchmark, and there is NOT a problem with the benchmark, only a problem with how people interpret performance differences, or a lack of understanding of the causes of those differences.

You are getting this the pushback from Foxery and Mattus because of the stance you have taken, and IMO, it's not a helpful stance.

Yes, people have concerns, but it would be more helpful if you helped to educate people instead of putting up a target for people to gripe. Address their concerns, not just incite people to complain.
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
User avatar
7im
 
Posts: 14648
Joined: Thu Nov 29, 2007 5:30 pm
Location: Arizona

Re: GPU Benchmarking concerns

Postby Karamiekos » Mon Sep 15, 2008 4:16 pm

7im wrote:Yes, but your title and your opening post make it sound like there is an actual problem with the benchmark, and there is NOT a problem with the benchmark, only a problem with how people interpret performance differences, or a lack of understanding of the causes of those differences.

You are getting this the pushback from Foxery and Mattus because of the stance you have taken, and IMO, it's not a helpful stance.

Yes, people have concerns, but it would be more helpful if you helped to educate people instead of putting up a target for people to gripe. Address their concerns, not just incite people to complain.

I'm not inciting people to complain, I am inviting them to bring "valid" concerns to the "appropriate table."
Karamiekos wrote:To consolidate valid concerns on how projects are benchmarked, please post here. Having them all in one spot will provide better data on your case/suggestion how to accurately account for PPD. Please Realize that Standford reserves the right to benchmark using the current procedures they have. Keep it clean.


How does consolidating valid concerns suggest there is a problem?
concern
something that interests you because it is important or affects you
http://www.wordreference.com/definition/concern

I have said time and time again, I do not see a real problem with the benchmark. I wanted to create a place for people to go without clogging other threads. At the same time I am trying to keep an open mind about peoples concerns, and be unbiased.
User avatar
Karamiekos
 
Posts: 177
Joined: Tue Jul 15, 2008 1:27 am

Re: GPU Benchmarking concerns

Postby codysluder » Mon Sep 15, 2008 7:10 pm

Karamiekos wrote:To consolidate valid concerns on how projects are benchmarked, please post here. Having them all in one spot will provide better data on your case/suggestion how to accurately account for PPD. Please Realize that Standford reserves the right to benchmark using the current procedures they have. Keep it clean.


So are you suggesting that the Mods should discard all the other discussions about GPU benchmarking? ....or maybe all future benchmarking discussions that are not posted in this thread?
codysluder
 
Posts: 2129
Joined: Sun Dec 02, 2007 1:43 pm

Re: GPU Benchmarking concerns

Postby Karamiekos » Mon Sep 15, 2008 7:36 pm

codysluder wrote:
Karamiekos wrote:To consolidate valid concerns on how projects are benchmarked, please post here. Having them all in one spot will provide better data on your case/suggestion how to accurately account for PPD. Please Realize that Standford reserves the right to benchmark using the current procedures they have. Keep it clean.


So are you suggesting that the Mods should discard all the other discussions about GPU benchmarking? ....or maybe all future benchmarking discussions that are not posted in this thread?


No just suggesting that there are strength in numbers, and if you put it all in one spot it might be more powerful, and possibly make more sense. It would definitely have more continuity not being spread out all over the place.

*shrugs* Take of it what you will. Didn't mean to step on any toes. Sorry.

Sorry to you too 7im, I didn't mean for it to come off like that.

"C'est la vie"
User avatar
Karamiekos
 
Posts: 177
Joined: Tue Jul 15, 2008 1:27 am

Re: GPU Benchmarking concerns

Postby bruce » Tue Sep 16, 2008 12:31 am

I've lived with the present benchmarking system long enough to generally accept the things it does well and the things it does not do well. I think the chance of it being changed significantly are pretty low, but that doesn't mean it's impossible to suggest a better alternative.

There was a formula suggested in the previous thread which was quickly refuted -- it won't work very well.

By definition, the benchmark machine has a constant PPD. Putting aside the hardware difference between machine A vs machine B or GPU C vs. GPU D, I think we should gather enough data to suggest a workable formula. That means that any proposed formula should give a reasonably constant PPD on the benchmark hardware. I'm not sure this is possible, but we can figure that out before we actually propose changing the points methodology.

What relatively simple formula will give a constant PPD on that hardware? Once something is proposed, we can attempt to apply the same formula to future WUs and all hardware and see where that puts us. Would it be fairer to others?

Any change in methodology is going to be controversial. If the change means I get more points, I like it. If the change means i get fewer points, I don't like it. Emotional responses are not related to scientific validity. (It's about like taxes. Any change to the tax laws that reduce my taxes is a good change, even if somebody else's taxes go up. Any change that raises my taxes is a bad change, even if somebody sincerely believes that it's fairer, overall.)
bruce
 
Posts: 23698
Joined: Thu Nov 29, 2007 11:13 pm
Location: So. Cal.

Next

Return to General GPU client issues

Who is online

Users browsing this forum: No registered users and 1 guest

cron