Comparing performance between folding clients

[sergd]

Hi all - is there an easy way to compare by how much my GPU client is faster than a CPU client? I keep hearing that it is much faster, but it would be nice to know by how much and, for example, how both of them compare to a PS3 client. From what I see from other posts, points and WU seems to be a client type specific, e.g. NVIDIA GPU points can't accurately compare to ATI GPU points. Is that the case? Is there a way to perform an apple to apple comparison between different clients (e.g. using the same WU compiled for different core and clients)?

[Zagen30]

Hi all - is there an easy way to compare by how much my GPU client is faster than a CPU client? I keep hearing that it is much faster, but it would be nice to know by how much and, for example, how both of them compare to a PS3 client. From what I see from other posts, points and WU seems to be a client type specific, e.g. NVIDIA GPU points can't accurately compare to ATI GPU points. Is that the case? Is there a way to perform an apple to apple comparison between different clients (e.g. using the same WU compiled for different core and clients)?

This can't happen because the same WUs are not made for multiple clients.

Are you asking about Points Per Day? If you get a monitoring client like FahMon and point it to the client directories, it'll calculate how many points per day each client produces (doesn't work with PS3s). I don't know the site, but there is a webpage that has a database of PPD for most, if not all, of the clients, with multiple hardware configurations. On the ATI and Nvidia forums there are charts of how many PPD most models get. PS3s get approximately 900 PPD when folding 24/7.

The problem with comparing non-PS3 clients is that it depends a good deal on your hardware. The GPU2 client can do as low as a couple hundred PPD (low-end ATI card) to several thousand (Nvidia GTX 280). The SMP client can range from the hundreds (low-end dual-core) to a couple thousand (Core i7). The uniprocesoor client is the only one, other than the PS3, that doesn't have a huge range (only looking at one instance being run), mainly because the current advances in CPU technology seem to be more in number of cores and less in more efficient architecture.

[kelliegang]

Client statistics by OS
The Pande Group from Stanford have done some conversions of the output from each different client type to make them all comparable... how this relates to points per day on each of the clients I've no idea...

If you average it out, each active playstation out there is producing 0.06 [x86]tflops but this is the only client you could possibly hope to come up with that kind of figure for... all the other clients produce a different ppd depending on the system they are being run on.

It'd be nice to have some form of comparitive figure though, would help in determining our personal "ideal" balance for systems which we run.

[7im]

While the GPU client may be 20x faster at folding than the CPU client, it only earns 10x the points, or whatever the current values are. This is because the work done by the GPU is of a very limited scope, and not 20x as useful as the CPU client. While nowhere near perfect, the best comparison to make at this time is the rate PPD of the benchmark system for each client. 110 CPU, 900 PS3, 1500 GPU, 1760 SMP.

Do NOT compare flops, as those numbers can be deceptive, especially when you take in to account the relative speeds I mentioned above. 20x the flops does not mean the client is 20x faster nor produces 20x the amount of useful science.

[jaak ennuste]

20x the flops does not mean the client is 20x faster nor produces 20x the amount of useful science.

Are Your sure? Running one 1 GFLOPS PC or running 20 PC-s each 1 GFLOPS. I really think that I give 20x more to science???!

Jaak

[7im]

Again, that is not an accurate assumption if you based it on the GFLOPS.

The CPU client is much slower, but can do a wide variety of calculations. The GPU is much faster, but can only do one type of calculation. Read the history section of the GPU FAQ for more info on the nature of GPU folding.

Note, I am not arguing against you. I am not saying the 20 CPUs are better. I am saying you can't make that judgement based on the GFLOPS. The GFLOPS count is deceptive in the GPU client, as it has to repeat some calculations twice because it can't store the results for some simulations. So while 20x faster, it might only be doing 10x the actual work. I don't have exact numbers, but that should demonstrate why you can't compare GFLOPS.

[kelliegang]

I thought points per day was based purely as a comparison of your system with the benchmark system for that client type and had no real useable inference to the science being achieved.
It would be nice to be able to compare all the different clients together is all, Granny smith apples to luscious red apples at least, rather than apples with oranges

[7im]

I thought points per day was based purely as a comparison of your system with the benchmark system for that client type and had no real useable inference to the science being achieved.
It would be nice to be able to compare all the different clients together is all, Granny smith apples to luscious red apples at least, rather than apples with oranges

Yes, and then no. You have it half right.

Because the benchmark PPD value is based primarily on scientific production, and since your system performance is evaluated relative to that benchmark, your system PPD is tied to the science it produces. It's not exactly linear, and science isn't the ONLY criteria, but science is the MAIN consideration in regards to the benchmark.

PPD is in the only apples to apples comparison you can make at this time. But again, it's not exact.

[kelliegang]

Cheers for the clarity.

Though if, in terms of importance, PS3 = CPU = GPU = SMP .... then swouldnt it be a good idea to apply some form of weighting so the ppd be more equal rather than things like 6,000 ppd for nvidia clients compared to 100-150ppd for cpu clients?

[Zagen30]

Cheers for the clarity.

Though if, in terms of importance, PS3 = CPU = GPU = SMP .... then swouldnt it be a good idea to apply some form of weighting so the ppd be more equal rather than things like 6,000 ppd for nvidia clients compared to 100-150ppd for cpu clients?

I think the issue is that it's kind of hard to rate the clients in terms of importance because they each have their advantages and weaknesses, and it's kind of hard to average out the benefits to some sort of hardline figure.

Maybe the only ones that could be compared are SMP vs. CPU since they both run on the same hardware. I think I once saw that part of the reason why the SMP client gets more PPD vs. running as many instances of the uniprocessor client as you have physical cores is that the SMP has the advantage of inter-core communication, allowing more science to be done (the other parts are that the SMP clients are still not nearly as stable as the CPU client and have much shorter deadlines).

[jaak ennuste]

Isn't there now apples-to-apples comparisin x86 FLOPS metrics up and running now?
http://fah-web.stanford.edu/cgi-bin/mai ... pe=osstats

[PlayLoud]

Not if what 7im said is true, in that the GPU clients have to rerun some of those calculations, because they do not retain the results in memory like a CPU would.

[kelliegang]

Well Jaak, if I understand what is being said, the argument is that flops dont represent the "science" as clearly as points per day does I think? Surely though you could easily give an x86 factor instead of points per day [figure out total flops produced by one client type, divide this by the points contributed by that client type; voila x86 rating per point]

[Pick2]

If the only thing the clients did were "flops" , It might be easier to figure. The clients are calculating the path each atom takes as it morphs from one compound into another. It is much more complex than "How many times can I multiply each second".

[bruce]

Though if, in terms of importance, PS3 = CPU = GPU = SMP .... then swouldnt it be a good idea to apply some form of weighting so the ppd be more equal rather than things like 6,000 ppd for nvidia clients compared to 100-150ppd for cpu clients?

I don't think that you can say that PS3 = CPU = GPU = SMP. In fact, all are important but it is very difficult to know how to compare them. To make a Waldorf salad, you need a mixture of fruits. If it's all apples or all oranges or all nuts, it's not a good salad. If you have two salads with different mixtures, both can be acceptable.

If there are reasonable variations in the production of the four clients, science will get done. If too many people choose NOT to run, say the SMP client becuase it's harder to make work, then the Pande Group might consider giving it more points to influence your choices, but the generally tend to make no changes as long as the overall mixture is in the reasonable range.