GPU rackmount farm - 38 GPUs

[Atlas Folder]

Hello F@H!

I'm excited to announce that last week I officially kicked off my 19" rackmounted GPU folding farm project with the creation of a blog and ordering of my testbed components. The goals of the project are to make a significant contribution to F@H research, to increase Folding@home's exposure with Huntington's Disease patients and families, to increase exposure of Huntington's Disease to existing folders, and to draw in volunteers. If you're interested in keeping up with the project feel free to check my blog "Atlas Folding" which is linked in my signature. And of course I will continue to post here as well.

Atlas Folder's initial hardware build out is 19 nVidia GTX295s for 33.97 TFLOPS.

If the machine goes together smoothly it will be fairly straightforward to add nodes and increase the TFLOPS of the system over time either with additional GTX295s or the next generation(s) of nVidia and ATI cards.

Also, to more strongly associate my F@H persona with my project I have changed my F@H forum account from "jfarque" to "Atlas Folder".

Thanks for the support of the F@H Forums and fold on!

Jason

[Ivoshiee]

Good luck for your project.
One similar project http://estoniadonates.wordpress.com/.

[alpha754293]

You might try this kind of PSU:
http://www.zippy.com.tw/P_product_detai ... MPN6-6C00F

It's the only one that I've been able to find that uses the ATX connector as I think that some/most of the other ones (like this one that's rated for 2720W: http://www.zippy.com.tw/P_product_detai ... W4-5R20V3H) uses probably a custom connector.

Any other power supply that I've been able to find comes with the case, and I think the cheapest I've seen was abour $2400. (I think it's a 1620W 3+1 redundant PSU).

Good luck with it.

[Tigerbiten]

If they are going to be pure GPU folders then I'd drop the 9950's and replace them with BE-2400's.
They may fold around 1-2% slower but you'd save 40-50 watts per blade.
Gives the PSU more headroom.

Luck ...........

[Atlas Folder]

alph754293: Thanks for the links, do you have prices on those? I looked at rack mounted power supplies but the cost per watt was pretty high for a donation enterprise. Trying to stretch my investment into the max ppd. My tests in the next day or two should let me see how much headroom the 1250 has. If it's not sufficient then I have to weigh things again.

Tigerbiten: Thanks very much for the tip. I'll take a look at the BE-2400's. I'm an Intel guy and am not that up on the ins and outs of the AMD processors. I mostly was looking at speed vs. price when selecting the 9950.

[jaak ennuste]

I wish You luck with Your GPU HPC project. Challenges in power supply costs, cooling and management. I read Your blog and understand the driving force behind it.

MORE SERIOUS PSU-S FOR 19" RACK, 6 KW AND MORE

Jaak

[Tigerbiten]

Which rack mount cases are you getting ??

Just a quick warning.
Standard ATX cases only have room for 7 expansion slots and to run 4x double width cards you need room for 8.
There was a thread on cases here -> http://foldingforum.org/viewtopic.php?f=38&t=6067

Luck ..........

[alpha754293]

alph754293: Thanks for the links, do you have prices on those? I looked at rack mounted power supplies but the cost per watt was pretty high for a donation enterprise. Trying to stretch my investment into the max ppd. My tests in the next day or two should let me see how much headroom the 1250 has. If it's not sufficient then I have to weigh things again.

Tigerbiten: Thanks very much for the tip. I'll take a look at the BE-2400's. I'm an Intel guy and am not that up on the ins and outs of the AMD processors. I mostly was looking at speed vs. price when selecting the 9950.

No, unfortuantely, I couldn't find prices for those.

You'd probably have to ask for a custom quote and I reckon that they're not going to be cheap. (Don't be surprised if they end up being a few thousand dollars).

So. You've got some options.

I've seen 5U cases (designed for the Tyan B4985 quad Socket F Opteron + an additional quad-Socket F daughter board for 8-sockets altogether) that has a 1620W PSU that comes with it. (As I mentioned before).

I found a few 9U cases that has a massive PSU as well.

Both are I think, IIRC, at least $2k a pop. (When you get into the higher power ranges, they're available, but they're definitely not cheap, and they're definitely more application specific.)

IF I understand it correctly, the current generation of video cards draw most of the power from the auxillary power connector rather than the main bus slot.

Therefore; it is potentially possible that you can get two separate PSUs, one where the "power on switch" is shorted so that it'll start up before the rest of the system.

The only other alternative that I've been able to find would be if you were to get a massive AC-DC converter, and then get DC-to-DC PSUs.

Pretty much regardless of which way you go, it isn't going to be cheap.

[fractal]

You need to balance rack space, power, cooling and cost.

Read, study and understand what the people who have built 4 GPU systems have to say about cooling. Then read it again.

38 GPU's at 100W per GPU is going to chew a lot of power, and convert it ALL to heat. Consider the logistics of getting four dedicated circuits to your rack. Consider the logistics of getting rid of the heat of 13000 BTU's worth of space heaters crammed into your rack. You will need a one ton air conditioner just for the GPU's (n.b. one ton really is a unit of measurement for air conditioners) once you get the heat out of the rack and into your room.

Every single cost / benefit analysis I have performed shows insignificant benefit in terms of PPD/w or PPD/$ from building four GPU systems over building two GPU systems. The only advantage is in space, and you loose that if you have to go to 7U cases.

I would suggest you do the price analysis with two GPU systems ($50 for CPU, $50 for MB, $100 for PSU, $200 for 4u case, rest of parts) vs four GPU systems ($50 for CPU, $150 for MB, $300+ for PSU, $??? for case, rest of parts) and decide if stuffing four GPU's in each box is that important. My experiments with three GPUs on a MSI K9A2 platinum convinced me that having the two slot GPU's on three slot spacing was MUCH easier to keep cool.

But, don't let me get in the way of your dream. All engineering is a making trade offs. Size, cost, performance, heat, maintainability all factor into the equation.

[alpha754293]

I second fractal's opinion. (Course, I'm also an engineer myself, so go figure).

But I just assumed that in preparing for and planning to build the rack that you would have already taking all of what he said into consideration.

In terms pure floating performance, yes, it is high (at 34.219 TFLOPS). On the assumption that a single GTX295 can do about 10k PPD, with 4 of them, you're only at 40k PPD.

I also figured that you probably already got the rack, probably an APC industrial cooling unit/rack that uses at least cold (tap) water (on the assumption that it isn't just out right chilled).

And that you're putting it in a dedicated room (not a closet) or piping the warmed water into your water heater as part of energy recovery.

In terms of PPD/FLOP, I don't think that you would be that much better off.

Course, if you've done all of this thinking already, and you've already worked all of it out; then by all means, go for it.

If you want to go for computational density, you can always split it off into two systems and mount them both in the same chassis by staggering the position of the cards. You'd probably have to build a custom case to be able to do that, but it's possible.

[Atlas Folder]

fractal, owning an industrial electronics design firm gives me a little bit of experience with the issues that you bring up.

I want to test these things for myself. I want to measure the currents and temperatures and see it firsthand.

I have a cooled computer room and 3 48U racks with dedicated heavy electrical circuits.

My initial thought and my fallback position is to use a 4U cantilevered shelf with two or three boards and a smaller PSU and I've run the costs on it. But I'm curious and I don't believe everything I read. The components that I have ordered are all reusable if and when I lower the system's density.

Since I've yet to find a rack mount style PSU that doesn't blow the budget while simultaneously adding wiring complexity I intend to stick with standard PC PSUs. My prediction is that the 1250W PSU is going to be working too hard for comfort in the worst-case testbed system that I'm building; density is great but not if a $300.00 PSU is going to blow a cap every five weeks. If the PSU isn't that badly loaded then the very next item of interest is thermals. If the heat is too much then the decision will likely be to lower density and reduce the power and thermal waste issues simultaneously.

Jason

[alpha754293]

fractal, owning an industrial electronics design firm gives me a little bit of experience with the issues that you bring up.

I want to test these things for myself. I want to measure the currents and temperatures and see it firsthand.

I have a cooled computer room and 3 48U racks with dedicated heavy electrical circuits.

My initial thought and my fallback position is to use a 4U cantilevered shelf with two or three boards and a smaller PSU and I've run the costs on it. But I'm curious and I don't believe everything I read. The components that I have ordered are all reusable if and when I lower the system's density.

Since I've yet to find a rack mount style PSU that doesn't blow the budget while simultaneously adding wiring complexity I intend to stick with standard PC PSUs. My prediction is that the 1250W PSU is going to be working too hard for comfort in the worst-case testbed system that I'm building; density is great but not if a $300.00 PSU is going to blow a cap every five weeks. If the PSU isn't that badly loaded then the very next item of interest is thermals. If the heat is too much then the decision will likely be to lower density and reduce the power and thermal waste issues simultaneously.

Jason

To be honest, I have no idea how those redundant PSUs are built.

It depends on your total budget, time, etc. in doing all this.

If none of those were really an issue, I'd probably retrofit the PSU and everything else onto water or non conductive mineral oil cooling. Hopefully by controlling the thermal, that controls the current leakage, which controls the thermal. And that way maybe that you won't be stressing the PSU as much.

I'm not sure how much power is drawn through the slots. Maybe two 1250W PSUs, where one is used to power the board (etc.) and ONE of the GTX295s, while the other PSU ONLY powers the other three GTXs and gets started before the rest of the system. AFAIK, I don't think that the card ought to really care WHERE it's getting it's power from (whether it's the primary or secondary PSU so long as it's getting it).

I know that for like the Tyan boards and also for 3DLabs cards, it used to be that if there's a power supply fail to the card, it will send a signal to HALT the system in order to avoid toasting it and current overdraw/overload via the slot.

I don't know if the GTX295s and the MSI board will have that.

You might want to take a look at the nVidia Quadro Plex. http://www.nvidia.com/page/quadroplex.html

http://www.nvidia.com/content/quadrople ... Jan08_.pdf

Those use only 640W to 1200W of power.

http://www.nvidia.com/page/quadroplex_c ... chart.html

(It's pretty expensive) but if you have experience with custom ICs, maybe you can design the custom power distribution board in order to ease your PSU woes. *shrug* It's an idea

[jaak ennuste]

Ive played with an idea to have low-power low-cost (low cost per performance) PSU-s for systems, but to power GPU-s from stand-alone 12V mass-produced power-supply. There are some new emerging technologies coming out in conjunction with electric card -- example http://www.zivanusa.com/pdf/NG5-7-9.pdf
But one possibility is to use low-cost mass-produced car battery charger-booster. Just some unfinished thoughts.

Jaak

[Atlas Folder]

I agree wholeheartedly that I'd like a better power solution. As Jaak and I have talked about the price of PSUs is currently one of the major costs of the system -- and that's if you use commodity PC power supplies. My company has some 19" rack power supplies of various kinds and I've searched around and looked at the pointers here, but for W/$ you just can't beat high volume PC supplies. This makes fractal correct in that spreading the load out to 3 or 2 cards per motherboard to reduce the load while simultaneously reducing the heat dissipation issue is the logical response. It adds cost, but not as much as jumping to industrial PSUs.

Jason

[Ivoshiee]

I agree wholeheartedly that I'd like a better power solution. As Jaak and I have talked about the price of PSUs is currently one of the major costs of the system -- and that's if you use commodity PC power supplies. My company has some 19" rack power supplies of various kinds and I've searched around and looked at the pointers here, but for W/$ you just can't beat high volume PC supplies. This makes fractal correct in that spreading the load out to 3 or 2 cards per motherboard to reduce the load while simultaneously reducing the heat dissipation issue is the logical response. It adds cost, but not as much as jumping to industrial PSUs.

Jason

But nothing prevents you setting up custom 19" unit containing a set of regular PSUs. Make one itself or order one (just the U count a a bit different than 1 or 2), custom cabling to all units and you'll have to look a bit closer into the system cooling as well.