Building a computer with several GPUs

[nchowlett]

Heya Folding mods (and users)!

I've done some work building and configuring a computer that has several GPUs dedicated to Folding@home, which are in parallel use.

From my perspective it wasn't too hard, just needed to spend most time sourcing cost-effective GPUs & some adapting hardware. I've written an article giving guidelines for those interested in building something similar, which can be read online: https://www.tessellate.science/microsco ... rcomputer/

Note I did had issues with FAHClient 7.6.21 so installed fah-client 8.1.13 instead (which completed it's install without any errors on Ubuntu 22 LTS). I also had to revert to a slightly older kernel to allow Ubuntu to boot in the first place.

Regards, Nick

[BobWilliams757]

Overall a good quick breakdown of information for someone wanting to contribute to folding. Over the years a number have done blogs and such, and the exposure certainly can't hurt anything. I know that one of the other forums members has done a number of projects on his systems on his site, https://greenfoldingathome.com/

But the link you had to the LTT thread has similar information on power levels and efficiency. He brings up power supply efficiency and others as well, but in the end any information helps.


Just curious, what did you end up using for your rig?

[nchowlett]

Thanks Bob. I used a HP z420 motherboard (that attached to a mining rig chassis without issue) with 4 x RTX 3060 Ti's attached using all available PCIE slots, powered with a EVGA 750 (Watt) GA supply. The power supply is under powered, but underclocking the GPU's has resulted in no apparent shutdowns so far!

[nchowlett]

Heads up I've run into issues with using an underpowered power-supply (I think), which I've manually got around via opening the window to cool the room down (which I believe is allowing the GPUs to run cooler and use less power), then quickly entering the command line and manually modifying power usage using code within the link I provided before.

Either that or when I open a browser power consumption exceeds the PSU). I'm trying to develop some automated script to do this on boot (my first one Ubuntu ignored for some reason), via throttling GPU usage before it enters the OS.

[bikeaddict]

PSUs typically have some headroom (20-30%) above their rated capacity before they will trip the protection. You should be able to output 900W+ if it's working properly.

My GPUs are set with a power limit using nvidia-smi -pl [limit] rather than limiting the clock speed. The GPU will boost to as high a clock as it's capable while not exceeding the set power limit.

You can see how much power a GPU is drawing with nvidia-smi dmon or watch -n1 nvidia-smi for realtime monitoring. And you can use a device like a Kill A Watt or Gardner Bender power meter to see how much the PSU is drawing from the wall combined with the 80 Plus rating to estimate the PSU's actual output.

I use a Kasa/TP-Link KP115 energy monitoring smart plug to monitor power use on my setup.

[nchowlett]

Thanks for the power details! Using my GPUs at full throttle might actually come very close to 1000W (in addition to CPU power), which might exceed even the headroom.

Either way, something weird is going on as my system is randomly stalling after weeks of folding consistently: physically looks like its working (fans spinning and lights), but cannot access OS anymore. Currently I just reboot when noticed, but looking at scripting a checker that emails me when this occurs.

[nchowlett]

Think I know what causes the stalling of folding. Our internet drops out intermittently (which I believe caused my OS's connection software to die), thus killing the request that asks for work from F@h's end.

[nchowlett]

Think I know what causes the stalling of folding. Our internet drops out intermittently (which I believe caused my OS's connection software to die), thus killing the request that asks for work from F@h's end.

Turns out this stalling was from F@h's software, it's been addressed and I've had no issues.