Guidance for building a folding dedicated computer

A forum for discussing FAH-related hardware choices and info on actual products (not speculation).

Moderator: Site Moderators

Forum rules
Please read the forum rules before posting.
boristsybin
Posts: 50
Joined: Mon Jan 16, 2017 11:40 am
Hardware configuration: 4x1080Ti + 2x1050Ti
Location: Russia, Moscow

Re: Guidance for building a folding dedicated computer

Post by boristsybin »

QuintLeo wrote:AMD FX series are a lot lower cost for the same number of cores,
...
Ignore "hyperthreaded" cores - they do NOT work well to support folding GPUs, they MIGHT be worth a QUARTER of a real core.
...
In fact, ignore AMD
AMD FX supports only pci-e v2, that, in theory, may limit gpu productivity

i3 4130 with HT enabled is enough for three 1080ti, so HT does work well to support folding GPUs

amd gpus loads CPU less than nVidia`s :
with amd gpu, FahCore uses 100% per thread on task start and and drops to ~60% per thread on main time of task count,
with nVidia, FahCore uses 100% of CPU thread whole folding time.
Image
rwh202
Posts: 425
Joined: Mon Nov 15, 2010 8:51 pm
Hardware configuration: 8x GTX 1080
3x GTX 1080 Ti
3x GTX 1060
Various other bits and pieces
Location: South Coast, UK

Re: Guidance for building a folding dedicated computer

Post by rwh202 »

Leonardo wrote:
foldy wrote:RX Vega 64 makes 600-850k PPD so that is still reasonable too but uses more power.
Unless your electricity rates are very low, I think that performance is not good considering the power consumption of that GPU.
I think it's worth pointing out to those that are concerned by power use, that both manufacturers ship their cards past the optimal point in the power / performance curve.
Good results can be had by setting lower power targets than those that the cards ship with.
I have a rig with a pair of 1080 FE with enforced power limits of 130 W on them (down from 180 W). Performance is still good (650 - 700 k PPD per card) and just less than 300 W from the wall.
Overclocked 1080s might do 900 k PPD, but for double the power draw.
foldy
Posts: 2061
Joined: Sat Dec 01, 2012 3:43 pm
Hardware configuration: Folding@Home Client 7.6.13 (1 GPU slots)
Windows 7 64bit
Intel Core i5 2500k@4Ghz
Nvidia gtx 1080ti driver 441

Re: Guidance for building a folding dedicated computer

Post by foldy »

GTX 1080 ti with 55% power limit 1000k PPD at 250 watts (incl. CPU) instead of 1200k PPD at 350 watts.
Leonardo
Posts: 261
Joined: Tue Dec 04, 2007 5:09 am
Hardware configuration: GPU slots on home-built, purpose-built PCs.
Location: Eagle River, Alaska

Re: Guidance for building a folding dedicated computer

Post by Leonardo »

Thanks, Foldy. I've never tried adjusting power downward, at least not for Folding. I may just give that a try this weekend and see how it shakes out.
Image
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: Guidance for building a folding dedicated computer

Post by SteveWillis »

I have a couple of GPUs that run hot, around 90C that I reduce power on to cool them off and slow their fans. I don't like to run much over 80C and 75% fan.
For anyone running nvidia cards on Linux:

To display the current power

Code: Select all

nvidia-smi 
To set the power for GPU 0

Code: Select all

sudo -n nvidia-smi -i 0 --persistence-mode=1
sudo -n nvidia-smi -i 0 --power-limit=115
Image

1080 and 1080TI GPUs on Linux Mint
SombraGuerrero
Posts: 118
Joined: Mon Mar 16, 2009 3:06 am

Re: Guidance for building a folding dedicated computer

Post by SombraGuerrero »

So, in reading this thread, taking some new power readings on my rig, and doing some of my own quick calculations, I realize that folding with my CPU adds about 60-70 watts to my power load. For that extra load, I'm getting roughly about 43 - 77 PPD/watt/WU. This pales in comparison to my single GPU, which gives me about 140 - 300 PPD/watt/WU. So, this means I'm getting not even 1 PPD/watt on my CPU. From an energy cost perspective, it seems like I'd be better off not folding with my CPU. Is that a fair assessment, or are my numbers too wishy-washy?
Leonardo
Posts: 261
Joined: Tue Dec 04, 2007 5:09 am
Hardware configuration: GPU slots on home-built, purpose-built PCs.
Location: Eagle River, Alaska

Re: Guidance for building a folding dedicated computer

Post by Leonardo »

Fair assessment. That is exactly why so many of us don't bother with setting up CPU slots.
Image
SombraGuerrero
Posts: 118
Joined: Mon Mar 16, 2009 3:06 am

Re: Guidance for building a folding dedicated computer

Post by SombraGuerrero »

I suppose there is an argument that the CPU cores obviously focus on different projects than the GPU cores, so I would hate to think that those get less attention, but I suppose there will always be plenty of other CPU folders out there.....
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Guidance for building a folding dedicated computer

Post by bruce »

The "general rule" does not apply to everybody. I can see several possibles.

1) You have a Windows Gaming machine or the equivalent Linux machine with one or more powerful GPUs and a CPU that supports many threads.
> 1A) You pay for power.
> 1B) You live in a home or a school Dorm where somebody else pays the electric bill -- or you have a large enough roof-top solar system so that you rarely have an electric bill.
2) you have a laptop with a weak iGPU that can't handle FAH so the CPU is your only choice.
3) You run Mac OS-X (where FAH does not support GPUs) so again, the CPU is your only choice..

You guys in group 1A decide CPUs aren't practical.
Those in groups 1B, 2, and 3 run with their CPUs.

Let's not denigrate the contributions that others can make just because you happen have a powerful system. All contributions to science are valuable.
SombraGuerrero
Posts: 118
Joined: Mon Mar 16, 2009 3:06 am

Re: Guidance for building a folding dedicated computer

Post by SombraGuerrero »

My feelings exactly, Bruce. I pay for electricity, but I wanted to run these numbers in the interest of awareness. Although it is an interesting exercise in conservation to run these types of numbers, I also feel that unless there is a grave need to cut consumption, it is sort've a slight to the CPU core project owners to ignore their work in the interest of "practicality."
QuintLeo
Posts: 52
Joined: Sat Jun 17, 2017 5:43 am

Re: Guidance for building a folding dedicated computer

Post by QuintLeo »

boristsybin wrote:
QuintLeo wrote:AMD FX series are a lot lower cost for the same number of cores,
...
Ignore "hyperthreaded" cores - they do NOT work well to support folding GPUs, they MIGHT be worth a QUARTER of a real core.
...
In fact, ignore AMD
AMD FX supports only pci-e v2, that, in theory, may limit gpu productivity

i3 4130 with HT enabled is enough for three 1080ti, so HT does work well to support folding GPUs

amd gpus loads CPU less than nVidia`s :
with amd gpu, FahCore uses 100% per thread on task start and and drops to ~60% per thread on main time of task count,
with nVidia, FahCore uses 100% of CPU thread whole folding time.
PCI 2.0 16x is more than enough for any GPU - even the 1080ti.
8x can lose a little bit, but commonly only a few percent at most even on the 1080ti.
Any quad core should be ENOUGH to support 4 x 1080ti - but have you priced Intel quad-cores vs the FX 6300 6-core or even the 8320e 8-core lately - and given my results as I talk about below, I'm not sure a single core is enough to support a 1080ti fully even WITH Hyperthreading enabled unless it's a very HIGH end CPU.

I tried to run *2* 1080 ti with a G4600 (Kaby Lake 2 core WITH hyperthreading, the SAME as a low-to-midrange Kaby Lake i3 except less cache) - the setup was DID CPU LIMIT on some work units, and was right on the edge most of the time on the rest.
Dedicated folding machine that wasn't running anything else, one 1080ti on a PCI-E 3.0 16x slot the other on a PCI-E 3.0 4x slot.
I can't recommend trying to run a 1080 ti on a SINGLE core - even with hyperthreading - as it gets very marginal.

AMD GPUs load the CPU a lot less in general because they're doing less work - FAH is not AMD GPU friendly, something about the differences between AMD and Nvidia GPU design make FAH work a LOT better on Nvidia GPUs.
QuintLeo
Posts: 52
Joined: Sat Jun 17, 2017 5:43 am

Re: Guidance for building a folding dedicated computer

Post by QuintLeo »

foldy wrote:The nvidia driver can only use one CPU core per GPU.
Looks like really old slow CPUs limit the new fast GPUs.
QuintLeo wrote: "My short-lived rig with a pair of GTX 1080ti running on a dual-core Intel G4600 (Kaby Lake 3.5 Ghz) WAS CPU LIMITING NOTICEABLY on both GPUs"
That is interesting, did anyone else had this problem with a new fast GPU but only dual core for 2 GPUs?
The pcie bandwidth limit is only a big problem on Windows. Using Linux you loose only some % and so using x1 risers on Linux is possible.
Machine in question was running Ubuntu 14.04 64-bit.
Joe_H
Site Admin
Posts: 7856
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Guidance for building a folding dedicated computer

Post by Joe_H »

QuintLeo wrote:I tried to run *2* 1080 ti with a G4600 (Kaby Lake 2 core WITH hyperthreading, the SAME as a low-to-midrange Kaby Lake i3 except less cache) - the setup was DID CPU LIMIT on some work units, and was right on the edge most of the time on the rest.
Dedicated folding machine that wasn't running anything else, one 1080ti on a PCI-E 3.0 16x slot the other on a PCI-E 3.0 4x slot.
That is marketed as a Pentium. Limitation you did not mention is that it only supports a total of 16 PCIe lanes, so depending on what else needed PCIe lanes such as for access to storage, there would not have been enough to run one GPU at 16x and another at 4x. Quite likely your 16x GPU was actually running at 8x at best, that would depend on your board.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
rwh202
Posts: 425
Joined: Mon Nov 15, 2010 8:51 pm
Hardware configuration: 8x GTX 1080
3x GTX 1080 Ti
3x GTX 1060
Various other bits and pieces
Location: South Coast, UK

Re: Guidance for building a folding dedicated computer

Post by rwh202 »

Joe_H wrote:
QuintLeo wrote:I tried to run *2* 1080 ti with a G4600 (Kaby Lake 2 core WITH hyperthreading, the SAME as a low-to-midrange Kaby Lake i3 except less cache) - the setup was DID CPU LIMIT on some work units, and was right on the edge most of the time on the rest.
Dedicated folding machine that wasn't running anything else, one 1080ti on a PCI-E 3.0 16x slot the other on a PCI-E 3.0 4x slot.
That is marketed as a Pentium. Limitation you did not mention is that it only supports a total of 16 PCIe lanes, so depending on what else needed PCIe lanes such as for access to storage, there would not have been enough to run one GPU at 16x and another at 4x. Quite likely your 16x GPU was actually running at 8x at best, that would depend on your board.
Actually, the 16 lanes are just the separate dedicated graphics lanes and are typically tied to one PCI-E slot (unless you have an SLI board with switches etc.). There are further lanes provided by the chipset that provide storage and the other slots, so 16x and 4x is quite normal on pentiums. They are then connected to the CPU over the DMI.
v00d00
Posts: 396
Joined: Sun Dec 02, 2007 4:53 am
Hardware configuration: FX8320e (6 cores enabled) @ stock,
- 16GB DDR3,
- Zotac GTX 1050Ti @ Stock.
- Gigabyte GTX 970 @ Stock
Debian 9.

Running GPU since it came out, CPU since client version 3.
Folding since Folding began (~2000) and ran Genome@Home for a while too.
Ran Seti@Home prior to that.
Location: UK
Contact:

Re: Guidance for building a folding dedicated computer

Post by v00d00 »

I have a question on PCI-E utilisation between Gen 2 and 3.

On my AMD FX i have Gen 2, my GPU PCIE utilisation is at 3-4% at any given time according to nvidia-smi, running a GTX 970.
On my Intel Skylake i7, my GPU PCIE utilisation is at 2-3% at any given time according to nvidia-smi, running on a different GTX 970.
Both systems running Slackware 14.x with a modern kernel. But I also got this same amount last year on Debian.

PPD is essentially identical between the two.

Does PCI-E gen 3 actually make any difference, because it looks like the amount of data being sent and received comes nowhere near to saturating the lanes?

As for what computer to build for dedicated, its an apples and oranges discussion. In reality it doesnt really matter. If it has a relatively modern quad core cpu, a board with at least 2 PCIE slots, a 4GB stick of ram and a decent 650-850W PSU, you are good to go for most modern graphics card from either manufacturer. I would say to go with SSD, not for the fact you need the speed, but that it uses very little power and you can generally pick up a cheap industrial grade drive (16GB is fine for OS and folding) for very little money from ebay. If your budget is unlimited you could go with 1080 Ti with either water cooling or hybrid cooling, or maybe a couple of 1060's.

Key things to consider include Power Usage, Noise and Ambient Temperature. If electricity costs a lot, then aim for something thats much more streamlined on the PPD/watt. If you live in a place where the average outside temperature is 30-40C then you may end up having to use a lot more power to keep your home cool when running folding gpu. Similarly dont run cpu client. 6000PPD on avg for an FX. My 970 uses only 60w more and does about 270k/day. Their are enough people running cpu/smp that dedicate server level systems to it. If you do it for the science and you live at the North Pole, then running cpu client is probably going to help keep your house warm. Otherwise I see it as a waste of power that could be used to run gpu.

As ever, YMMV.
Image
Post Reply