Dedicated rig High end GPU Low end CPU?

A forum for discussing FAH-related hardware choices and info on actual products (not speculation).

Moderator: Site Moderators

Forum rules
Please read the forum rules before posting.
rwh202
Posts: 425
Joined: Mon Nov 15, 2010 8:51 pm
Hardware configuration: 8x GTX 1080
3x GTX 1080 Ti
3x GTX 1060
Various other bits and pieces
Location: South Coast, UK

Re: Dedicated rig High end GPU Low end CPU?

Post by rwh202 »

JonasTheMovie wrote:Today I got myself a powermeter and the results are 200W at bootup, 500W when folding. Wish I had the meter before I upgraded the drivers.
It´s nice to see that the new drivers keeps the load at 100%, no fluctuation as I had before. Increase of PPD seems to be 15%+, really nice.
I was getting keen to try a Fury until I saw those numbers - Are they just for just the folding PC? And this is a single Fury-X? If so, then wow, they're thirsty!
JonasTheMovie
Posts: 88
Joined: Wed Jan 06, 2016 4:16 am
Location: Northern Sweden

Re: Dedicated rig High end GPU Low end CPU?

Post by JonasTheMovie »

At the time the WU had a calculated ppd at 480k, right now it´s around 460W for 380k. I might get another power meter to verify.
That the card draws 300W at full load is expected with a TDP of 275W.
Image
rwh202
Posts: 425
Joined: Mon Nov 15, 2010 8:51 pm
Hardware configuration: 8x GTX 1080
3x GTX 1080 Ti
3x GTX 1060
Various other bits and pieces
Location: South Coast, UK

Re: Dedicated rig High end GPU Low end CPU?

Post by rwh202 »

JonasTheMovie wrote:At the time the WU had a calculated ppd at 480k, right now it´s around 460W for 380k. I might get another power meter to verify.
That the card draws 300W at full load is expected with a TDP of 275W.
Thinking about it, I'd be inclined to suspect the power meter - it may be showing apparent power not real power and be influenced by the power factor of the PSU. 200 W base load for the system is too high and 500 W total is a lot of heat to expel from a small enclosure - the vents would feel like a jet exhaust.
For comparison, I've got an old i7 rig with dual GPUs and a bronze PSU that idles at 80 W and still only draws 420 W for 750k
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Dedicated rig High end GPU Low end CPU?

Post by bruce »

The power meter should have a choice between displaying Watts or Volt-amperes.
JonasTheMovie
Posts: 88
Joined: Wed Jan 06, 2016 4:16 am
Location: Northern Sweden

Re: Dedicated rig High end GPU Low end CPU?

Post by JonasTheMovie »

The powermeter I have has no settings to compensate for reactive power, it always show W, as in real power. But it reads out a power factor of 90, as in 0.90, which concurs with the voltage and amperage. Still seems high with 200W at boot. I kind of regret buying this PSU, its a pos, already changed the fan to keep it quiet, not only the fan but the controller circuit made noise.
Image
Nathan_P
Posts: 1180
Joined: Wed Apr 01, 2009 9:22 pm
Hardware configuration: Asus Z8NA D6C, 2 x5670@3.2 Ghz, , 12gb Ram, GTX 980ti, AX650 PSU, win 10 (daily use)

Asus Z87 WS, Xeon E3-1230L v3, 8gb ram, KFA GTX 1080, EVGA 750ti , AX760 PSU, Mint 18.2 OS

Not currently folding
Asus Z9PE- D8 WS, 2 E5-2665@2.3 Ghz, 16Gb 1.35v Ram, Ubuntu (Fold only)
Asus Z9PA, 2 Ivy 12 core, 16gb Ram, H folding appliance (fold only)
Location: Jersey, Channel islands

Re: Dedicated rig High end GPU Low end CPU?

Post by Nathan_P »

I did recommend buying a decent platinum rated PSU but even so, a system pulling 200w at idle doesn't sound right. My daily rig with a pair of x5670's, 2 SSD's, 2 HDD's, 2 optical drives and a GTX 670 only pulls 140w at idle and my 980ti/970 rig running on a Z9PE D8 idles at 100w.

Bear in mind that TDP is the card running flat out and they always err on the side of caution, my 970 has a TDP of 160w yet under a full folding load its only ever pulled a max of 140w according to my power meter, however a fury x can pull over 400w at full folding load so your 500w number is not far off.
Image
JonasTheMovie
Posts: 88
Joined: Wed Jan 06, 2016 4:16 am
Location: Northern Sweden

Re: Dedicated rig High end GPU Low end CPU?

Post by JonasTheMovie »

Well, it idles at 120W with everything fully loaded and folding paused. 200W was at bootup as I wrote, right before core kicked in and shot it up to 500W. Maybe I should of made that more clear. So the Fury seems to draw 300-400W depending on workload. It´s nice to see my average going up to 375kppd, but it should come up a bit more in a couple of days, the rig had some downtime. With electricity costs here in northern Sweden being quite low (you can heat your house on electricity with freezing winter outside) the efficiency wasn´t my concern, but the build quality of a platinum PSU would of been nice.
Image
FldngForGrandparents
Posts: 70
Joined: Sun Feb 28, 2016 10:06 pm

Re: Dedicated rig High end GPU Low end CPU?

Post by FldngForGrandparents »

Couple of things to add that I have not seen discussed here. First my credentials as I am not just a self proclaimed guru. I am a systems engineer for a large mobile company. I am a RHCE/RHCSA and Windows admin with many years under my belt as the primary decider for hardware decisions of 10's of thousands of servers. So I tend to test and research plenty before I make decisions. Even on my home gear.

I am running dedicated folding machines. I am currently running GTX 970/980 reference and Asus Strix GTX 980Ti OC's with AMD 8350 and Phenom II X6 1100T CPU's all under Ubuntu 14.04 and only doing GPU folding. The OS does play a big part in your PPD. I saw 30-40k per card performance gain over Windows 7/10.

Another thing I see people on here saying is get the cheapest CPU when using GPU's. That is not always the case. As Nvidia GPU's require a lot CPU core for each card. My testing has shown that I gain an additional 20k or so of PPD per card on my 8350 compared to the 1100T CPU. I haven't had the resources to fully test if this is due to the CPU core speed increase (500Mhz on boost) or the bus/cache. I'll try with a fx-9590 next to see how much extra I get.

My next testing will be to see what the difference is between PCIE 1x and 16x is and if this will have any impact with running with Nvidia GPU's. I do not have any AMD GPU's as they require much more power and I have not seen compelling evidence that the cost/PPD is any better.

One last note about power. One of my machines I am testing/tuning with is a AMD Phenum II 1100T with a GTX 980 reference and the Asus 980Ti OC. Both cards have a heavy overclock. This machine also has a SSD and two 3TB drives/CD-ROM. I am running with a 700w PS without any issues. nvidia-smi shows 140w draw on GTX 980 and 240w on GTX 980Ti.

I have been buying used GTX cards from craigslist for smoking deals. Definitely a good place to pick them up and save some cash.

I will be posting a full how-to for building a dedicated Ubuntu folding server in the next week. As the docs I see are usually pieced together and don't include things like how to over-clock, additional monitoring tools, etc.

Brad
Image

Dedicated to my grandparents who have passed away from Alzheimer's

Dedicated folding rig on Linux Mint 19.1:
2 - GTX 980 OC +200
1 - GTX 980 Ti OC +20
4 - GTX 1070 FE OC +200
3 - GTX 1080 OC +140
1 - GTX 1080Ti OC +120
PS3EdOlkkola
Posts: 184
Joined: Tue Aug 26, 2014 9:48 pm
Hardware configuration: 10 SMP folding slots on Intel Phi "Knights Landing" system, configured as 24 CPUs/slot
9 AMD GPU folding slots
31 Nvidia GPU folding slots
50 total folding slots
Average PPD/slot = 459,500
Location: Dallas, TX

Re: Dedicated rig High end GPU Low end CPU?

Post by PS3EdOlkkola »

I do not have any AMD GPU's as they require much more power and I have not seen compelling evidence that the cost/PPD is any better.
Regarding PCIe bus width and speed using AMD GPUs (Fury X) on AMD processors (FX-8350) and Intel processors (4771 and 4790K) the average PPD results are remarkably close:

All systems: Win7 x64, 16GB+ DDR3 (CAS 9), no paging file, SSDs, no SMP folding enabled, Fury X at stock GPU clock (1.050 GHz), 1,000+ watt platinum power supplies.

Project 9172:

FX-8350 on PCIe 2.0 8x: Average PPD = 337,768
i7-4771 on PCIe 3.0 8x: Average PPD = 337,768
i7-4790K on PCIe 3.0 8x: Average PPD = 342,604

It surprised me that the i7-4771 was exactly the same as the FX-8350 on an interface that is half as fast as the Intel CPU. At least for AMD GPUs (on this wu). the speed and width of the PCIe interface is not a huge factor in PPD performance. I'm currently (re)building a i7-3930K equipped with two Fury X's on PCIe 3.0 x16 interfaces. If there is an improvement in PPD given the wider PCIe lanes, I'll post the results.
Image
Hardware config viewtopic.php?f=66&t=17997&p=277235#p277235
JonasTheMovie
Posts: 88
Joined: Wed Jan 06, 2016 4:16 am
Location: Northern Sweden

Re: Dedicated rig High end GPU Low end CPU?

Post by JonasTheMovie »

PS3EdOlkkola, have you updated to Vulcan beta yet?
Image
mmonnin
Posts: 324
Joined: Wed Dec 05, 2007 1:27 am

Re: Dedicated rig High end GPU Low end CPU?

Post by mmonnin »

What is the PCI-E bandwidth usage % on your FX and i7-4771? My Win7 3770k 3.0 16x is currently using 17-19% with a GTX-970. Although my 980Ti in Ubuntu is sitting at just 1% PCI bandwidth utilization. If there was limited PCI-E bandwidth then Linux may be better for someone.
FldngForGrandparents
Posts: 70
Joined: Sun Feb 28, 2016 10:06 pm

Re: Dedicated rig High end GPU Low end CPU?

Post by FldngForGrandparents »

mmonnin wrote:What is the PCI-E bandwidth usage % on your FX and i7-4771? My Win7 3770k 3.0 16x is currently using 17-19% with a GTX-970. Although my 980Ti in Ubuntu is sitting at just 1% PCI bandwidth utilization. If there was limited PCI-E bandwidth then Linux may be better for someone.
Interesting enough I just brought up another machine with a AMD 8150. So running a AMD 8150, 8350, and AMD Phenum II 1100T. Some cards are in 16x and some in 4x. I can see that with folding this won't make a difference. I could even use 1x as the most throughput I have seen is 120MB/s of TX and 182MB/s of RX. These are just bursts as they mostly idle at 15MB/s TX and 60MB/s RX. A 1x PCIe maxes out at 250MB/s.

I think I am going to merge all six of my GPU's under the single 8150 on my GA-990FXA-UD3 R5 motherboard. And use these adapters. http://amzn.com/B00HCBOHKO Two 750W PS should do the trick.
Image

Dedicated to my grandparents who have passed away from Alzheimer's

Dedicated folding rig on Linux Mint 19.1:
2 - GTX 980 OC +200
1 - GTX 980 Ti OC +20
4 - GTX 1070 FE OC +200
3 - GTX 1080 OC +140
1 - GTX 1080Ti OC +120
7im
Posts: 10189
Joined: Thu Nov 29, 2007 4:30 pm
Hardware configuration: Intel i7-4770K @ 4.5 GHz, 16 GB DDR3-2133 Corsair Vengence (black/red), EVGA GTX 760 @ 1200 MHz, on an Asus Maximus VI Hero MB (black/red), in a blacked out Antec P280 Tower, with a Xigmatek Night Hawk (black) HSF, Seasonic 760w Platinum (black case, sleeves, wires), 4 SilenX 120mm Case fans with silicon fan gaskets and silicon mounts (all black), a 512GB Samsung SSD (black), and a 2TB Black Western Digital HD (silver/black).
Location: Arizona
Contact:

Re: Dedicated rig High end GPU Low end CPU?

Post by 7im »

There are known performance bottlenecks at 1x despite your current estimates. I personally saw a 15% PPD decrease running a GTX 760 in a 16x slot wired at 1x. Good luck running that many.
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
FldngForGrandparents
Posts: 70
Joined: Sun Feb 28, 2016 10:06 pm

Re: Dedicated rig High end GPU Low end CPU?

Post by FldngForGrandparents »

7im wrote:There are known performance bottlenecks at 1x despite your current estimates. I personally saw a 15% PPD decrease running a GTX 760 in a 16x slot wired at 1x. Good luck running that many.
Well I just ordered a couple of these: http://amzn.com/B00IMYODGS I will see what the performance difference is in a couple days.
Image

Dedicated to my grandparents who have passed away from Alzheimer's

Dedicated folding rig on Linux Mint 19.1:
2 - GTX 980 OC +200
1 - GTX 980 Ti OC +20
4 - GTX 1070 FE OC +200
3 - GTX 1080 OC +140
1 - GTX 1080Ti OC +120
PS3EdOlkkola
Posts: 184
Joined: Tue Aug 26, 2014 9:48 pm
Hardware configuration: 10 SMP folding slots on Intel Phi "Knights Landing" system, configured as 24 CPUs/slot
9 AMD GPU folding slots
31 Nvidia GPU folding slots
50 total folding slots
Average PPD/slot = 459,500
Location: Dallas, TX

Re: Dedicated rig High end GPU Low end CPU?

Post by PS3EdOlkkola »

@JonasTheMovie - I don't use beta drivers. I have too many rigs to maintain so I keep everything on the same driver version. Once a new driver has been proven to be stable and their is a performance improvement and it's officially WHQL released, then I'll upgrade. I've been bitten too many times in the past on upgrading drivers too soon, so I take a more conservative approach.

@ mmonnin - from what I recall, front-side bus utilization was very low using AMD/ATI GPUs, approximately 5% to 10% loading.
Image
Hardware config viewtopic.php?f=66&t=17997&p=277235#p277235
Post Reply