AMD CPU systems & PCIE 4.0

Moderators: Site Moderators, FAHC Science Team

Theodore
Posts: 118
Joined: Sun Feb 10, 2019 2:07 pm

AMD CPU systems & PCIE 4.0

Post by Theodore »

I've finished my personal research on AMD CPUs for powering GPUs for folding.
Is it worth jumping ship from Intel to AMD?

Short answer:
Depends...
Intel CPUs are limited to 16 PCIE lanes, which provides up to 3 GPUs, at PCIE 4x speeds or higher (in other words, no PCIE bandwidth limitation for GPUs in Linux).
Even the power hungriest RTX 2080 ti, at 300W (230W in power cap mode), would use no more than 1000W on the wall (800W when power limiting).
So there's clearly room for more! Though the 16 lanes aren't sufficient to add a 4th GPU without PCIE bandwidth limitation.

AMD CPUs give you access to 24 PCIE lanes instead of 16, which allows you to run an extra GPU at PCIE 4x speed, where as some slots on Intel boards would fall back to PCIE 1x speeds when adding more than 3 dedicated GPUs.


The long answer is:
It seems that the most interesting CPU to choose for folding is the AMD Ryzen 5 2600 CPU @ USD $148; or the Ryzen 3 1200 or 1300x.
Which ever CPU you choose, depends on how many GPUs you plan on running.
The Ryzen 3 CPUs have only 4 cores and are limited to 4 GPUs in Linux (3 in Windows). Great, if you don't plan on expanding beyond this. Also great that it can run each card at PCIE 4x or higher speeds.
The Ryzen 5 also allows for 4 GPUs at PCIE 4x, but allows for up to 9-10 GPUs (CPU thread limited) at reduced PCIE 1x speeds for some of the cards.
The Ryzen 7, I found, has a too high TDP (95W), and too many threads (16). Finding a motherboard that runs more than 7 GPUs is not efficient, nor realistic; so I won't choose this CPU for folding.

AMD CPUs have 32 PCIE lanes, and just like with Intel, only 16 PCIE lanes are directed towards PCIE slots, and good for up to 4x GPUs at the full PCIE 4x speed; 4 go to the chipset, which provides USB and Sata connections, etc...
But AMD also has an additional 4x PCIE lane for an m.2 slot, (This m.2 slot, with the right adapter can host a full size GPU at PCIE 4x speed, and in theory this is good for a fifth* GPU), where as Intel boards (being limited to only 16 lanes) will steal these PCIE lanes from the other GPU slots.

* Sadly, for the 5 GPU system, no manufacturer makes motherboards with 4x PCIE 4 lane slots, AND a 4x speed m.2 slot.
Nearly all motherboards provide the primary PCIE slot with 16 or 8 lanes. Not 4; which makes that the maximum you'll find is 3 GPUs at PCIE 4x or faster, PLUS one GPU fitted in the m.2 slot!
Any additional card beyond this 4 GPU setup, and some of the slots, as well as the extra slots, will fall back to (or are already at) PCIE 1x speed.


I said AMD has 32 PCIE slots, so the 8 missing PCIE lanes (16 go to PCIE slots, 4 to the chipset, and 4 go to the m.2 adapter) appear to be designated for the integrated graphics.
The Ryzen 5 (and Ryzen 3 1200/1300x) don't have the integrated graphics so the lanes are internally sealed, and supposedly inaccessible (unless someone can find the correct CPU pins and tap off the PCIE lanes from there).

So the benefit AMD offers over Intel is basically 4 PCIE lanes extra, on a 3 GPU system, or 1 extra GPU at PCIE 4x speed.
It's something, but maybe not enough to jump ship.


If you're looking to expand your Intel system beyond the 3 GPUs at PCIE 4x speed or higher, and want to eliminate a PCIE 1x speed bottleneck, perhaps it's good to wait for PCIE 4.0 (or even PCIE 5.0) to come out both on Motherboard and GPU; as PCIE 4.0 1x speed roughly equals PCIE 3.0 2x speed, and PCIE 5.0 roughly equals PCIE 3.0 4x speed.
And PCIE 3.0 2x speed for folding, is about where the bottleneck significantly reduces on RTX 2080 cards or lower in Linux.
PCIE 3.0 4x is still recommended for RTX 2080 ti cards in Linux.

Then again, if you'd run 4x RTX 2080 cards on a Linux based Ryzen 3 1200 or 1300x system, you'd probably be running THE most efficient system you could possibly run for folding, without major PCIE bottlenecks.

The only factor I haven't yet factored in, is that AMD motherboards might not provide sufficient power over their motherboard slots, to power 3 or 4 graphics cards with the 35W required to run a PCIE 4x slot at. Powered risers like these will be required!

Another one, is power consumption, and performance of a Ryzen 3 quadcore chip, VS a quadcore i3 or i5 Intel chip; and see if Ryzen offers the same performance as Intel in this matter, or if 2Ghz AMD CPUs will bottleneck a 2060, and 3Ghz a 2080 ti.
I hope some people with more experience can chime in. Perhaps compare results on Intel VS AMD CPUs, and how they affect folding and performance, as well as if AMD is as power efficient as Intel when it comes to folding?
Last edited by Theodore on Sat Jun 08, 2019 7:27 am, edited 7 times in total.
foldy
Posts: 2061
Joined: Sat Dec 01, 2012 3:43 pm
Hardware configuration: Folding@Home Client 7.6.13 (1 GPU slots)
Windows 7 64bit
Intel Core i5 2500k@4Ghz
Nvidia gtx 1080ti driver 441

Re: AMD CPU systems

Post by foldy »

On Windows you would even need pcie 3.0 x8 for fast GPUs to not get any bottleneck but x4 is still acceptable. On Linux you can run a gtx 1070 even on pcie 3.0 x1 riser without big bottleneck.
Theodore
Posts: 118
Joined: Sun Feb 10, 2019 2:07 pm

Re: AMD CPU systems

Post by Theodore »

Another thing I just thought of,
It's not sure if the PCIE 4x adapter from m.2 to PCIE, actually functions at PCIE 4x speeds, or only at PCIE 1x speeds.
If the adapters, which use a 4x slot, are actually working at 1x speed, there's no reason at all to switch over to AMD.
Nathan_P
Posts: 1180
Joined: Wed Apr 01, 2009 9:22 pm
Hardware configuration: Asus Z8NA D6C, 2 x5670@3.2 Ghz, , 12gb Ram, GTX 980ti, AX650 PSU, win 10 (daily use)

Asus Z87 WS, Xeon E3-1230L v3, 8gb ram, KFA GTX 1080, EVGA 750ti , AX760 PSU, Mint 18.2 OS

Not currently folding
Asus Z9PE- D8 WS, 2 E5-2665@2.3 Ghz, 16Gb 1.35v Ram, Ubuntu (Fold only)
Asus Z9PA, 2 Ivy 12 core, 16gb Ram, H folding appliance (fold only)
Location: Jersey, Channel islands

Re: AMD CPU systems

Post by Nathan_P »

Theodore wrote:Another thing I just thought of,
It's not sure if the PCIE 4x adapter from m.2 to PCIE, actually functions at PCIE 4x speeds, or only at PCIE 1x speeds.
If the adapters, which use a 4x slot, are actually working at 1x speed, there's no reason at all to switch over to AMD.
If you do use an adapter and only get x1 speed, the problem is with the adapter and not the m.2 slot/mobo
Image
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: AMD CPU systems

Post by bruce »

This riser will work at 4x as long as the slot it's used in is 4x. Narrower risers are 1x, no matter what slot they're connected to.
JimboPalmer
Posts: 2573
Joined: Mon Feb 16, 2009 4:12 am
Location: Greenwood MS USA

Re: AMD CPU systems

Post by JimboPalmer »

[This is wild speculation, not a promise. It is the truth as I understand it, but I have a tenuous grip on reality]

PCI-E 4.0 has twice the bandwidth of PCI-E 3.0. This could mean that 4.0 2x would be as fast as 3.0 4x. (Obviously both the motherboard and GPU must support PCI-E 4.0 as well as the CPU.

The additional bandwidth COULD improve the GPU utilization, if less time is spent loading WU data on the card, more time can be executing the WU.

OR it could support more GPU cards with similar throughput to half as many 3.0 GPUs.

{again Ryzen 2 supports PCI-E 4.0 on the CPU, existing motherboard may support a single PCI-E 4.0 slot, new motherboards would be needed for multiple 4.0 slots. And then AMD or Nvidia needs to make cards running at PCI-E 4.0}
Tsar of all the Rushers
I tried to remain childlike, all I achieved was childish.
A friend to those who want no friends
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: AMD CPU systems

Post by bruce »

@ JimboPalmer

My speculation agrees with your speculation. Moving the same data along parallel wires or using fewer wires at a higher clock rate can get it there at the same time. The only reason that faster GPUs need more PCIe bandwidth is that if they're fast enough to run out of data before it has received the next block of data, it will simply wait with nothing to do for as long as is necessary.

That's one advantage of folding with a CPU ... the data path from main ram is a lot faster that if it has to go through a relavely slow data bus.
antropofob
Posts: 59
Joined: Mon Aug 22, 2011 8:03 am

Re: AMD CPU systems

Post by antropofob »

bruce wrote:This riser will work at 4x as long as the slot it's used in is 4x. Narrower risers are 1x, no matter what slot they're connected to.
How come those thin ribbon wires used do not melt while transferring 75W PCIE power?
foldy
Posts: 2061
Joined: Sat Dec 01, 2012 3:43 pm
Hardware configuration: Folding@Home Client 7.6.13 (1 GPU slots)
Windows 7 64bit
Intel Core i5 2500k@4Ghz
Nvidia gtx 1080ti driver 441

Re: AMD CPU systems

Post by foldy »

That is not a good riser, it should always be a powered riser.
https://www.moddiy.com/products/PCI%252 ... rrencyId=1
MeeLee
Posts: 1375
Joined: Tue Feb 19, 2019 10:16 pm

Re: AMD CPU systems

Post by MeeLee »

I'm speculating that the same thing would happen as with USB 2.0 to 3.0.
When USB 2.0 devices are ran on USB 3.0 ports, they work faster(17MB/s on 3.0 vs 12MB/s on 2.0).

PCIE 4.0 motherboards might alleviate some of the bandwidth constrains, like overhead, even if only ran at 3.0 speeds.
Meaning, a PCIE 4.0 1x slot, would work faster than a PCIE 3.0 1x slot, even when both are ran at 3.0 speeds; but not as fast as at full 4.0 speeds.
At least, that's what I'm hoping.

I'm not sure if motherboard manufacturers will only enable PCIE 4.0 speeds on one slot, and leaving the other slots 3.0.
It would make little sense, and also, most motherboards work in a one type configuration; meaning, if set to 3.0, all slots will run 3.0.
If set to 2.0, all slots will fall back to 2.0.
There is a possibility that only one slot will be offered at 4.0 speeds, especially in the beginning, but the chances of that happening is rather low.
It would suck for early adopters...

Pcie 5.0 is already defined. It would surprise me if they just don't skip 4.0 and go straight to 5.0.



I would find it interesting to see if 6 to 8 core CPUs will not be able to run 6 to 8 GPUs 'unrestricted'; meaning, they'd be able to run quite close to their maximum rated PPDs, from a PCIE 4.0 1x port.
Theodore
Posts: 118
Joined: Sun Feb 10, 2019 2:07 pm

Re: AMD CPU systems

Post by Theodore »

We'll know soon enough.
AMD will release the new Ryzen chips with pcie 4.0 support by 7/7.
They will also update their GPU line, which has been mentioned to have 50% better efficiency than Vega, and supports pcie 4.0.
It will make their cards more competitive with NVidia for folding.

50 something new motherboards will support the new CPU, though not much is known about which boards will also support pcie 4.0.


https://www.theverge.com/platform/amp/c ... date-specs
Nathan_P
Posts: 1180
Joined: Wed Apr 01, 2009 9:22 pm
Hardware configuration: Asus Z8NA D6C, 2 x5670@3.2 Ghz, , 12gb Ram, GTX 980ti, AX650 PSU, win 10 (daily use)

Asus Z87 WS, Xeon E3-1230L v3, 8gb ram, KFA GTX 1080, EVGA 750ti , AX760 PSU, Mint 18.2 OS

Not currently folding
Asus Z9PE- D8 WS, 2 E5-2665@2.3 Ghz, 16Gb 1.35v Ram, Ubuntu (Fold only)
Asus Z9PA, 2 Ivy 12 core, 16gb Ram, H folding appliance (fold only)
Location: Jersey, Channel islands

Re: AMD CPU systems

Post by Nathan_P »

PCIe 5.0 is supposed to be for the server market, don't expect it in the consumer space anytime soon.
Image
MeeLee
Posts: 1375
Joined: Tue Feb 19, 2019 10:16 pm

Re: AMD CPU systems

Post by MeeLee »

in this video (at timeframe 2:21), they're showcasing one of the motherboards supporting the Ryzen 3.
It appears that it has at least 3x full size PCIE 4.0 slots built in.

https://youtu.be/TkSTp_tJI2o?t=141
kiore
Posts: 931
Joined: Fri Jan 16, 2009 5:45 pm
Location: USA

Re: AMD CPU systems

Post by kiore »

The news looks very promising for the new generation Ryzen with the 3700x looking a very sweet spot. Still 5 weeks till release and exposure to real world conditions. I have had my hopes dashed previously (did anyone say Bulldozer?) but for Folding a 16 thread CPU with a real TDP of 65watts with good GPU support can't be bad news.
Image
i7 7800x RTX 3070 OS= win10. AMD 3700x RTX 2080ti OS= win10 .

Team page: http://www.rationalskepticism.org
JimF
Posts: 652
Joined: Thu Jan 21, 2010 2:03 pm

Re: AMD CPU systems

Post by JimF »

kiore wrote:I have had my hopes dashed previously (did anyone say Bulldozer?) but for Folding a 16 thread CPU with a real TDP of 65watts with good GPU support can't be bad news.
I hope so. But my Ryzen 2700 (16 virtual cores) did not do nearly as well as my i7-8700 (12 virtual cores), both on Ubuntu 18.04, as I have posted elsewhere. I suspect something in the CPU architecture favors Intel. But if it is a compiler difference, maybe it can be fixed.

On most other projects (WCG, etc.) the Ryzen does as well, or sometimes better than the Intel.
Post Reply