PCI-e bandwidth/capacity limitations

A forum for discussing FAH-related hardware choices and info on actual products (not speculation).

Moderator: Site Moderators

Forum rules
Please read the forum rules before posting.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: PCI-e splitter?

Post by bruce »

Well, I've got a few more GPUs than slots so maybe I'll try running something on x1 and see if it can do ANYTHING USEFUL Clearly PCIe will be the limiting factor, not GPU throughput.

(I've never been good about selling stuff that has been replaced by an upgrade.)

Why not?
Nathan_P
Posts: 1180
Joined: Wed Apr 01, 2009 9:22 pm
Hardware configuration: Asus Z8NA D6C, 2 x5670@3.2 Ghz, , 12gb Ram, GTX 980ti, AX650 PSU, win 10 (daily use)

Asus Z87 WS, Xeon E3-1230L v3, 8gb ram, KFA GTX 1080, EVGA 750ti , AX760 PSU, Mint 18.2 OS

Not currently folding
Asus Z9PE- D8 WS, 2 E5-2665@2.3 Ghz, 16Gb 1.35v Ram, Ubuntu (Fold only)
Asus Z9PA, 2 Ivy 12 core, 16gb Ram, H folding appliance (fold only)
Location: Jersey, Channel islands

Re: PCI-e splitter?

Post by Nathan_P »

OK, 1st test on an actual WU - haven't had a chance to do any proper testing as win7 won't install properly and windows is a lot easier to run a bunch of WU tests on the Linux. This is taken from my hfm logs on my main folding machine running the zorin 9 distro.

PCIe 2.0 @ x16 project 10490 tpf 2.40 to 2.42 on a GTX 970. PCIe 3.0 @ x16, project 10490 tpf 2.40 - no change between PCIe 2.0 and 3.0 x16 at this time - this bears out the gaming performance speeds found elsewehere.

PCIe 2.0 @ x8 project 9151 tpf 1.13 to 1.26 on a GTX 980ti, PCIe 3.0 x 16 project 9151 1.07 - 6 second change between current and previous recorded minimum time - this takes ppd from 422k PPD to 479k PPD. Again the gaming performance speeds do show a drop off at PCIe 2.0 x8

As projects run through over the next couple of days i'll post more results and try some PCIe 1.1 tests but early indications are that PCIe 2.0 x8 and below do affect performance and the faster the card the bigger the hit.
Image
artoar_11
Posts: 657
Joined: Sun Nov 22, 2009 8:42 pm
Hardware configuration: AMD R7 3700X @ 4.0 GHz; ASUS ROG STRIX X470-F GAMING; DDR4 2x8GB @ 3.0 GHz; GByte RTX 3060 Ti @ 1890 MHz; Fortron-550W 80+ bronze; Win10 Pro/64
Location: Bulgaria/Team #224497/artoar11_ALL_....

Re: PCI-e splitter?

Post by artoar_11 »

I'm not sure if the games and FAH scaled PCIe xx transfer equally. Just for comparison:

Image
Source: http://www.techpowerup.com/reviews/AMD/ ... ng/18.html
foldy
Posts: 2061
Joined: Sat Dec 01, 2012 3:43 pm
Hardware configuration: Folding@Home Client 7.6.13 (1 GPU slots)
Windows 7 64bit
Intel Core i5 2500k@4Ghz
Nvidia gtx 1080ti driver 441

Re: PCI-e splitter?

Post by foldy »

We expect FAH to have a major performance hit like 50% on lower than pcie 2.0 x8 with fastest GPUs because of the high bus bandwidth usage.
Other than this games relative performance summary which has only 11% performance hit even on pcie 1.1 x4
foldy
Posts: 2061
Joined: Sat Dec 01, 2012 3:43 pm
Hardware configuration: Folding@Home Client 7.6.13 (1 GPU slots)
Windows 7 64bit
Intel Core i5 2500k@4Ghz
Nvidia gtx 1080ti driver 441

Re: PCI-e splitter?

Post by foldy »

@Nathan_P: Maybe you want to try this linux tool to monitor pcie bus usage
https://github.com/mountassir/gMonitor

So you can see what bus usage your GTX 980ti hit at PCIe 2.0 @ x8.
I guess around 80% which will already be high enough to increase work unit data transfer latency and so increase TPF and reduce PPD.

The nvidia-settings command used in this tool for pcie bus usage
"nvidia-settings -t -q " << gpuId << "/GPUUtilization | tr ',' '\n' |grep PCIe|sed 's/[^0-9]//g'";
Nathan_P
Posts: 1180
Joined: Wed Apr 01, 2009 9:22 pm
Hardware configuration: Asus Z8NA D6C, 2 x5670@3.2 Ghz, , 12gb Ram, GTX 980ti, AX650 PSU, win 10 (daily use)

Asus Z87 WS, Xeon E3-1230L v3, 8gb ram, KFA GTX 1080, EVGA 750ti , AX760 PSU, Mint 18.2 OS

Not currently folding
Asus Z9PE- D8 WS, 2 E5-2665@2.3 Ghz, 16Gb 1.35v Ram, Ubuntu (Fold only)
Asus Z9PA, 2 Ivy 12 core, 16gb Ram, H folding appliance (fold only)
Location: Jersey, Channel islands

Re: PCI-e splitter?

Post by Nathan_P »

I'm going to measure more projects from HFM overnight, a couple of results do not make conclusive results and a couple of the WU I've truned in today don't have any data from the old rig, 970 is staying consistent with previous results though.

More Linux testing Thursday, Linux isn't my thing - it took me an hour to get samba working yesterday.
Image
hiigaran
Posts: 134
Joined: Thu Nov 17, 2011 6:01 pm

Re: PCI-e splitter?

Post by hiigaran »

While more data is being gathered, I'd like to sidetrack a little. Though my original intention was to find a solution to PCI-e capacity issues on a motherboards, these initial results heavily suggest that it may be impossible to use consumer-grade hardware to create a single, or even dual system with high GPU capacity. At least, not without a severe reduction in output. However, since 3.0 x8 equivalent bandwidth seems to be a viable option, I'd like to completely forget about any possibility of splitters, and instead look at the possibility of x16 risers.

Now I am fully aware of their existence. However, every flexible x16 riser I have seen to far is a bulky ribbon cable with limited length. Comparing this to the flexible x1 risers using a USB3.0 cable, which offers much more freedom, and length. Has anyone ever attempted a cross between the two? A x16 capacity riser with the flexibility of the x1s? If not, what's the longest x16 riser anyone has seen?
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: PCI-e splitter?

Post by bruce »

I was wondering that myself.

Two thoughts:

* A motherboard with 20 lanes can drive a x16 slot and a x1 slot. If you want to use the x1 slot, you're limited to the x1 bandwidth.

* Will the bandwidth of your extended x16 (V-whatever) slot be limited by the bandwidth of USB?
Nathan_P
Posts: 1180
Joined: Wed Apr 01, 2009 9:22 pm
Hardware configuration: Asus Z8NA D6C, 2 x5670@3.2 Ghz, , 12gb Ram, GTX 980ti, AX650 PSU, win 10 (daily use)

Asus Z87 WS, Xeon E3-1230L v3, 8gb ram, KFA GTX 1080, EVGA 750ti , AX760 PSU, Mint 18.2 OS

Not currently folding
Asus Z9PE- D8 WS, 2 E5-2665@2.3 Ghz, 16Gb 1.35v Ram, Ubuntu (Fold only)
Asus Z9PA, 2 Ivy 12 core, 16gb Ram, H folding appliance (fold only)
Location: Jersey, Channel islands

Re: PCI-e splitter?

Post by Nathan_P »

hiigaran wrote:While more data is being gathered, I'd like to sidetrack a little. Though my original intention was to find a solution to PCI-e capacity issues on a motherboards, these initial results heavily suggest that it may be impossible to use consumer-grade hardware to create a single, or even dual system with high GPU capacity. At least, not without a severe reduction in output. However, since 3.0 x8 equivalent bandwidth seems to be a viable option, I'd like to completely forget about any possibility of splitters, and instead look at the possibility of x16 risers.

Now I am fully aware of their existence. However, every flexible x16 riser I have seen to far is a bulky ribbon cable with limited length. Comparing this to the flexible x1 risers using a USB3.0 cable, which offers much more freedom, and length. Has anyone ever attempted a cross between the two? A x16 capacity riser with the flexibility of the x1s? If not, what's the longest x16 riser anyone has seen?
The best consumer, (lga 1150/1151) boards will have 4 x16 slots that run at x8, preferably PCIe3 to avoid any loss - even then you are limited to top end offerings, I'm running all these tests on an Asus Z87-WS. 2011 gives you more flexibility but I haven't researched that area. The boards with the most slots and expansion potential that I know of are the Z9PE-D8 and newer Z10PE-D8, even then the 7 x16 slots will only run 2 at x16 and 5 at x8.

From memory the longest x16 riser I have seen is about 6 inches, IIRC something to do with signal loss if they go any longer
Image
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: PCI-e splitter?

Post by bruce »

In some cases, PCIe x1 is going to work but it's probably wrong to recommend it for folks with fast GPUs.

Environment tested:
GPU: GT 740 (Fermi)
Linux
FAHCore_21 (Project 11414)
PCIe v2.0

Results:
x16 utilization: 14%
TPF: 48:36
PPD: 11092

x1 utilization: 19%
TPF: 48:47
PPD: 11039

In this particular case, the 1x riser keeps the GPU at 100%. at least during the routine analysis portions of the WU. It might or might not show different results during startup/finish/checkpointing/etc. but that's a small portion of the run.

Environment tested (as above except):
GPU: GTX 980 (Maxwell)
FAHCore_18 (Project 9160)

Results:
x16 utilization: 1%
GPU utilization: 99-100%
TPF: 2:26
PPD: 138877

x1 utilization:
TPF:
PPD:
hiigaran
Posts: 134
Joined: Thu Nov 17, 2011 6:01 pm

Re: PCI-e splitter?

Post by hiigaran »

The splitter? Wait, did you actually get your hands on one? Or is this something else?
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: PCI-e splitter?

Post by bruce »

I have two different brands which convert an x1 PCIe slot to USB port and convert a USB connection to an x16 slot ... with an added power connection.

Installing them was less than straightforward but I was able to fix all the problems I encountered and got my second GPU to run over an x1 connection. I tried it with the GPU to winch my monitor was attached and so far have been unable to do that. (The next thing I'm going to try is reenabling my Monitor to the iGPU first, then moving attaching the GPU -- sort of like using the GPU in PhysX mode)

Next on the agenda is to connect a full-height GPU to a half-height system -- thought that might also be a good application for a x16 riser cable.
hiigaran
Posts: 134
Joined: Thu Nov 17, 2011 6:01 pm

Re: PCI-e splitter?

Post by hiigaran »

I have two different brands which convert an x1 PCIe slot to USB port and convert a USB connection to an x16 slot ... with an added power connection.
Did you mean one brand that is x1 to USB, and another brand that is USB to x16, or did you mean two brands of the same thing? If the latter, is that x1 -> USB -> x16, and if so, isn't that just the flexible risers that are used in bitcoin farming?

Or is it something completely different, and indicative of my lack of sleep?
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: PCI-e splitter?

Post by bruce »

Each one is x1 > USB > x16

See the picture (more or less) in the 3rd post on page 1 of this topic and a lot of discussion in between.
hiigaran
Posts: 134
Joined: Thu Nov 17, 2011 6:01 pm

Re: PCI-e splitter?

Post by hiigaran »

Right, so it's a riser. This is where my confusion was stemming from.

Moving on, looking at your initial data for the 980 bandwidth, it shows only 1% usage of x16. Based on that it sounds like x1 would work with only 16% of max bandwidth. With wildly varying data, could it be that different projects have different bandwidth requirements?
Post Reply