PCIE x speeds for F@H gpu crunching

If you think it might be a driver problem, see viewforum.php?f=79

Moderators: Site Moderators, FAHC Science Team

rickoic
Posts: 322
Joined: Sat May 23, 2009 4:49 pm
Hardware configuration: eVga x299 DARK 2070 Super, eVGA 2080, eVga 1070, eVga 2080 Super
MSI x399 eVga 2080, eVga 1070, eVga 1070, GT970
Location: Mississippi near Memphis, Tn

Re: PCIE x speeds for F@H gpu crunching

Post by rickoic »

After reading your suggestions, and they were all helpful in one way or another. My decision is this:
I like the idea of using a mining frame. With an adaptor I can use 2 power supplies as the frame is designed for that (some have the adaptor already installed).
With ribbon risers I can install my gpu's with plenty of apace between them for better cooling. As well as having a fan far to increase air flow.
Easy to see any dust buildup and whisk it away with a small broom or compressed air.
And the frame is cheaper than a regular case.

Tks to all for the help and advice.
Rick
I'm folding because Dec 2005 I had radical prostate surgery.
Lost brother to spinal cancer, brother-in-law to prostate cancer.
Several 1st cousins lost and a few who have survived.
rickoic
Posts: 322
Joined: Sat May 23, 2009 4:49 pm
Hardware configuration: eVga x299 DARK 2070 Super, eVGA 2080, eVga 1070, eVga 2080 Super
MSI x399 eVga 2080, eVga 1070, eVga 1070, GT970
Location: Mississippi near Memphis, Tn

Re: PCIE x speeds for F@H gpu crunching

Post by rickoic »

OK, been too long and I haven't been able to find the instructions on how to get the NVIDIA drivers installed.. As well as the FAHClient install, and what are the changes needed for FAHDisplay?

Got UBUNTU 20.04 installed and running.

Much TKS
Rick
I'm folding because Dec 2005 I had radical prostate surgery.
Lost brother to spinal cancer, brother-in-law to prostate cancer.
Several 1st cousins lost and a few who have survived.
PantherX
Site Moderator
Posts: 7020
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: PCIE x speeds for F@H gpu crunching

Post by PantherX »

This is what I have read so far:
1) Ensure that Ubuntu is fully updated.
2) Install the proprietary drivers by Nvidia
3) Install OpenCL package (sudo apt-get install ocl-icd-opencl-dev)
4) Install FAHClient

If you mean FAHViewer or FAHControl instead of FAHDisplay, I am aware that FAHControl has python 2 dependencies and that you may need to modify a file to get that sorted out (someone with more Linux experience will advise you).
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
rickoic
Posts: 322
Joined: Sat May 23, 2009 4:49 pm
Hardware configuration: eVga x299 DARK 2070 Super, eVGA 2080, eVga 1070, eVga 2080 Super
MSI x399 eVga 2080, eVga 1070, eVga 1070, GT970
Location: Mississippi near Memphis, Tn

Re: PCIE x speeds for F@H gpu crunching

Post by rickoic »

As I said its been a full 10 years or more since I've played with Linux. So step by step instructions would really be a BIG help.

I tried the Install OpenCL instruction and got Unable to locate package...
I have downloaded NVIDIA-Linux-x86_64-440.82.run in my Downloads folder but what are the instructions to install it?

Much thanks
Rick
I'm folding because Dec 2005 I had radical prostate surgery.
Lost brother to spinal cancer, brother-in-law to prostate cancer.
Several 1st cousins lost and a few who have survived.
ajm
Posts: 754
Joined: Sat Mar 21, 2020 5:22 am
Location: Lucerne, Switzerland

Re: PCIE x speeds for F@H gpu crunching

Post by ajm »

There is a good how-to here: https://linustechtips.com/main/topic/10 ... lding-rig/

Now, this how-to is for Ubuntu 18.04. For 20.4 and the python 2 problem (only concerns FAHControl, so not absolutely necessary), you may want to try this: viewtopic.php?p=321446#p321446
MeeLee
Posts: 1375
Joined: Tue Feb 19, 2019 10:16 pm

Re: PCIE x speeds for F@H gpu crunching

Post by MeeLee »

Nvidia updates it's drivers all the time.
Installation procedures change by the minute.
My last known procedure was the following:
After downloading the .run files, make the file executable (rightclick menu, or do chmod +x NvidiaDriver.run).
then reboot, go into recovery mode (grub, advanced boot, boot into terminal).
I usually do:
sudo init 3
sudo service sddm stop
(sddm can be gddm3 in your case, just look up how to disable the desktop manager of your distribution).
Then go to the nvidiaDriver.run folder.
Execute by:
sudo ./NVidiaDriver.run
(with the appropriate Nvidia driver file there)
Aknowledge/overwrite everything if the popups ask anything.
Some drivers crash at the 32 bit stage.
You don't need to install the 32 bit libraries.
It saves installation trouble, and isn't really needed anyway.

then reboot.

Your PC can also have fan curve, power settings and over/under-clocking options enabled by doing:
sudo nvidia-xconfig --cool-bits=28
Then reboot.
Do this only if you have 1 Nvidia GPU

If you have multiple Nvidia GPUs, it's strongly recommended to run 18.04 instead of 20.04, and input the next commands instead:
sudo nvidia-xconfig --enable-all-gpus
sudo nvidia-xconfig --cool-bits=28

Last I remember, the above 2 lines will crash the desktop on 20.04.
Not sure if the crash is limited to SDDM (as 18.04 uses LightDM), and if Ubuntu 19.04 and up (gddm2/gddm3) is affected by it or not.
You can give it a try. I just haven't successfully found a way to undo the desktop crashes '--enable-all-gpus' causes on SDDM, other than a complete uninstall of the drivers (or OS).
Joe_H
Site Admin
Posts: 7856
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: PCIE x speeds for F@H gpu crunching

Post by Joe_H »

rickoic wrote:I have downloaded NVIDIA-Linux-x86_64-440.82.run in my Downloads folder but what are the instructions to install it?
For most Linux distributions you can connect to a repository that will have the nVidia drivers packaged up for proper installation on that version of Linux. That is usually easier than installing from the .run file provided for download from nVidia.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
The_Bad_Penguin
Posts: 19
Joined: Mon Jan 07, 2008 12:06 am

Re: PCIE x speeds for F@H gpu crunching

Post by The_Bad_Penguin »

Just as a follow-up / FYI.

Don't know what the policy is about putting in links, so I will just note that I came across a new (June 16, yesterday) YouTube video entitled "TWO Graphics Cards in ONE Slot?!" that discusses essentially pcie bifurcation.

You can search under that title or the content creator, "Techquickie".

FWIW.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: PCIE x speeds for F@H gpu crunching

Post by bruce »

Links are prohibited until you've used the forum for a while.
MeeLee
Posts: 1375
Joined: Tue Feb 19, 2019 10:16 pm

Re: PCIE x speeds for F@H gpu crunching

Post by MeeLee »

I've only once tried a $200 PCIE x16 to 4x x4 slots, and it wasn't supported by the bios.
The other time, I tried a PCIE x1 splitter to 4x USB (for USB risers) and even then it wasn't supported (PCIE error from bios).
The only time it worked, was when one of my motherboards told me there are more PCIE lanes than the CPU could supply (I believe it was an MSI MPG board).
And it wouldn't have made sense either, even on core 21, to split an x1 slot in anything.
PCIE 3.0 x1 was barely enough for core 21. Anything slower would not have worked well...Let alone on core 22.
The_Bad_Penguin
Posts: 19
Joined: Mon Jan 07, 2008 12:06 am

Re: PCIE x speeds for F@H gpu crunching

Post by The_Bad_Penguin »

I have a tangential question, and figured it was close enough to this thread not to warrant a new thread.

What is really the maximum number of gpu's you can run off a motherboard?

For this question, ignore power consumption, physical spacing, etc.

Just concern ourselves with PCIE lanes.

I understand that a motherboard's chip set determines the number of available PCIE lanes, e.g., 16, 36, 40.

So, if the chipset has 40 lanes, and has 3 pcie-x16's (electrically), that alone is 48 lanes, without taking into account any other peripherals wanting PCIE lanes.

What happens then?

Can you F@H crunch with 48 lanes on a 40 lane chipset?

Are there chipsets which will allow for more than 40 PCIE lanes? A motherboard for AMD threadripper 39xx cpu's perhaps?
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: PCIE x speeds for F@H gpu crunching

Post by bruce »

FAHClient allows you to configure 10 GPUs numbered 0 through 9.
Joe_H
Site Admin
Posts: 7856
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: PCIE x speeds for F@H gpu crunching

Post by Joe_H »

For F@h, the answer is 10. The client can enumerate from 0-9, it currently will not utilize more than 10 GPUs.

As for the rest, it will be up to the logic board and its chipset. The lanes would need to be coming from a combination of the ones supported directly by the processor and others supplied by one or more bridge chips. In the example you propose, if there are 40 lanes and three 16x slots at least one of them will be running at 8x or less.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
The_Bad_Penguin
Posts: 19
Joined: Mon Jan 07, 2008 12:06 am

Re: PCIE x speeds for F@H gpu crunching

Post by The_Bad_Penguin »

OK, so let me see if I am correctly understanding.

For the sake of this example, let us assume an ASUS P9X79-E WS motherboard has an Intel X79 chipset with 48 lanes of PCIE. Let us also assume that the motherboard also has 7 PCIE slots: 4 are PCIE-x16 (electrically) and 3 are PCIE-x8 (electrically). Let us assume (using risers) we are able to put modern gpu's in each of the 7 slots. By my math, that is 4x16lanes + 3x8lanes; 64 lanes + 24 lanes = 88 lanes. 88 > 48, thus all the x16's would run at x8, and all of the x8's would run at x4 (32+12 = 44), So 44 out of 48 lanes would be used by the 7 gpu's, each running at 50% of their pcie-x electrical abilities, with 4 pcie lanes left over for other system use.

I think I saw that the chipset for current AMD ThreadRipper motherboards is TRX40, and does have 88 PCIE lanes.

Have to look to see if what sTRX4 motherboards offer as far as maximum number of physical slots of pcie-x16 (electrical) and pcie-x8 (electrical).

If I understand everything, if such a motherboard exists with 5 pcie-x16 electrical slots (and 88 pcie lanes), it could gpu F@H crunch with no downgrading / degradation.

EDIT --> Apparently there are only 72 usable of the 88 total pcie lanes.
"... because we’re also increasing the CPU<->chipset link from 4x Gen4 to 8x Gen4—quadruple the bandwidth vs. 2nd Gen TR. Extra data pins between the chipset and CPU make this possible, so you’ll be able to hang more I/O off the motherboard at full performance.”
Last edited by The_Bad_Penguin on Sat Jun 27, 2020 8:15 pm, edited 1 time in total.
ajm
Posts: 754
Joined: Sat Mar 21, 2020 5:22 am
Location: Lucerne, Switzerland

Re: PCIE x speeds for F@H gpu crunching

Post by ajm »

It's not that "simple". The manufacturers only give general indications and how the board will split the lanes when they are too scarce can be a bit of a mystery.
Gigabyte propose an online system for that, where you can choose your MB, your CPU and how you intend to populate the PCI slots (and shared M.2 slots) - the system then tells you exactly where you'll get what: https://www.gigabyte.com/FileUpload/glo ... /x299.html
But I don't know for other manufacturers. If you don't find such a system, it would be safer to ask them directly before the purchase.
Post Reply