Linux folding with multiple Nvidia GPUs

Moderators: Site Moderators, FAHC Science Team

Re: Linux folding with multiple Nvidia GPUs

Postby hiigaran » Tue Feb 06, 2018 10:29 pm

I saw nothing when I clicked on my name. Maybe something I could have done via the terminal?

Either way, I'm currently cloning. Just stuck on the final part with grub-install, but I'll be out of the country for work for a week, so I won't be able to do anything more with it until I get back.
User avatar
hiigaran
 
Posts: 134
Joined: Thu Nov 17, 2011 7:01 pm

Re: Linux folding with multiple Nvidia GPUs

Postby stealthswifttlk » Mon Mar 19, 2018 10:09 am

Did you ever get it working with the cloned software? If not then might I hazard a suggestion that it isn't a display problem that you are having per-se but perhaps that you are having a PCIe bus problem? The only thing that you have changed between a working configuration and a non-working config in your circumstances is the installation of the M.2 card, where as the SATA drive works fine... From my own experiences, M2 cards have different PCIe bus routings, especially to enable faster performing NVMe drives... The fact that this problem is manifesting itself across multiple OSs, multiple kernels and even the same OS config that you have from your SATA drive suggests that this may be the case.

Two questions to check this theory: What UEFI/Bios settings are you changing between SATA and M2? (is it literally just this, or are you changing AHCI mode etc too?) This question has a bearing - if you are keeping the same mode (Your SATA should be set at AHCI for optimal performance of it) Can you run your M2 card in AHCI and if so, can you clone your SATA drive raw contents to the M2 drive using something like clonezilla (didn't appear you were doing a sector-for-sector copy as you were having to mess about with the grub config?) If you then try to boot the exact same known working configuration from the M2 and it still isn't working, you've worked towards resolving that it isn't a software configuration issue and something else is at play here. That only really leaves the PCIe bus itself and it's M2 entry point, or the M2 card itself perhaps having some sort of defect

I'd perhaps also test by pulling all the cards and gradually reintroducing them on different PCIe slots until you get a failure - might reinforce suggestion of conflict.

tl:dr: I think you've run out of PCIe lanes - the M2 card is probably conflicting with one of (perhaps your primary 1x16 Gen3) or multiple graphics cards on the PCIe bus... M2 cards share PCIe bandwidth. Clue to my thinking here was in your post that gave the error message that it had fallen off the bus.
stealthswifttlk
 
Posts: 14
Joined: Fri Aug 29, 2008 10:01 pm
Location: Sheffield, UK

Re: Linux folding with multiple Nvidia GPUs

Postby hiigaran » Tue Apr 24, 2018 9:29 pm

Completely forgot about this thread.

Yes. I did manage to get CloneZilla to work after resizing partitions from the original hard drive. Both rigs worked successfully. No reduction in performance from the PCI lanes either.

I did manage to recently mess up one of my rigs, though. Ran into issues after an update and reboot, but I can't figure out the cause. Either way, I'm going to attempt once again to perform a clean install once Ubuntu 18.04 is released in the next couple of days. I will likely use this thread for any possible (probable?) issues that may arise during the setup. If I can't fix it, I'll just give BOINC some love, I guess.

By the way, is fahclient still v7.4.4, or is there something else in the works? Beta software, perhaps? Might be worth looking into if there are additional problems.
User avatar
hiigaran
 
Posts: 134
Joined: Thu Nov 17, 2011 7:01 pm

Re: Linux folding with multiple Nvidia GPUs

Postby toTOW » Tue Apr 24, 2018 10:21 pm

There is 7.4.16 in beta, you can find it in this part of the forum : viewforum.php?f=104
Folding@Home beta tester since 2002. Folding Forum moderator since July 2008.

FAH-Addict : latest news, tests and reviews about Folding@Home project.

Image
User avatar
toTOW
Site Moderator
 
Posts: 5651
Joined: Sun Dec 02, 2007 11:38 am
Location: Bordeaux, France

Re: Linux folding with multiple Nvidia GPUs

Postby hiigaran » Sat Apr 28, 2018 1:38 am

Cheers.

With the release of 18.04, getting F@H running with nVidia GPUs was ridiculously easy. From a clean install:

Code: Select all

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt full-upgrade
sudo apt install nvidia-384 nvidia-cuda-toolkit
wget https://folding.stanford.edu/releases/beta/release/fahclient/debian-stable-64bit/v7.4/fahclient_7.4.16_amd64.deb
sudo dpkg -i fahclient_7.4.16_amd64.deb
sudo wget http://fah-web.stanford.edu/file-releases/public/GPUs.txt -P /var/lib/fahclient



After that, just your standard config.xml stuff, and any fan controller scripts.
User avatar
hiigaran
 
Posts: 134
Joined: Thu Nov 17, 2011 7:01 pm

Re: Linux folding with multiple Nvidia GPUs

Postby hiigaran » Sun Apr 29, 2018 9:09 pm

So I decided to run FAHClient --help and one option caught my eye. There is an option for gpu-usage with a user-defined percentage value that can be set. Does this actually work? I thought GPUs basically folded in an all or nothing manner.

EDIT: Almost forgot. I was looking through the help to try and find a way to pause GPU folding without having to use FAHControl. Is that possible, or is my option option sudo /etc/init.d/FAHClient stop and sudo /etc/init.d/FAHClient start?

EDIT2: Was looking in the wrong area. --send-pause [slot] was what I needed.
User avatar
hiigaran
 
Posts: 134
Joined: Thu Nov 17, 2011 7:01 pm

Re: Linux folding with multiple Nvidia GPUs

Postby Joe_H » Mon Apr 30, 2018 12:33 am

hiigaran wrote:So I decided to run FAHClient --help and one option caught my eye. There is an option for gpu-usage with a user-defined percentage value that can be set. Does this actually work? I thought GPUs basically folded in an all or nothing manner.

This is a legacy setting from an experiment in controlling GPU usage that was in one of the very early beta test versions of the Version 7 client. I don't recall the details from half a dozen or so years ago, but do recall that it did not work very well.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Joe_H
Site Admin
 
Posts: 6613
Joined: Tue Apr 21, 2009 5:41 pm
Location: W. MA

Re: Linux folding with multiple Nvidia GPUs

Postby hiigaran » Mon Apr 30, 2018 12:51 am

Pity. It's probably the biggest hindrance to GPU folding for many. To my understanding, it's more a problem on the driver side of things, isn't it? Not much can be done from the F@H dev's end.

Also, I've noticed that GPU folding on Ubuntu seems to take its toll more than it does in Windows 10 on the same hardware. Even with hardware acceleration disabled in Firefox, there's still noticeable lag in videos, animated gifs, and even just using the autoscrolling function. I wonder why...
User avatar
hiigaran
 
Posts: 134
Joined: Thu Nov 17, 2011 7:01 pm

Re: Linux folding with multiple Nvidia GPUs

Postby toTOW » Mon Apr 30, 2018 10:59 am

hiigaran wrote:Pity. It's probably the biggest hindrance to GPU folding for many. To my understanding, it's more a problem on the driver side of things, isn't it? Not much can be done from the F@H dev's end.

Actually, I think that it would be more harmful to the hardware, because the solution was to cycle load and idle phases triggered by software, which would stress more the components than continuous load ...

hiigaran wrote:Also, I've noticed that GPU folding on Ubuntu seems to take its toll more than it does in Windows 10 on the same hardware. Even with hardware acceleration disabled in Firefox, there's still noticeable lag in videos, animated gifs, and even just using the autoscrolling function. I wonder why...

The answer is simple : different drivers models. In Windows, WDDM (Windows Display Driver Model) flavours user experience over compute performance ... thank Microsoft for that choice :( ...
User avatar
toTOW
Site Moderator
 
Posts: 5651
Joined: Sun Dec 02, 2007 11:38 am
Location: Bordeaux, France

Re: Linux folding with multiple Nvidia GPUs

Postby SteveWillis » Mon Apr 30, 2018 11:24 am

hiigaran wrote:Pity. It's probably the biggest hindrance to GPU folding for many. To my understanding, it's more a problem on the driver side of things, isn't it? Not much can be done from the F@H dev's end.

Also, I've noticed that GPU folding on Ubuntu seems to take its toll more than it does in Windows 10 on the same hardware. Even with hardware acceleration disabled in Firefox, there's still noticeable lag in videos, animated gifs, and even just using the autoscrolling function. I wonder why...


You might find it has something to do with your Linux distribution. When I was on Linux mint 17.3 it was very noticeable and I had to write a script to temporarily pause GPU 0 (and unpause it automatically after a set time as I often forgot to do it manually) when I wanted to watch a video. Now that I'm on 18.3 I don't notice the problem nearly as much. That script, pauseFahGPU0.sh, along with my more important script, reboot.sh, can be found at https://drive.google.com/drive/folders/ ... sp=sharing
Feel free to PM me.
Image

1080 and 1080TI GPUs on Linux Mint
SteveWillis
 
Posts: 409
Joined: Fri Apr 15, 2016 1:42 am

Re: Linux folding with multiple Nvidia GPUs

Postby hiigaran » Mon Apr 30, 2018 4:54 pm

Was just a curiosity mostly. I've got a few scripts ready to go if need be. Besides, I don't mind a little FPS problem if it means better computing. At least audio doesn't stutter.

By the way, is the latest stable nVidia driver still stuck at 384? I know 390 and 396 don't work. What about 387?
User avatar
hiigaran
 
Posts: 134
Joined: Thu Nov 17, 2011 7:01 pm

Re: Linux folding with multiple Nvidia GPUs

Postby bollix47 » Mon Apr 30, 2018 7:57 pm

FYI

I'm currently using NVIDIA-SMI 390.48 - Driver Version: 390.48 on ubuntu 18.04 and have had no problems with it.
bollix47
 
Posts: 2871
Joined: Sun Dec 02, 2007 6:04 am
Location: Canada

Re: Linux folding with multiple Nvidia GPUs

Postby hiigaran » Mon Apr 30, 2018 8:40 pm

Odd. I just checked and mine is indeed 390, even though I specified 384 through apt install.

Okay then.
User avatar
hiigaran
 
Posts: 134
Joined: Thu Nov 17, 2011 7:01 pm

Re: Linux folding with multiple Nvidia GPUs

Postby bollix47 » Mon Apr 30, 2018 8:57 pm

I believe something similar happened with my 18.04 setup. i.e. I installed 384 and afterwards it updated to 390.
bollix47
 
Posts: 2871
Joined: Sun Dec 02, 2007 6:04 am
Location: Canada

Re: Linux folding with multiple Nvidia GPUs

Postby bruce » Wed May 02, 2018 8:12 am

toTOW wrote:The answer is simple : different drivers models. In Windows, WDDM (Windows Display Driver Model) favours user experience over compute performance ... thank Microsoft for that choice :( ...

Windows does have a performance setting that allows you to give a preference to foreground or background tasks. I suspect the only difference is the frequency at which Windows interrupts active tasks to run the sckedulars which then sorts competing tasks and decides which tasks to assign to the available CPU threads. The Linux scheduler has similar setting if you feel like researching them, but in both cases, they can change when you update your OS.

Pretending that there are only two things, a FAHCore and a desktop update responding to a mouse click, competing for a single CPU resource (not a factual example), the FAHCore that hogs the CPU for (relatively) long periods of time, must be interrupted to run the scheduler so the mouse update can be processed. It finishes quickly, the scheduler runs again once the CPU is idle, it notices that the FAHCore has been interrupted and still needs processing, so resources are given to it.

The relative timesteps produced by the Windows scheduler setting and the timestep at which FAHCore runs out of one block of work and enqueues the next one are important. In the case of GPU work, the latter timestep is highly dependent on the structure of the active project AND on the speed of your GPU, The assumption that FAH can figure out how frequently it should insert artificial GPU wait-stats and for how long isn't a viable assumption -- especially since it has no information about whatever other tasks you might be running, The drivers and the scheduler have a lot more information.
bruce
 
Posts: 20019
Joined: Thu Nov 29, 2007 11:13 pm
Location: So. Cal.

PreviousNext

Return to V7.4.4 Public Release Windows/Linux/MacOS X (deprecated)

Who is online

Users browsing this forum: No registered users and 1 guest

cron