HOWTO: How I got my AMD card folding on Linux

It seems that a lot of GPU problems revolve around specific versions of drivers. Though AMD has their own support structure, you can often learn from information reported by others who fold.

Moderators: Site Moderators, FAHC Science Team

Frogging101
Posts: 85
Joined: Wed Mar 25, 2020 2:39 am
Location: Canada

HOWTO: How I got my AMD card folding on Linux

Post by Frogging101 »

My setup is as follows:

Code: Select all

Radeon R9 290 (Hawaii) with free software drivers (amdgpu/mesa)
Intel Core i7-4790
Linux Mint 18.1 64-bit
Linux kernel v5.5.1
I created this guide to share my findings and experience in setting up Folding@Home on this computer. I hope that it can provide some insight and help to others.

Installing the AMDGPU-PRO OpenCL implementation

I use the free software GPU driver stack. Unfortunately, I could not get Folding@Home to work with the free software OpenCL implementations from ROCm or Mesa. So I will not bother explaining how to set those up. If you have them, you may want to remove them in case they interfere.

What did work was something suggested by dschuermann on IRC (#radeon on freenode; great people there). This allows you to use the OpenCL implementation from the proprietary AMDGPU-PRO driver, without installing the whole driver.
  1. Get the AMDGPU-PRO driver package:
    1. Go to https://www.amd.com/en/support and select your graphics card.
    2. Scroll down to Ubuntu x86 64-bit and download the file. (currently: amdgpu-pro-19.50-967956-ubuntu-18.04.tar.xz)
    3. Extract it to an empty directory somewhere. Do not run the install script or anything else.
  2. Extract the OpenCL driver from the appropriate .deb file.
    For Vega 10 and later products, this is the .deb file matching the pattern opencl-amdgpu-pro-icd_*_amd64.deb.
    For products older than Vega 10, this is the .deb file matching the pattern opencl-orca-amdgpu-pro-icd_*_amd64.deb.
    1. In a terminal in the directory with the .deb files, run the following:

      Code: Select all

      # Extract the contents of the .deb package to the current directory
      dpkg -x DEB_FILE . # replace DEB_FILE with the appropriate pattern from above
      
      # Replace the path in the .icd file with the full path to where we're going to put the library
      sed -i 's|^|/opt/amdgpu-pro/lib/x86_64-linux-gnu/|' etc/OpenCL/vendors/amdocl*.icd
      
      # Copy the etc and opt directories extracted from the package to the system root.
      sudo cp -r etc opt /
      
Fixing the permissions issue (clGetDeviceIDs() returned -1)

Add the fahclient user to the "video" and "render" (if you have it) groups:

Code: Select all

sudo usermod -aG video fahclient
sudo usermod -aG render fahclient # don't worry if you don't have a "render" group
This is necessary because FAHClient must be able to access devices in /dev/dri, such as /dev/dri/card0 and /dev/dri/renderD128. Some distributions place the latter under the "render" group instead of the "video" group, so you may need both if your system is like that.

Some donors have been getting "OpenCL: Not detected: clGetDeviceIDs() returned -1" unless they run FAHClient as root, and adding the fahclient user to the video group doesn't fix it.

I observed this myself, and upon investigating, I discovered the problem. The official Folding@Home packages (at least the Debian-based ones) set up FAHClient to be launched by an init.d script, which launches FAHClient with "--run-as fahclient", so that it runs as the "fahclient" user.

The problem is that when FAHClient is launched with the --run-as option, it does not give itself the supplementary groups of the user passed into the --run-as option. Thus the FAHClient process will not have the permissions of the groups that the "fahclient" user is a member of. You can see this for yourself:

Code: Select all

$ pidof FAHClient
26801 26796
$ sudo cat /proc/26796/status | grep 'Groups:'
Groups:
The lack of any group numbers after "Groups:" shows that the FAHClient process does not have any supplementary groups, even though it is running as the "fahclient" user. This is due to a bug in FAHClient.

For the developers: If I had to guess (I have not verified this), this is probably happening because FAHClient uses setuid() to drop privileges with the "--run-as" argument. This is not sufficient. You must call initgroups() before setuid() to give the process the supplementary groups of the target user.

TL;DR For the users: To work around this, I created a systemd unit to start FAHClient instead of the init.d script. This works by using the systemd unit's "User" option to change users instead of having FAHClient do it.

First, stop FAHClient ("sudo systemctl stop FAHClient").

Create /etc/systemd/system/FAHClient.service (The name matters; calling it this will automatically supersede the init.d script of the same name), with the following contents:

Code: Select all

[Unit]
Description=Folding@Home Client
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/FAHClient /etc/fahclient/config.xml --pid-file=/var/run/fahclient/fahclient.pid --daemon
PIDFile=/var/run/fahclient/fahclient.pid
User=fahclient
RuntimeDirectory=fahclient
WorkingDirectory=~

[Install]
WantedBy=multi-user.target
Then do the following:

Code: Select all

sudo systemctl daemon-reload
sudo systemctl start FAHClient.service
# And, if you want it to start automatically on boot...
sudo systemctl enable FAHClient.service
Last edited by Frogging101 on Tue Jan 05, 2021 4:48 pm, edited 13 times in total.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: How I got my R9 290 folding on Linux

Post by bruce »

Thank you for the detailed HOWTO.

I suggest you make the same report at https://github.com/FoldingAtHome/fah-issues/issues. Someday the Developers will have time to examine those issues and fix most of them ... including things like your "This is a bug in FAH" mentioned above.

In the meantime, others will find it here.
Paragon
Posts: 139
Joined: Fri Oct 21, 2011 3:24 am
Hardware configuration: Rig1 (Dedicated SMP): AMD Phenom II X6 1100T, Gigabyte GA-880GMA-USB3 board, 8 GB Kingston 1333 DDR3 Ram, Seasonic S12 II 380 Watt PSU, Noctua CPU Cooler

Rig2 (Part-Time GPU): Intel Q6600, Gigabyte 965P-S3 Board, EVGA 460 GTX Graphics, 8 GB Kingston 800 DDR2 Ram, Seasonic Gold X-650 PSU, Artic Cooling Freezer 7 CPU Cooler
Location: United States

Re: HOWTO: How I got my R9 290 folding on Linux

Post by Paragon »

Nice fix, I'll try it out next time I'm tinkering with my Ubuntu box (currently folding under root). I'll try to keep an eye out for people asking about this and point them here
Revenant
Posts: 3
Joined: Wed Jun 08, 2011 11:37 pm

Re: HOWTO: How I got my R9 290 folding on Linux

Post by Revenant »

Very useful. Thank you for the post. Trying this out now.
ImageImage
dmk1000
Posts: 5
Joined: Sat Mar 28, 2020 5:03 pm

Re: HOWTO: How I got my R9 290 folding on Linux

Post by dmk1000 »

Thanks. This also worked for the Ryzen Vega 8 Mobile in my Ryzen 5 2400G.
I grabbed the AMD drives for Vega 56 an they've worked fine.
Image
Rainmaker
Posts: 15
Joined: Sat Mar 28, 2020 5:59 pm
Hardware configuration: AMD Threadripper 3960X underclocked/undervolted to 3.7GHz 1.0125v
Asus RoG Strix TRX40-E Gaming motherboard
32GB DDR4 3600MHz 16,16,16,36 (Samsung B-Die)
Sapphire Vega 56 overclocked/undervolted to 1662MHz 1070mV
Samsung Evo Plus 1TB NVMe SSD
Linux FTW

Re: HOWTO: How I got my R9 290 folding on Linux

Post by Rainmaker »

I signed up just to say thank you for this! I primarily run Linux on my desktop/workstation (ThreadRipper 3960X, Vega 56) and ended up running FaH under Windows 10 for Workstations because it just kept failing to find OpenCL on Linux. I pulled out my hair for days, trying the FOSS opencl-icd as well as the (finally releaed) amdgpu-pro for 18.04.4 - headless *and* full - to no avail.

With your systemd service file I have now got everything running properly. Again, thanks! :)
Juanro49
Posts: 16
Joined: Wed Mar 18, 2020 8:10 pm
Contact:

Re: HOWTO: How I got my R9 290 folding on Linux

Post by Juanro49 »

Frogging101 wrote:My setup is as follows:

Code: Select all

Radeon R9 290 (Hawaii) with free software drivers (amdgpu/mesa)
Intel Core i7-4790
Linux Mint 18.1 64-bit
Linux kernel v5.5.1
I created this guide to share my findings and experience in setting up Folding@Home on this computer. I hope that it can provide some insight and help to others.

Installing the AMDGPU-PRO OpenCL implementation

I use the free software GPU driver stack. Unfortunately, I could not get Folding@Home to work with the free software OpenCL implementations from ROCm or Mesa. So I will not bother explaining how to set those up. If you have them, you may want to remove them in case they interfere.

What did work was something suggested by dschuermann on IRC (#radeon on freenode; great people there). This allows you to use the OpenCL implementation from the proprietary AMDGPU-PRO driver, without installing the whole driver.
  1. Get the AMDGPU-PRO driver package:
    1. Go to https://www.amd.com/en/support and select your graphics card.
    2. Scroll down to Ubuntu x86 64-bit and download the file. (currently: amdgpu-pro-19.50-967956-ubuntu-18.04.tar.xz)
    3. Extract it to an empty directory somewhere. Do not run the install script or anything else.
  2. Extract the OpenCL driver from the appropriate .deb file.
    For Vega 10 and later products, this is the .deb file matching the pattern opencl-amdgpu-pro-icd_*_amd64.deb.
    For products older than Vega 10, this is the .deb file matching the pattern opencl-orca-amdgpu-pro-icd_*_amd64.deb.
    1. In a terminal in the directory with the .deb files, run the following:

      Code: Select all

      # Extract the contents of the .deb package to the current directory
      dpkg -x DEB_FILE . # replace DEB_FILE with the appropriate pattern from above
      
      # Replace the path in the .icd file with the full path to where we're going to put the library
      sed -i 's|^|/opt/amdgpu-pro/lib/x86_64-linux-gnu/|' etc/OpenCL/vendors/amdocl*.icd
      
      # Copy the etc and opt directories extracted from the package to the system root.
      sudo cp -r etc opt /
      
Fixing the permissions issue (clGetDeviceIDs() returned -1)

Some donors have been getting "OpenCL: Not detected: clGetDeviceIDs() returned -1" unless they run FAHClient as root, and adding the fahclient user to the video group doesn't fix it.

I observed this myself, and upon investigating, I discovered the problem. The official Folding@Home packages (at least the Debian-based ones) set up FAHClient to be launched by an init.d script, which launches FAHClient with "--run-as fahclient", so that it runs as the "fahclient" user.

The problem is that when FAHClient is launched with the --run-as option, it does not give itself the supplementary groups of the user passed into the --run-as option. Thus the FAHClient process will not have the permissions of the groups that the "fahclient" user is a member of. You can see this for yourself:

Code: Select all

$ pidof FAHClient
26801 26796
$ sudo cat /proc/26796/status | grep 'Groups:'
Groups:
The lack of any group numbers after "Groups:" shows that the FAHClient process does not have any supplementary groups, even though it is running as the "fahclient" user. This is due to a bug in FAHClient.

For the developers: If I had to guess (I have not verified this), this is probably happening because FAHClient uses setuid() to drop privileges with the "--run-as" argument. This is not sufficient. You must call initgroups() before setuid() to give the process the supplementary groups of the target user.

TL;DR For the users: To work around this, I created a systemd unit to start FAHClient instead of the init.d script. This works by using the systemd unit's "User" option to change users instead of having FAHClient do it.

Create /etc/systemd/system/FAHClient.service (The name matters; calling it this will automatically supersede the init.d script of the same name), with the following contents:

Code: Select all

[Unit]
Description=Folding@Home Client
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/FAHClient /etc/fahclient/config.xml --pid-file=/var/run/fahclient/fahclient.pid --daemon
PIDFile=/var/run/fahclient/fahclient.pid
User=fahclient
RuntimeDirectory=fahclient
WorkingDirectory=~

[Install]
WantedBy=multi-user.target
Thanks, it work for me, but I also need to extract opencl-amdgpu-pro-comgr to detect GPU Navi
Consulta como ver TV y leer noticias desde tu movil en una sola app con FeedTV.
Apoya la investigación desde esta imagen Image
Rainmaker
Posts: 15
Joined: Sat Mar 28, 2020 5:59 pm
Hardware configuration: AMD Threadripper 3960X underclocked/undervolted to 3.7GHz 1.0125v
Asus RoG Strix TRX40-E Gaming motherboard
32GB DDR4 3600MHz 16,16,16,36 (Samsung B-Die)
Sapphire Vega 56 overclocked/undervolted to 1662MHz 1070mV
Samsung Evo Plus 1TB NVMe SSD
Linux FTW

Re: HOWTO: How I got my R9 290 folding on Linux

Post by Rainmaker »

Removed as the advice causes issues for some configurations.
Last edited by Rainmaker on Sun Mar 29, 2020 2:14 pm, edited 1 time in total.
Juanro49
Posts: 16
Joined: Wed Mar 18, 2020 8:10 pm
Contact:

Re: HOWTO: How I got my R9 290 folding on Linux

Post by Juanro49 »

Rainmaker wrote:
Juanro49 wrote:Thanks, it work for me, but I also need to extract opencl-amdgpu-pro-comgr to detect GPU Navi
That reminds me...

OP, you don't need to dig out random .deb files to install the AMD OpenCL stuff. In your OP you say that you need/want the free AMD display driver. So do I, for vaapi to work in patched Chromium. If you install the full proprietary AMDGPU/AMDGPU-PRO driver stack Chromium can't use the driver with VAAPI and reverts to software decoding for video.

There is, however, an easier way. You can invoke the AMD installer script but tell it to only install OpenCL.

For Vega and later cards:

Code: Select all

amdgpu-pro-install --headless --opencl=pal
For cards before Vega:

Code: Select all

amd-pro-install --headless --opencl=legacy
The --headless flag tells it to not install the main graphics components. You can also combine opencl options if you needed to (opencl=legacy,pal). This leaves the open graphics stack working OK and installs the opencl stuff. Worked fine for me on Ubuntu 18.04.4. Just offering this in case it helps out you, or someone else.
This option installs some additional software instead of just OpenCL support, I tried this option weeks ago and I had problems viewtopic.php?f=81&t=32831
Consulta como ver TV y leer noticias desde tu movil en una sola app con FeedTV.
Apoya la investigación desde esta imagen Image
Rainmaker
Posts: 15
Joined: Sat Mar 28, 2020 5:59 pm
Hardware configuration: AMD Threadripper 3960X underclocked/undervolted to 3.7GHz 1.0125v
Asus RoG Strix TRX40-E Gaming motherboard
32GB DDR4 3600MHz 16,16,16,36 (Samsung B-Die)
Sapphire Vega 56 overclocked/undervolted to 1662MHz 1070mV
Samsung Evo Plus 1TB NVMe SSD
Linux FTW

Re: HOWTO: How I got my R9 290 folding on Linux

Post by Rainmaker »

Juanro49 wrote:This option installs some additional software instead of just OpenCL support, I tried this option weeks ago and I had problems viewtopic.php?f=81&t=32831
Ah, thanks for the heads up. It worked OK for me but in that case the manual way is better to be safe.
Frogging101
Posts: 85
Joined: Wed Mar 25, 2020 2:39 am
Location: Canada

Re: HOWTO: How I got my R9 290 folding on Linux

Post by Frogging101 »

Yeah, I didn't use the install script because I don't expect it to be designed for this very specific use case, and I don't trust it to do it properly.
warrelium
Posts: 6
Joined: Mon Mar 30, 2020 3:23 am

Re: HOWTO: How I got my R9 290 folding on Linux

Post by warrelium »

Hey Frogging101, I installed the proprietary OpenCl. Thank you for the insight. I'm on Ubuntu 18.04.4

However, even after adding fahclient to video group and with your service unit I am unable to get the FAH Client to launch with the permissions needed to get past the (clGetDeviceIDs() returned -1) error.

After creating /etc/systemd/system/FAHClient.service the FAHClient process does not start at boot

If I run FAHClient with -u root then it sees the openCl device and I am able to properly fold.

Is there a different approach I need to take with the FAHClient.service to get it to work on Ubuntu 18.04.4?


EDIT:
Tonight I learned that you have to enable the service to get it to run. The service now launches the FAHClient on boot after I did the following:

Code: Select all

sudo chmod 644 /etc/systemd/system/FAHClient.service

Code: Select all

sudo systemctl enable FAHClient
Frogging101
Posts: 85
Joined: Wed Mar 25, 2020 2:39 am
Location: Canada

Re: HOWTO: How I got my R9 290 folding on Linux

Post by Frogging101 »

Thanks warrelium. I forgot about that. I updated the OP.
toTOW
Site Moderator
Posts: 6296
Joined: Sun Dec 02, 2007 10:38 am
Location: Bordeaux, France
Contact:

Re: HOWTO: How I got my R9 290 folding on Linux

Post by toTOW »

Stickyfied topic so that we can refer people to it.
Image

Folding@Home beta tester since 2002. Folding Forum moderator since July 2008.
Ksenio
Posts: 1
Joined: Wed Apr 01, 2020 4:02 pm

Re: HOWTO: How I got my R9 290 folding on Linux

Post by Ksenio »

I tried following this, but I have still not made it work. I am running RX550 and Mint 19.3 . I have installed orca, but I am still getting this error
16:05:49:ERROR:WU00:FS01:Failed to start core: OpenCL device matching slot 1 not found, try setting 'opencl-index' manually

Any ideas what I could explore more? I am really determined to make it work :)
Post Reply