GUIDE: FAH on Linux from scratch

[MeeLee]

Here's a quick guide on installing FAH on Lubuntu (one of the Ubuntu/Linux variants of my preference).
I went with Lubuntu, as it uses LXDE, which is an easier Linux desktop variant for Windows users.

This guide would work for Ubuntu as Debian alike, is for future reference as it has many commands that need to be gathered all over the internet, and for helping out people transferring to Linux from Windows.
This guide is made for most systems with Nvidia graphics cards.

The order in which the commands are entered is important, to avoid installation errors.

• Download and install Lubuntu 18.04 ISO

Use the 64 bit version, even with only 2GB of RAM (can run on 1GB of RAM with Swap but 2GB or more of RAM is recommended)

https://lubuntu.net/downloads/ (1.1GB)

Lubuntu 18.04 is an LTS (Long Term Support) and considered most stable.

Alternatively you can download version 18.10 as well.
https://lubuntu.net/lubuntu-18-10-cosmi ... -released/ (1,6GB)
(19.04 and up currently doesn't support Python 2, a dependency for FAH).

• Write ISO to USB

- From Linux:

Code: Select all

sudo dd if=Downloads/lubuntu-18.04-alternate-amd64.iso of=/dev/sdb

if= download location of the ISO file + iso file name.
of= location of the USB flash drive. type 'df-h' to verify location by drive size, or use a gui program like 'gparted' or 'disks' to format a drive and find it's location.
do not use dd on partitions (like SDB1, SDB2, SDC1, ...). ISO files can only be extracted on the drive's root (SDB, SDC,...)

- From Windows, I would recommend to use one of the following programs, whichever does not give errors on your system:
Rufus, RMPrepUSB, YUMI, or UnetBootin

• Verify BIOS etc..

- Set bios to accept Legacy boots, (disable UEFI only)
- Dual Boot with Windows isn't recommended, but possible. Not explained here.

• Install ISO

Boot from USB FLASH drive, and select start Lubuntu or install.
From a live boot (start Lubuntu from Flash USB), it is best to install the OS without doing anything else in the background. Start live OS is great to get a feel for the operating system, but when you want to install the OS, try not to click on anything else, lest it would freeze up during installation.
Follow the prompts.
Best to install fresh on a clean SSD.
Installations on partitions are more difficult, and boot issues with dual booting can cause problems.
If a clean install isn't available, you can do a manual installation too.
Lubuntu will need a primary FAT32 boot partition of at least 300MB (preferably 500MB) mounted as '/boot/efi/', and best is an EXT4 drive of at least 15GB mounted as '/'

• Video drivers

Once installed and rebooted, start with downloading the nVidia .run file drivers from:
https://www.geforce.com/drivers
Prefer to download driver version 418.74 for now (Edit: 430.14 is available now, and should install the same).
The download will save a .run file on your downloads directory.

If you have trouble getting your wireless drivers to work, I would recommend to use wired LAN or use this $10 wifi dongle. It works out of the box on almost all Linux variants, without the need for special drivers:
https://www.amazon.com/gp/product/B00KV9TQXM

Then install needed dependencies from terminal (CTRL + ALT + T):

Code: Select all

sudo apt-get install gcc
sudo apt-get install make

After the download is complete, log out, press CTRL + ALT + F1, and log in again in the terminal (dos like screen).
if this is unsuccessful, you can also press CTRL + ALT + F2, and do things from there.
type the following commands:

Code: Select all

sudo service lightdm stop

If error, try:

Code: Select all

sudo lightdm stop

or:

Code: Select all

sudo service sddm stop

Then type:

Code: Select all

sudo init 3

Sometimes re-logging in is required after this step.

Go to your Downloads directory usually /home/'YOURLOGINNAME'/Downloads, or type:

Code: Select all

cd Downloads

and type:

Code: Select all

chmod +x ./NVIDIA-Linux-x86_64-418.74.run

or, whatever file version you've downloaded.
Typing './NVID' and pressing the 'tab' button should reveal possible options.
This command allows you to execute the run file.

Install video driver:

Code: Select all

sudo ./NVIDIA-Linux-x86_64-418.74.run

Follow along with some error codes, and allow older files to be overwriting newer files. The installation works fine,
Reboot, type:

Code: Select all

reboot

• Update linux

Next up, modify the sources list; type:

Code: Select all

sudo nano /etc/update-manager/release-upgrades

Use the keyboard keys, and verify if the last line says the following (if it says 'Prompt=LTS', change it):

Code: Select all

Prompt=normal

Press CTRL + X (exit), and Y (save), and enter (confirm).

Then update dependencies:

Code: Select all

sudo su
apt-get update
apt-get install ocl-icd-opencl-dev
apt-get upgrade
apt-get dist-upgrade

When finished, reboot:

Code: Select all

reboot

In terminal type:

Code: Select all

update-manager

Follow the screens and wait for the update to finish.

Reboot. (from GUI or from terminal type: Reboot)

To verify the update from a terminal type:

Code: Select all

lsb_release -a

It should display something like:

Code: Select all

Distributor ID:	Ubuntu
Description:		Ubuntu 18.10
Release:			 18.10
Codename:			Cosmic

• Enable all GPUs

Once in GUI, you can enable overclocking and power adjustments on all graphics cards, by opening a terminal (CTRL + ALT + T) and typing:

Code: Select all

sudo su
nvidia-xconfig --enable-all-gpus
nvidia-xconfig --cool-bits=28
reboot

• Install FAH

Go to:
https://foldingathome.org/start-folding/
and download the 2 deb files (FAHClient, and FAHControl). FahViewer is optional.
Go to your download folder, and install the deb files with the included installer (GDebi Package installer, or Discover, by double clicking the files).
Start with FAHClient, then FAHControl
Follow the screen, make sure you enter your user name, team, and key (key can be requested here, in case you don't have one)
Save the key in a special place, like in your email inbox. You might need it again.

• Configure FAH

Open the start-bar > education > FAHControl
click 'configure', 'expert'.
Change 'GPU=false' to GPU=true
Enter,
Save.

Open a terminal (CTRL + ALT + T)
Type:

Code: Select all

sudo /etc/init.d/FAHClient restart

In FAHControl, click 'configure' > 'slots'
Remove CPU slot (preferably if you have GPUs, but optional)
Add one GPU slot for each GPU card you have installed or want to use for FAH

They should now be accepted.
If not, try restarting the service again in terminal:

Code: Select all

sudo /etc/init.d/FAHClient restart

[/i]

• Control performance and Power limitations

Once FAH works fine on GPU, one can use following commands:
- To view all cards, power consumption, temperature and fan control (update every 1 second):

Code: Select all

watch -n 1 nvidia-smi

- To limit power to a GPU (lower electric bill), where '0' is the first GPU and '1' is the second GPU, and '130' and '150' is the preferred GPU power draw.
Example:

Code: Select all

sudo nvidia-smi -i 0 -pl 150
sudo nvidia-smi -i 1 -pl 130

- Fan control and GPU/VRAM over/underclocking can be done via the GUI:
click: Start-bar > Preferences > NVidia X Server Settings
You can also see how adjusting the power in nvidia-smi on the terminal window, affects GPU frequency in Nvidia X Server Settings.

• Tips

- Use fan control to keep GPU temperatures below decimal values, eg: below 80C, 70C, 60C, or 50C. Performance is usually lowered at these thresholds. Meaning, it's better to run a card at 68C than at 72C; or at 59C vs 61C. But running them at 54C vs 58, or 62C vs 66C does not change GPU frequency.
- Lower power values for saving energy, and running cards cooler; allows higher overclock too.
- Place your case air intake and exhaust fan strategically, for optimal cooling, or use an open style mining rack.
- Don't fold over 83C on RTX and GTX cards. Performance suffers and errors occur when cards run too hot, or are running slightly too high overclocks.
- Disable screen saver.

• Geforce RTX tuning for Folding

- Run them from a cold start, at their highest power rating (usually max power levels of RTX cards run from 170W on the 2060s, to 300W on the 2080 ti).

Code: Select all

sudo nvidia-smi -i 0 -pl 300

- Start folding.
- Write down the stock GPU boost frequency (1.800-2.025Mhz) at highest power levels when the card is cold (within the first minute of folding).
This is the frequency you will want to aim to overclock to.
- Lower card consumption to ~135W for base overclocking

Code: Select all

sudo nvidia-smi -i 0 -pl 135

- Overclock (start with 90Mhz on RTX cards, and slowly increase in increments of 5Mhz about every few minutes, until a BAD STATE is recorded, and WU fails, then back down by 20Mhz; and if any BAD STATES are recorded from there, decrease further by 5Mhz until no more BAD STATEs are recorded, or WUs are lost.
This procedure usually takes a few days to get it right.
Once you know your max GPU overclock (eg: 110Mhz) before WUs will go bad,
- Increase power levels by 5 or 10 Watt (sudo nvidia-smi -i 0 -pl 140; sudo nvidia-smi -i 0 -pl 145) to where you'll reach near to the specific boost frequency (1.800-2.025Mhz) of the card when it was cold.
You'll notice that increasing power consumption from 135W to higher, will also increase boost frequency.
Don't go over the stock boost frequency (not the one that's stated on the box, or sales site, but the boost frequency the card was running on at full power levels).
You'll also notice that once you'll reach the optimal power setting, the GPU boost frequency doesn't really increase much anymore with each additional watt, and it nets nearly no more additional PPDs. Meaning, you'll be consuming a lot more electricity for only little more PPD; and in most cases,the tradeoff isn't worth it.
Increasing the power consumption of the cards, will make them run hotter. Overclocking doesn't.
Fan speed needs to be readjusted after each power level change. Fan speeds are best kept between 50-80% when folding 24/7 (not obnoxiously loud, but enough to keep the cards running in the 60-70C range).
- If any error (BAD STATE) occurs, lower GPU overclock by 5Mhz decrements.

It is preferred to run the card at as low power usage as possible, and as high GPU frequency as is stable.
Take in consideration weather or temperature changes, that can affect performance.

[antropofob]

Thanks for the GUIDE!
A couple of questions:
How did you resolve FAHControl Python dependencies issue?
Is it not much simpler to install Nvidia driver via ppa:graphics-drivers/ppa method? They also contain opencl driver so no need for installing any dependencies to make it work. Also no need for stopping/restarting lightdm.
Are you sure that "does not match topology" errors are related to overclocking? I am having them regardless of overclocking/PL tinkering.
Regarding "nvidia-smi -pm 1" command that enables persistence mode: are you sure it survives reboot? I have to add it to /etc/rc.local to make power limit changes permanent.
All of these questions are tied to my experience with installing and running Xubuntu 16.04, so bear that in mind.
Thanks.

[foldy]

"does not match topology" warnings are not related to overclocking and can be ignored. "Bad state detected" errors are related to overclocking.

[MeeLee]

@antropofob:
- I use Lubuntu 18.10, which still uses Python 2.
Python 2 is reported to be unstable on Disco (19.04), the main reason I went back to 18.10.
https://launchpad.net/ubuntu/+source/py ... s/2.7.16-1
In the future this issue will probably be resolved, but as for now, the latest version supporting FAH (and Python 2.7) is 18.10
- The Nvidia driver from .run file allows you to overclock all cards (enable coolbits), without desktop errors for systems with multi GPUs. The other method has given me errors in the past. This method just simply works.
- Lightdm sometimes still runs (edit: I see you're running Xubuntu, which might or might not run lightdm). If it does, the Nvidia.run file won't install. In any case, typing the line doesn't harm, and occasionally solves problems installing the .run file when the service doesn't fully stops.
- Topology: Not sure about overclocking, read it somewhere. I've removed it for now as my GPU seems to fold well past 130Mhz overclock (with topology errors), but without crashes.
- I'm going to remove the line about persistent mode, as Nvidia appears to be removing the feat with upcoming drivers. It'll be replaced by persistent daemon. I'll have to read up on that, thanks!
https://docs.nvidia.com/deploy/driver-p ... nce-daemon

[bruce]

At some point, a version of FAHClient will be released that doesn't require Python 2. (No predicted date.) That will help.

[MeeLee]

Shouldn't be too hard, for someone who is familiar with Python.
Would make for a good Thesis for programmers (convert program successfully to Python 3).

The differences between Python 2 and 3 are mostly a couple of syntaxis and a few different commands.
Most of the code should run just the same on Python 3 libraries.

[MeeLee]

Seems like it's confirmed that Ubuntu will no longer support Python 2 in future releases.

At first I thought that perhaps Python 2 would be adopted to work in Ubuntu 19, but it seems like this won't happen.
Ubuntu 18 (and it's variants) comes standard with Python 3, but Python 2 could still be installed on it.
Ubuntu 19 and up will no longer support it.
Ubuntu 18.10 (2018/2019), is the last one to support Python 2.7

The article also describes that Python 2 will completely abandoned by 2020.
Source: https://utcc.utoronto.ca/~cks/space/blo ... LTSLinuxes
@Bruce:
Perhaps a bug report /feat request to port FAH to Python 3, can be prioritized a bit higher?
It's only 6 months away now, and I don't know if there are any beta versions working on Python 3 yet?

[bruce]

I believe development may be working on a client that doesn't use Python2. That has not been confirmed and there is no ETA, but it seems like an essential change which would be part of FAH's commitment to support the Linux distros listed on the website.

It has been accepted as a DEFECT
https://github.com/FoldingAtHome/fah-issues/issues/858

[MeeLee]

The defect is talking about Ubuntu 12, 7 year old OS.
No mention there about Python 2.7 getting discontinued.

[Theodore]

I re-installed everything straight from Lubuntu 18.10 iso, rather than go the long way of updating 18.04.
The long way (update 18.04 to 18.10), especially when shifting from lightdm to sddm desktop, didn't work well, like one of the issues was that it gave a lot of duplicate icons and programs.

The only issue I had with direct 18.10 install, was to make sure the destination drive is either pre formatted or raw, before installation.
Lubuntu 18.04 handles clean installations correctly, but 18.10 needs a clean drive for best results.
Selecting the fresh installation option, doesn't work with some info still on the drive (I presume UEFI related).

It had to be done, because my system, though behind 2 routers, didn't have a software firewall, and was subjected to the Linux attacks, with network slowdown.
Network traffic slowed to 8 kbps, and broadcast very intermittent. Took nearly a full hour to upload 8 pending WUs.
I don't understand what anyone has to gain, hacking that server. They're just slowing down fah progress, and there is no other valuable information on that server.
I'd recommend anyone to update to 18.10-1 from 18.xx, or at the least update to Linux kernel 4.18.0.20, or higher, where most issues are resolved.
Source

Also, a lot of the procedures can be done without rebooting.

If anything goes wrong with the video drivers, all it takes is to reinstall them again,by repeating the.run file procedures.
As long as Lubuntu has the nouveau drivers installed, as primary drivers, it'll work.
If it somehow tries to install the NVidia package drivers, things get messed up.

Update procedures try to install the NVidia drivers in the 18.04 to 18.10 update, which is why you'd have to install the run file early on in the upgrade procedure.
Not necessary when starting from a clean install from 18.10.

So far, I have the impression that 18.10-1 is stable, but the newest 430.26 drivers are a bit slower than the older drivers, by a margin of 5 to 10%. I could have also had some slow WUs, so in time I'll be able to tell more.

[bruce]

I don't understand what anyone has to gain, hacking that server. They're just slowing down fah progress, and there is no other valuable information on that server.

The hackers don't know that -- just like the robocallers don't know (or care) whose phone number they're spamming.

[Theodore]

Yup, I'm reporting a 10% performance drop on the 430.26 drivers!
It's been consistent with the exact same settings, for the past 30 hours.

In the beginning, I was skeptical about drivers losing performance, but for some reason, my PPD count has been 10% lower, and the only variable is the newer GPU driver.

My other PC has also received the newer Linux 4.18.0.20 core, still runs the older 418 drivers, and hasn't seen any drop in performance, which means it's the drivers.

[bruce]

Are you comparing the points for different projects or have you explicitly determined that you're running the same projects?

[Theodore]

Are you comparing the points for different projects or have you explicitly determined that you're running the same projects?

An average of 3 days of folding now...
I'm looking into finding how to safely remove the newer drivers in Linux, and install older ones.

[Theodore]

Performance difference with the newer drivers now comes within 3-5% margin.
Still slower than 418 drivers, but a lot better than the initial 6-10% slowdown.
This was due to one of my GPUs losing performance over time.
One of the GPUs kept occasionally sending bad WUs with the new drivers, and I had to lower the overclock on one of the cards by 5mhz.
This drop in overclock would not really be measurable and wouldn't reflect in the 3-5% overall drop in performance I'm currently measuring.
On the other hand, each card was fed 1 to 3 watts extra... So there's a chance that the newer drivers still are slower than the old drivers by a margin of >5%.