How to set up Folding@home on FreeNAS/TrueNAS - a guide

FAH provides a V7 client installer for Debian / Mint / Ubuntu / RedHat / CentOS / Fedora. Installation on other distros may or may not be easy but if you can offer help to others, they would appreciate it.

Moderators: Site Moderators, FAHC Science Team

How to set up Folding@home on FreeNAS/TrueNAS - a guide

Postby Jesse_V » Tue Sep 15, 2020 4:59 am

This is a guide on how to configure a PC to manage a large storage pool using FreeNAS and running Folding@home at the same time. Up until now, to the extent of my research on this forum and across the Internet, they seemed fairly mutually exclusive. There are alternatives, such as Ansible, but FreeNAS is the most popular and widely-recommended OS for a NAS. However, it is based on FreeBSD, which is not supported by Folding@home, and it is not possible to run F@h inside a FreeBSD jail in FreeNAS. Therefore another solution is needed.

First, I tried setting up a lightweight headless Debian VM inside FreeNAS. I installed the Nvidia drivers and set it up for F@h as normal. The BSD hypervisor, bhyve, does support GPU passthrough. However, Phoronix reported that performance is around 66% compared to bare metal. This is not good for F@h and not really acceptable for the high-priority COVID Moonshot sprint projects. I therefore gave up on FreeNAS for F@h at first.

However, over the past few months, iXsystems started unifying the FreeNAS and TrueNAS platforms and announced TrueNAS SCALE as a new project. As TrueNAS SCALE is based on Debian, F@h is supported and I took a closer look at it. Here is how you can get Folding@home up and running on TrueNAS SCALE:

  1. Follow the guides to install TrueNAS onto a USB flashdrive.
  2. Use the web interface to import or create a storage pool and get TrueNAS set up.
  3. Use the terminal interface on the PC to drop into a shell.
  4. Install the Nvidia drivers with:
    Code: Select all
    apt-get --no-install-recommends install nvidia-driver nvidia-opencl-icd nvidia-cuda-mps nvidia-smi nvidia-tesla-440-opencl-icd

    You can also use nvidia-detect to list detected hardware and suggest the correct driver. Its possible that ocl-icd-opencl-dev or nvidia-opencl-dev could be the packages strictly necessary for a minimal system, but the above sets up everything as needed.
  5. Use the web interface to restart the system.
  6. Double-check that the drivers are correctly installed. You can run nvidia-smi to see the status of connected graphics cards. I also installed "clinfo" and ran
    Code: Select all
    clinfo | less
    to run a test of OpenCL and CUDA.
  7. Install FAHClient on the system. As of writing, the FAHClient package for Debian can be found from the download page at specifically with this link. This URL can be fetched with wget and installed with
    Code: Select all
    dpkg -i fahclient*.deb
  8. Follow the initial configuration guide to set up the F@h client for the first time. I suggest selecting "Full" to ensure that the client is set up for all hardware.
  9. Modify the config.xml file at /etc/fahclient/config.xml. Add the following lines:
    Code: Select all
    <allow v='127.0.0.1 192.168.0.0/16'/>
    <password v='myfahpassword/>

    The network subnet can be customized for your LAN to allow a connection from FAHControl on another PC. The password should also be changed.
  10. Start F@h with either
    Code: Select all
    systemctl start FAHClient
    or
    Code: Select all
    /etc/init.d/FAHClient start
    I prefer the latter.
  11. Examine the logs with
    Code: Select all
    tail -30 /var/lib/fahclient/log.txt
    and ensure that it is running correctly. The beginning of the log should list detected GPUs and find the OpenCL and CUDA platforms. If workunits are running, especially GPU workunits, you've got it set up correctly!
  12. Connect FAHControl. On another PC, open FAHControl, click "Add" in the bottom-left corner, add the IP address of the NAS and the above password, and click Save. Assuming no firewalls are in the way for TCP/36330, it should connect. This will make it easy to control this instance on the NAS.
  13. Back on the TrueNAS terminal, navigate to the root directory, which should be the default home location for the root user.
  14. Back up the fahclient user with
    Code: Select all
    tail -1 /etc/passwd > fah_user.txt
    This is critical as TrueNAS SCALE seems to regenerate /etc/passwd at boot. Without this entry, FAHClient will not be able to start.
  15. Use the web interface to set up a startup script. As of writing, this can be found at Data Protection > Init/Shutdown Scripts. Create a POSTINIT script that runs the following command:
    Code: Select all
    cat /root/fah_user.txt >> /etc/passwd; /etc/init.d/FAHClient start
  16. Create a SHUTDOWN script that runs the following command to shut down F@h:
    Code: Select all
    /etc/init.d/FAHClient stop
  17. Exit the shell in the TrueNAS terminal to return to the main menu.
  18. Use the web interface to restart the system. From the terminal, see that F@h is shutting down. At startup, you should hear the fans kick up, or check the logs to double-check that F@h is running again. If not, review /etc/passwd to verify that the entry for fahclient is present at the bottom of the file.
  19. Write a F@h monitoring script to run on the TrueNAS terminal screen. I created monitor.sh:
    Code: Select all
    #!/usr/bin/env bash

    watch -t -c -p -n 3 " \
        nvidia-smi | grep -v -e '^\s*$' \
        && echo \
        && top -b -n 1 | head -13 \
        && echo \
        && tail -5 /var/lib/fahclient/log.txt"

    I also added some lines that used netstat, vnstat, and df to display network activity and disk consumption, but the above works for F@h.
  20. It should be set now!

I am seeing about a 20 to 25% gain in improvement over Windows, which is fairly common for Linux, so I'm happy. Another option is to use a Docker instance. I tried both the official Docker container and a third-party container, and while both recognized the GPUs, they failed to accept the OpenCL drivers or open a port for remote control. I suspect (but did not prove) that this is due to the early development of TrueNAS SCALE, so its something that I might circle back to next year when TrueNAS SCALE is fully released. In any case, installing F@h natively on the system works too. Please post below if this also works for you or if you have any suggestions for this guide.
F@h is now the top computing platform on the planet and nothing unites people like a dedicated fight against a common enemy. This virus affects all of us. Lets end it together.
Jesse_V
Site Moderator
 
Posts: 2851
Joined: Mon Jul 18, 2011 5:44 am
Location: Western Washington

Re: How to set up Folding@home on FreeNAS/TrueNAS - a guide

Postby PantherX » Tue Sep 15, 2020 5:08 am

Fantastic write-up, Jesse_V! Thanks for sharing the detailed steps and rational as to why it's needed :eugeek:
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
User avatar
PantherX
Site Moderator
 
Posts: 6591
Joined: Wed Dec 23, 2009 10:33 am
Location: Land Of The Long White Cloud

Re: How to set up Folding@home on FreeNAS/TrueNAS - a guide

Postby Jesse_V » Wed Sep 16, 2020 8:18 am

Caution: the F@h installation, configuration of Docker images, and other changes are specific to the running version of TrueNAS SCALE and do not seem to follow from one version to the next. After an update, the default home directory (/root) seems normal, but the system is using a fresh image, so F@h does not seem to be installed, /etc/fahclient is not found, and Docker reports that no images are available. I need to further test this to see if this is only happening to the version update I just did, or if this is a general issue. I believe it's possible to get these services to stay within /root/ or a ZFS volume, which should survive between updates.
Jesse_V
Site Moderator
 
Posts: 2851
Joined: Mon Jul 18, 2011 5:44 am
Location: Western Washington


Return to Q&A about unsupported distros of Linux

Who is online

Users browsing this forum: No registered users and 1 guest

cron