NVIDIA GPU3 Linux/Wine Headless Install Guide

[Hyperlife]

I have a couple of headless Ubuntu SMP folders that each have an open PCI-E slot. Recently I picked up an EVGA 9600 GSO to see if I could get it folding under Linux on one of these bare-bones folding rigs. After a little trial and error, I got it to work from the console, without running X. I thought it would be helpful to put together a step-by-step guide to running the GPU client on a headless system, without running X or other unnecessary software.

This guide requires a little familiarity with Linux, but I'll try to be as descriptive as possible while still keeping it brief. It covers the whole install, from getting the Ubuntu ISO to running the GPU3 client. It's much easier to cut and paste the commands from this guide instead of typing them in by hand, so I recommend following this guide while running an ssh connection from another computer.

1. Download the ISO of the 64-bit version of Ubuntu Server 9.04. Here's a download link to mirrors.kernel.org. Burn the ISO and boot from the CD.

2. Go through the install, accepting all defaults (careful: this will wipe your hard drive, as this guide is designed for dedicated folding rigs). When asked which server software to include near the end of the install, select OpenSSH Server if you will be accessing the box remotely (move the cursor down with the tab key and then hit the spacebar when the cursor is next to the OpenSSH Server box).

3. Upon completion, the system will reboot. Log in on the console using the username and password you created during the install (or use ssh to log in from another system if you selected OpenSSH Server).

4. Update the system software and restart by entering the following three commands:

Code: Select all

sudo apt-get update
sudo apt-get dist-upgrade
sudo reboot

You will be asked for your password when using the sudo command. Rebooting will use the latest kernel version you just downloaded and installed. If you don't reboot, you may download the wrong header files in Step 7.

5. Log back in after rebooting, then install the latest version of Wine:

Code: Select all

sudo sh -c "echo 'deb http://ppa.launchpad.net/ubuntu-wine/ppa/ubuntu jaunty main' >> /etc/apt/sources.list"
sudo sh -c "echo 'deb-src http://ppa.launchpad.net/ubuntu-wine/ppa/ubuntu jaunty main' >> /etc/apt/sources.list"
sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 5A9A06AEF9CB8DB0
sudo apt-get update
sudo apt-get install wine


Note: the forum's line-wrapping may split the first two commands into two lines, depending on the width of your browser -- they should be on one line only.

6. Download the latest NVIDIA Linux 64-bit driver and the CUDA 3.0 32-bit toolkit:

Code: Select all

wget http://us.download.nvidia.com/XFree86/Linux-x86_64/256.35/NVIDIA-Linux-x86_64-256.35.run
wget http://developer.download.nvidia.com/compute/cuda/3_0/toolkit/cudatoolkit_3.0_linux_32_ubuntu9.04.run

Watch the word wrap on the second command.

7. Before you install the NVIDIA driver, you need to install some Linux build utilities and header files:

Code: Select all

sudo apt-get install build-essential linux-headers-`uname -r`

Those are backward apostrophes surrounding the uname command, which allows the output of uname to be incorporated into the filename of the headers file. Cut and paste from the above line if you have trouble downloading the proper headers.

8. Install the NVIDIA driver:

Code: Select all

sudo sh NVIDIA-Linux-x86_64-256.35.run

Follow the prompts. Go ahead and install the OpenGL libraries too. Answer "Yes" when asked if you would like to run the nvidia-xconfig utility.

9. Install the NVIDIA 32-bit toolkit:

Code: Select all

sudo sh cudatoolkit_3.0_linux_32_ubuntu9.04.run

Use the default path by hitting Enter when prompted.

10. Link the toolkit:

Code: Select all

sudo sh -c "echo '/usr/local/cuda/lib' > /etc/ld.so.conf.d/cuda.conf"
sudo ldconfig

11. Create the WINE directory structure:

Code: Select all

wine notepad

Disregard the errors; all we need is the directory structure created in ~/.wine when you run it the first time.

12. Download the latest Wine wrappers, save them to Wine's windows/system32 directory, and create a few symlinks:

Code: Select all

cd ~/.wine/drive_c/windows/system32
wget http://www.linuxfah.info/wrapper-3.0/cudart/cudart.dll.so
wget http://www.linuxfah.info/wrapper-3.0/cufft/cufft.dll.so
ln -s cudart.dll.so cudart32_30_14.dll
ln -s cudart.dll.so cudart.dll
ln -s cudart.dll.so nvcuda.dll
ln -s cufft.dll.so cufft32_30_14.dll
ln -s cufft.dll.so cufft.dll

13. Test to make sure the wrapper is properly linked:

Code: Select all

ldd ~/.wine/drive_c/windows/system32/cudart.dll

The output should look something like this:

Code: Select all

        linux-gate.so.1 =>  (0xf7706000)
        libcudart.so.3 => /usr/local/cuda/lib/libcudart.so.3 (0xf7697000)
        libwine.so.1 => /usr/lib32/libwine.so.1 (0xf7556000)
        libm.so.6 => /lib32/libm.so.6 (0xf752f000)
        libc.so.6 => /lib32/libc.so.6 (0xf73cc000)
        libdl.so.2 => /lib32/libdl.so.2 (0xf73c8000)
        libpthread.so.0 => /lib32/libpthread.so.0 (0xf73af000)
        librt.so.1 => /lib32/librt.so.1 (0xf73a6000)
        libstdc++.so.6 => /usr/lib32/libstdc++.so.6 (0xf72b7000)
        libgcc_s.so.1 => /usr/lib32/libgcc_s.so.1 (0xf72a7000)
        /lib/ld-linux.so.2 (0xf7707000)

If you see any of the libraries with "not found" then the linking didn't work. Go back to step 10.

14. Create the following script to initialize CUDA without running X:

Code: Select all

cd
nano -w cudainit

Paste the following into the file:

Code: Select all

#!/bin/bash

PATH=$PATH:/usr/local/cuda/bin

modprobe nvidia

if [ "$?" -eq 0 ]; then

# Count the number of NVIDIA controllers found.
N3D=`/usr/bin/lspci | grep -i NVIDIA | grep "3D controller" | wc -l`
NVGA=`/usr/bin/lspci | grep -i NVIDIA | grep "VGA compatible controller" | wc -l`

N=`expr $N3D + $NVGA - 1`
for i in `seq 0 $N`; do
mknod -m 666 /dev/nvidia$i c 195 $i;
done

mknod -m 666 /dev/nvidiactl c 195 255

else
exit 1
fi

To exit nano and save the file, hit CTRL-X.

15. Make the cudainit file executable.

Code: Select all

chmod u+x cudainit

16. Run the cudainit script as root.

Code: Select all

sudo ./cudainit

You should now have a few extra devices in /dev; check to make sure:

Code: Select all

ls /dev/nv*

If you have one video card, you should see the following:

Code: Select all

/dev/nvidia0  /dev/nvidiactl

You will need to rerun this script before starting the client whenever you reboot your system. You may want to include this in a login/startup script so it executes automatically whenever you reboot.

17. Time to download the console version of the GPU3 folding client (finally!):

Code: Select all

wget http://www.stanford.edu/~friedrim/.Folding@home-Win32-GPU_XP-631.zip -O Folding@home-Win32-GPU_XP-631.zip

18. Ubuntu doesn't come with an unzip program, so we'll install it:

Code: Select all

sudo apt-get install unzip

19. Create a directory to keep your folding files, then unzip the client into it, and delete the Stanford-supplied .dll files:

Code: Select all

mkdir ~/fahgpu3
unzip Folding@home-Win32-GPU_XP-631.zip -d fahgpu3
rm ~/fahgpu3/*.dll

20. Time for the moment of truth: let's start the client! For all GeForce cards in the 8 series, 9 series, 100 series, 200 series, and 300 series (non-Fermi):

Code: Select all

cd ~/fahgpu3
nice wine Folding@home-Win32-GPU.exe -verbosity 9 -forcegpu nvidia_g80

For all 400-series (Fermi) GeForce cards:

Code: Select all

cd ~/fahgpu3
nice wine Folding@home-Win32-GPU.exe -verbosity 9 -forcegpu nvidia_fermi

If you have more than one video card, you can run a separate instance of the client using the -gpu flag. Use -gpu 0 for the first card, then -gpu 1 for the second card, etc. If you have only one card, you don't need the -gpu flag. Make sure that you assign different Machine IDs during the configuration if you run more than one client!


To make sure the GPU client doesn't use an entire CPU core, use the Auto-Mator script to start the client. Auto-Mator will lower the CPU demand of the GPU client by automatically adjusting the SLEEPWAIT environmental variable:

For non-Fermi cards listed above:

Code: Select all

cd ~/fahgpu3
wget http://www.linuxfah.info/wrapper-3.0/auto-mator-g80.sh
chmod +x auto-mator-g80.sh
./auto-mator-g80.sh

For Fermi:

Code: Select all

cd ~/fahgpu3
wget http://www.linuxfah.info/wrapper-3.0/auto-mator-fermi.sh
chmod +x auto-mator-fermi.sh
./auto-mator-fermi.sh

Your GPU fan is not automatically controlled when running headless. To set the fan, install and configure the nvclock utility:

Code: Select all

sudo apt-get install automake
wget http://www.linuxhardware.org/nvclock/nvclock0.8b4.tar.gz
tar xzvf nvclock0.8b4.tar.gz
cd nvclock0.8b4
./autogen.sh
./configure --disable-nvcontrol
make
sudo make install

You can now set your fan to "auto" with the following command:

Code: Select all

sudo nvclock -f -F auto

And you can see your fan's current speed with:

Code: Select all

sudo nvclock -i

Since we're running 64-bit Ubuntu, we can run an SMP client at the same time if you have a multi-core CPU. I recommend Ivoshiee's excellent finstall script. Just enter the following commands:

Code: Select all

cd
wget http://ra.vendomar.ee/~ivo/finstall
chmod u+x finstall
./finstall smp
foldingathome/folding start

More info is available at the finstall wiki page.


Some notes:

I prefer to run the GPU3 client using the screen utility. To install:

Code: Select all

sudo apt-get install screen

To start, just type "screen -S GPU3" at the prompt, and you will create a new terminal named GPU3. Start the folding client (or the Auto-Mator script), then you can "detach" the screen by pressing CTRL-A, release, then d. To reconnect, type "screen -r GPU3" at the prompt. Type "man screen" at the prompt for more info if you're not familiar with it. (If you only run one screen, you can start it by simply typing "screen" and reconnecting with "screen -r". The use of the -S flag lets you label your screen sessions, so it's easier to reconnect to the session you want when multiple sessions are running.)

When you stop folding by hitting CTRL-C, make sure all of your core files terminate before restarting the client. If the core doesn't terminate and you restart, the WU will EUE. After exiting the client, type "top" to see if any processes labeled "FahCore_11.exe" or "FahCore_15.exe" are still running. Note the PID number of the process on the left, then quit out of top (hit "q"), then type "kill" followed by the PID number and hit Enter. Check "top" again to make sure it's killed.

For now, I recommend using Ubuntu 9.04 instead of 9.10 or 10.04, especially if you want to run an SMP client at the same time. SMP performance on 9.10 is much slower than 9.04, and 10.04 has some extra steps needed to get the SMP client running.

For GPU2: Special thanks to all who contributed to the GPU2 Linux/WINE thread, especially shatteredsilicon, actong, Shelnutt2, and Mindmatter. Extra special thanks to ihaque for his input from the Pande Group side of things.

For GPU3: Special thanks to Shelnutt2, Sidicas, and the testers in this thread for all their hard work in getting GPU3 to run! For additional information and more general instructions for running GPU3 with X, check out Shelnutt2's excellent GPU3 Linux wiki page.

Edit 11/11/08: ineffective nice command removed
Edit 12/09/08: Changed recommended Ubuntu version to 8.04.1, updated driver & client links, added nice command back, added dist-upgrade & reboot to Step 4, clarified multi-GPU use
Edit 2/3/09: Updated recommended Ubuntu version to 8.04.2
Edit 2/25/09 & 2/26/09: Warning about new Core 14 WUs that crash with current CUDA wrapper; new wrapper to be released soon
Edit 3/2/09: New wrapper posted for Core 14 WUs, cleaned up installation text, linked to latest NVIDIA driver, added more names to thank-you section
Edit 3/4/09: Deleted unnecessary symlink step, updated wrapper link to GPU2 wiki
Edit 3/6/09: Fixed and reordered wrapper test step
Edit 5/5/09: Updated link to latest NVIDIA driver
Edit 8/1/09: Removed warning about Core 14, updated links to Ubuntu 8.04.3 and NVIDIA driver 180.60
Edit 9/8/09: Fixed typo spotted by bollix47
Edit 7/25/10: Major rewrite for GPU3, addition of nvclock instructions

[bollix47]

This appears to be an excellent quide. Thanks very much for creating it.

Sorry if this question is not a sensible one, but if one is not running 'headless' and is running 'X' using Ubuntu 8.10 desktop version will this quide still work as expected? Any different steps other than not downloading and installing the server version?

[Shadowtester]

May I suggest this command for running the gpu2 client with screen it will start a screen session in the background and name it GPU2 to make it easy to identify in case you have more than one screen session this is the command I use normally which works great. You could also add additional flags such as -forceasm and -advmethods if wanted I normally use both so they will work fine with the gpu2 client using wine and you could also create a simple file with the following setup and add run it in your startup script so that any time you reboot the gpu2 client would get started automatically. I named the following file gpu2 and made it executable so to start the gpu2 client I just have to run that file its easier than typing the entire command all the time and to check the status in screen all you need to do is.

Code: Select all

screen -r GPU2


Then when finished just use the following to detach the screen session and exit
"CTRL-A D"

gpu2

Code: Select all

cd ~/fahgpu2
screen -A -m -d -S GPU2 nice -n10 wine Fah-GPU.exe -forcegpu nvidia_g80 -verbosity 9


[shatteredsilicon]

A couple of points:
1) Look into using the NVreg_DeviceFileGID and NVreg_DeviceFileMode options on the kernel module (note: the latter expects a decimal paramter, standard UNIX permissions are usually specified in octal so make sure you do the conversion!). It will avoid the ugly manual creation of device nodes with permissions. The nVidia kernel module should handle that for us.

2) No, this won't work if you are also running X. Logging in will change permissions on the device nodes to the current user which will break things, if not immediately then when the next WU tries to run. If you are using X, fire up the client when you've logged in.

[jcfuller]

Up and running. Thank You very much.
I installed samba so now I need to figure out how to make my fah directory shared.

James

[tmoble]

What an outstandingly excellent guide. Keystroke perfect. Was able to ssh in on the first try, GPU F@H started up and worked on the first try. Thanks for taking time and effort to do this. It took me most of a week to get it running the old-fashioned way under Fedora, compiling all the stuff from scratch, etc. I blew away the Fed9 install because I was having trouble with individual processor core crashes and not having any luck troubleshooting them. I was running the native SMP client, got to where it would crash about every half hour. Not OCd either.

System is an AMD quad 9600, 4GB RAM, Gigabyte 8800 with 256MB. I understand there's new WU's out that are slow. The first WU is a 5748, it's taking 2:15 to do a step, I'm used to the 5506 WUs that complete in about 1:10. Is this expected? almost 4 hours per WU?

Also n otice the GPU client is using very little processor, under 20% most of the time. Expected? It was killing a whole core before.

[shatteredsilicon]

Yes, the new wrapper actually makes the CPU monitoring process sleep inbetween polling the GPU, rather that polling all the time. This has been the case for a fair few months, though. It could be you were using one of the early versions of the wrapper.

[KE1HA]

Samba discussion moved here to the link below.

http://foldingforum.org/viewtopic.php?f=15&t=6836&p=67668#p67668

[jcfuller]

KE1HA,
Thanks for the walk through but been there done that and it does not work (for me)
I don't want to hijack this thread into a samba issue but ....
The thing that gets me is in nautilus it's a right click on a folder select shared and
bingo anyone on the network can access it without log ins and passwords??
Why can't this functionality be available from the CLI?

I started another thread in case someone may have some other ideas.

http://foldingforum.org/viewtopic.php?f=15&t=6836&p=67668#p67668

James

[jcfuller]

High Praises for this walkthrough.
The biggest problem I had was my clutsy typing and aging eyes.
Would it be possible for someone to put all of the steps in a text file available with a
wget;and then using screen,nano or some other method copy and paste the lines from the text
file to the command prompt? I realise a full blown script for the whole operation might be
possible but it seems cut and paste might offer a simpler(?) solution for now.
Not to clutter this thread I started another one for the discussion of methods to accomplish
this goal.




James

[uncle_fungus]

I've uploaded the list of commands here (with no annotations): http://fire-salamander.co.uk/uploads/Fo ... ess_nv.txt
Seperate commands are delimited by blank lines, and I've altered the cudainit step to remove the need for pasting into nano.

[osgorth]

Can someone please clarify if this works with multiple GPUs, and if so, if it works truly headless - i.e. without VGA dummies or anything like that?

[jcfuller]

I've uploaded the list of commands here (with no annotations): http://fire-salamander.co.uk/uploads/Fo ... ess_nv.txt
Seperate commands are delimited by blank lines, and I've altered the cudainit step to remove the need for pasting into nano.

Thank you.
Now that I discovered putty all should be fine if I set up another box.

James

[Hyperlife]

This appears to be an excellent quide. Thanks very much for creating it.

Sorry if this question is not a sensible one, but if one is not running 'headless' and is running 'X' using Ubuntu 8.10 desktop version will this quide still work as expected? Any different steps other than not downloading and installing the server version?

If you haven't installed the NVIDIA driver, you'll need to restart X after installing it. I would shut X down before installation, though I'm not sure if this is necessary.

You should also be able to skip the cudainit file in Step 17 -- merely starting X once will create the devices in /dev for you. You will need to set the PATH environmental variable to include /usr/local/cuda/bin elsewhere (such as in /etc/environment).

Other than that, as shatteredsilicon mentioned, you should be able to simply start the client while running X.

[Hyperlife]

A couple of points:
1) Look into using the NVreg_DeviceFileGID and NVreg_DeviceFileMode options on the kernel module (note: the latter expects a decimal paramter, standard UNIX permissions are usually specified in octal so make sure you do the conversion!). It will avoid the ugly manual creation of device nodes with permissions. The nVidia kernel module should handle that for us.

Thanks, I'll look into that. The script that creates the device nodes came from this post from an NVIDIA staffer, so I figured if they suggest it, it should work OK. Your suggestion does sound cleaner, though.