GPU3 (Core 15) & Linux discussion

[Sidicas]

Hey it might work, and I don't think it can hurt anything but I ment that it probably doesn't work. Shouldn't have said it's not a good idea

Yes you're right, it won't work..

[MtM]

Ok last thought before I leave you pro's to it

Could it be that the client is changed and now uses nvcuda.dll to ask for a driver version ( wine registries! ) instead of the old scenario where the core would ask for a driver version through the runtime?

Could that be the problem you are having?

[friedrim]

Could it be that the client is changed and now uses nvcuda.dll to ask for a driver version ( wine registries! ) instead of the old scenario where the core would ask for a driver version through the runtime?

The client did change in this regard. Previously it used the Direct3D API to determine the board type. Now the client checks for nvcuda.dll. If the dll is present, then the client assumes it is running on a CUDA-enabled machine. If nvcuda.dll is not present, then client falls back to the Direct3D interface, i.e., ATI boards are still being detected through Direct3D. Also the client is looking for the CUDA runtime lib to set the gpu species.

[Sidicas]

Code: Select all


cudaError_t WINAPI wine_cudaGetTextureReference( struct textureReference **texRef, const char* symbol){
        WINE_TRACE("\n");
        return cudaGetTextureReference( texRef, symbol );
}



Should be:

Code: Select all


cudaError_t WINAPI wine_cudaGetTextureReference( const struct textureReference **texRef, const char* symbol){
        WINE_TRACE("\n");
        return cudaGetTextureReference( texRef, symbol );
}



[Sidicas]

I do believe I have a possible working solution


Here's what I've got...

CUDA Toolkit 2.3 AND CUDA Toolkit 3.0 installed

cufft_gpu3.so.dll (Compiled with CUDA ToolKit 3.0):
http://shelnutt2.host56.com/cufft

cudart_gpu3.so.dll (Compiled with CUDA ToolKit 3.0):
http://shelnutt2.host56.com/cudart

cudart.dll (Compiled with CUDA ToolKit 2.3):
http://www.hyperchronos.org/cudart.dll.so

nvcuda.dll -> cudart.dll
cufft32_30_14 -> cufft_gpu3.so.dll
cudart32_30_14 -> cudart_gpu3.so.dll

Code: Select all

sidicas@Belldandy:~/.FAHGPU3$ ls -al cudart*
lrwxrwxrwx 1 sidicas sidicas     18 2010-06-03 23:23 cudart32_30_14.dll -> cudart_gpu3.dll.so
-rw-r--r-- 1 sidicas sidicas  99037 2010-06-03 23:26 cudart.dll
-rwxr--r-- 1 sidicas sidicas 125670 2010-06-03 19:16 cudart_gpu3.dll.so


Code: Select all

sidicas@Belldandy:~/.FAHGPU3$ ls -al cufft*
lrwxrwxrwx 1 sidicas sidicas    17 2010-06-03 23:26 cufft32_30_14.dll -> cufft_gpu3.dll.so
-rwxr--r-- 1 sidicas sidicas 81917 2010-06-03 19:27 cufft_gpu3.dll.so


Code: Select all

sidicas@Belldandy:~/.FAHGPU3$ ls -al *.dll
lrwxrwxrwx 1 sidicas sidicas    18 2010-06-03 23:23 cudart32_30_14.dll -> cudart_gpu3.dll.so
-rw-r--r-- 1 sidicas sidicas 99037 2010-06-03 23:26 cudart.dll
lrwxrwxrwx 1 sidicas sidicas    17 2010-06-03 23:26 cufft32_30_14.dll -> cufft_gpu3.dll.so
lrwxrwxrwx 1 sidicas sidicas    10 2010-06-03 23:30 nvcuda.dll -> cudart.dll


Running GPU3 client Core11 right now on a GTS 250 with no problems... We'll see if I can pick up any Core15 workunits and if so, if it runs them properly...

[Sidicas]

OH YEA!!! WE ARE GETTING CLOSE.. GPU3 client working perfectly with Core11.. Downloads and tries to run Core15...

Code: Select all

sidicas@Belldandy:~/.FAHGPU3$ wine ./Folding@home-Win32-GPU.exe -forcegpu nvidia_g80

Note: Please read the license agreement (Folding@home-Win32-GPU.exe -license). Further
use of this software requires that you have read and accepted this agreement.

[03:47:28] cudaRuntime lib not found.
[03:47:28] Gpu species not recognized.


--- Opening Log file [June 4 03:47:28 UTC]


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.30r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: Z:\home\sidicas\.FAHGPU3
Executable: ./Folding@home-Win32-GPU.exe
Arguments: -forcegpu nvidia_g80 -gpu 0 -verbosity 9

[03:47:28] - Ask before connecting: No
[03:47:28] - User name: Sidicas (Team 70567)
[03:47:28] - User ID: 26027C39578EBBC6
[03:47:28] - Machine ID: 2
[03:47:28]
[03:47:28] Gpu species not recognized.
[03:47:28] Loaded queue successfully.
[03:47:28]
[03:47:28] + Processing work unit
[03:47:28] Core required: FahCore_15.exe
[03:47:28] - Autosending finished units... [June 4 03:47:28 UTC]
[03:47:28] Trying to send all finished work units
[03:47:28] + No unsent completed units remaining.
[03:47:28] - Autosend completed
[03:47:28] Core not found.
[03:47:28] - Core is not present or corrupted.
[03:47:28] - Attempting to download new core...
[03:47:28] + Downloading new core: FahCore_15.exe
[03:47:28] Downloading core (/~pande/Win32/x86/NVIDIA/Fermi/Core_15.fah from www.stanford.ed
u)
[03:47:29] Initial: AFDE; + 10240 bytes downloaded
[03:47:29] Initial: CC12; + 20480 bytes downloaded
[03:47:29] Initial: FCB5; + 30720 bytes downloaded
[03:47:29] Initial: EC54; + 40960 bytes downloaded
[03:47:29] Initial: 0977; + 51200 bytes downloaded
[03:47:29] Initial: 7240; + 61440 bytes downloaded
[03:47:29] Initial: 2ACA; + 71680 bytes downloaded
[03:47:29] Initial: FC28; + 81920 bytes downloaded
[03:47:29] Initial: 4F5A; + 92160 bytes downloaded
[03:47:29] Initial: 2DFF; + 102400 bytes downloaded
[03:47:29] Initial: 4564; + 112640 bytes downloaded
[03:47:29] Initial: 7241; + 122880 bytes downloaded
[03:47:29] Initial: BE2C; + 133120 bytes downloaded
[03:47:29] Initial: 05EB; + 143360 bytes downloaded
[03:47:29] Initial: 962A; + 153600 bytes downloaded
[03:47:30] Initial: DD27; + 163840 bytes downloaded
[03:47:30] Initial: E8AC; + 174080 bytes downloaded
[03:47:30] Initial: BA52; + 184320 bytes downloaded
[03:47:30] Initial: BEDE; + 194560 bytes downloaded
[03:47:30] Initial: 926D; + 204800 bytes downloaded
[03:47:30] Initial: 8263; + 215040 bytes downloaded
[03:47:30] Initial: C9F0; + 225280 bytes downloaded
[03:47:30] Initial: 5B0E; + 235520 bytes downloaded
[03:47:30] Initial: B6D2; + 245760 bytes downloaded
[03:47:30] Initial: 3DFA; + 256000 bytes downloaded
[03:47:30] Initial: DE29; + 266240 bytes downloaded
[03:47:30] Initial: D79C; + 276480 bytes downloaded
[03:47:30] Initial: F712; + 286720 bytes downloaded
[03:47:30] Initial: 7056; + 296960 bytes downloaded
[03:47:30] Initial: 8722; + 307200 bytes downloaded
[03:47:30] Initial: 138A; + 317440 bytes downloaded
[03:47:30] Initial: EE68; + 327680 bytes downloaded
[03:47:30] Initial: 76A7; + 337920 bytes downloaded
[03:47:30] Initial: 2314; + 348160 bytes downloaded
[03:47:30] Initial: FF13; + 358400 bytes downloaded
[03:47:30] Initial: E0D7; + 368640 bytes downloaded
[03:47:30] Initial: 9BC3; + 378880 bytes downloaded
[03:47:30] Initial: 4A6F; + 389120 bytes downloaded
[03:47:30] Initial: DAAF; + 399360 bytes downloaded
[03:47:30] Initial: CF6C; + 409600 bytes downloaded
[03:47:30] Initial: D4CC; + 419840 bytes downloaded
[03:47:30] Initial: E8F2; + 430080 bytes downloaded
[03:47:30] Initial: 783A; + 440320 bytes downloaded
[03:47:30] Initial: 21E4; + 450560 bytes downloaded
[03:47:30] Initial: 656F; + 460800 bytes downloaded
[03:47:30] Initial: C964; + 471040 bytes downloaded
[03:47:30] Initial: 2FEF; + 481280 bytes downloaded
[03:47:31] Initial: D2B9; + 491520 bytes downloaded
[03:47:31] Initial: 4855; + 501760 bytes downloaded
[03:47:31] Initial: B0D7; + 512000 bytes downloaded
[03:47:31] Initial: CFDF; + 522240 bytes downloaded
[03:47:31] Initial: FC70; + 532480 bytes downloaded
[03:47:31] Initial: 8E65; + 542720 bytes downloaded
[03:47:31] Initial: 9DDD; + 552960 bytes downloaded
[03:47:31] Initial: 3C3A; + 563200 bytes downloaded
[03:47:31] Initial: 6000; + 573440 bytes downloaded
[03:47:31] Initial: 219B; + 583680 bytes downloaded
[03:47:31] Initial: 09C2; + 593920 bytes downloaded
[03:47:31] Initial: A23C; + 604160 bytes downloaded
[03:47:31] Initial: 3D4F; + 614400 bytes downloaded
[03:47:31] Initial: 2D24; + 624640 bytes downloaded
[03:47:31] Initial: 32FC; + 634880 bytes downloaded
[03:47:31] Initial: 8212; + 645120 bytes downloaded
[03:47:31] Initial: 58B4; + 655360 bytes downloaded
[03:47:31] Initial: 54A5; + 665600 bytes downloaded
[03:47:31] Initial: 89F1; + 675840 bytes downloaded
[03:47:31] Initial: 62F9; + 686080 bytes downloaded
[03:47:31] Initial: 4BDC; + 696320 bytes downloaded
[03:47:31] Initial: BCEA; + 706560 bytes downloaded
[03:47:31] Initial: 378C; + 716800 bytes downloaded
[03:47:31] Initial: F004; + 727040 bytes downloaded
[03:47:31] Initial: 7F99; + 737280 bytes downloaded
[03:47:31] Initial: 67F0; + 747520 bytes downloaded
[03:47:31] Initial: 3EF8; + 757760 bytes downloaded
[03:47:31] Initial: FD57; + 768000 bytes downloaded
[03:47:31] Initial: 0614; + 778240 bytes downloaded
[03:47:31] Initial: 0489; + 788480 bytes downloaded
[03:47:31] Initial: 8E87; + 798720 bytes downloaded
[03:47:31] Initial: E26F; + 808960 bytes downloaded
[03:47:31] Initial: B950; + 819200 bytes downloaded
[03:47:31] Initial: 91E3; + 829440 bytes downloaded
[03:47:31] Initial: 978A; + 839680 bytes downloaded
[03:47:32] Initial: F23D; + 849920 bytes downloaded
[03:47:32] Initial: 733B; + 860160 bytes downloaded
[03:47:32] Initial: 9201; + 870400 bytes downloaded
[03:47:32] Initial: C0E4; + 880640 bytes downloaded
[03:47:32] Initial: 1032; + 890880 bytes downloaded
[03:47:32] Initial: 59DB; + 901120 bytes downloaded
[03:47:32] Initial: 62FA; + 911360 bytes downloaded
[03:47:32] Initial: 6388; + 921600 bytes downloaded
[03:47:32] Initial: B4E6; + 931840 bytes downloaded
[03:47:32] Initial: 79D2; + 942080 bytes downloaded
[03:47:32] Initial: CC40; + 952320 bytes downloaded
[03:47:32] Initial: 094A; + 962560 bytes downloaded
[03:47:32] Initial: ACD3; + 972800 bytes downloaded
[03:47:32] Initial: C93B; + 983040 bytes downloaded
[03:47:32] Initial: 7223; + 993280 bytes downloaded
[03:47:32] Initial: A946; + 1003520 bytes downloaded
[03:47:32] Initial: 7883; + 1013760 bytes downloaded
[03:47:32] Initial: 0D06; + 1024000 bytes downloaded
[03:47:32] Initial: 0522; + 1034240 bytes downloaded
[03:47:32] Initial: 66A2; + 1044480 bytes downloaded
[03:47:32] Initial: 8242; + 1054720 bytes downloaded
[03:47:32] Initial: CF97; + 1064960 bytes downloaded
[03:47:32] Initial: 5CD9; + 1075200 bytes downloaded
[03:47:32] Initial: B7CE; + 1085440 bytes downloaded
[03:47:32] Initial: 6464; + 1095680 bytes downloaded
[03:47:32] Initial: D78E; + 1105920 bytes downloaded
[03:47:32] Initial: 8A2A; + 1116160 bytes downloaded
[03:47:32] Initial: 0939; + 1126400 bytes downloaded
[03:47:32] Initial: B9AA; + 1136640 bytes downloaded
[03:47:32] Initial: F3C8; + 1146880 bytes downloaded
[03:47:32] Initial: 6362; + 1157120 bytes downloaded
[03:47:32] Initial: 318E; + 1167360 bytes downloaded
[03:47:32] Initial: 508F; + 1177600 bytes downloaded
[03:47:32] Initial: 0F0C; + 1187840 bytes downloaded
[03:47:33] Initial: D3A7; + 1198080 bytes downloaded
[03:47:33] Initial: 9D9D; + 1208320 bytes downloaded
[03:47:33] Initial: 0F12; + 1218560 bytes downloaded
[03:47:33] Initial: A4EC; + 1228800 bytes downloaded
[03:47:33] Initial: D951; + 1239040 bytes downloaded
[03:47:33] Initial: 1964; + 1249280 bytes downloaded
[03:47:33] Initial: D08F; + 1259520 bytes downloaded
[03:47:33] Initial: 7C41; + 1269760 bytes downloaded
[03:47:33] Initial: 6829; + 1280000 bytes downloaded
[03:47:33] Initial: CE47; + 1290240 bytes downloaded
[03:47:33] Initial: 28A2; + 1300480 bytes downloaded
[03:47:33] Initial: 3D91; + 1310720 bytes downloaded
[03:47:33] Initial: 4354; + 1320960 bytes downloaded
[03:47:33] Initial: 45AA; + 1331200 bytes downloaded
[03:47:33] Initial: D858; + 1341440 bytes downloaded
[03:47:33] Initial: ED9D; + 1351680 bytes downloaded
[03:47:33] Initial: 3170; + 1361920 bytes downloaded
[03:47:33] Initial: 66C9; + 1367545 bytes downloaded
[03:47:33] Verifying core Core_15.fah...
[03:47:33] Signature is VALID
[03:47:33]
[03:47:33] Trying to unzip core FahCore_15.exe
[03:47:34] Decompressed FahCore_15.exe (4136960 bytes) successfully
[03:47:39] + Core successfully engaged
[03:47:44]
[03:47:44] + Processing work unit
[03:47:44] Core required: FahCore_15.exe
[03:47:44] Core found.
[03:47:44] Working on queue slot 07 [June 4 03:47:44 UTC]
[03:47:44] + Working ...
[03:47:44] - Calling '.\FahCore_15.exe -dir work/ -suffix 07 -nice 19 -checkpoint 30 -verbos
e -lifeline 35 -version 630'

Warning: Ignoring unknown arg
Warning: Ignoring unknown arg
[03:47:44]
[03:47:44] *------------------------------*
[03:47:44] Folding@Home GPU Core -- Beta
[03:47:44] Version 2.09 (Thu May 20 11:58:42 PDT 2010)
[03:47:44]
[03:47:44] Build host: SimbiosNvdWin7
[03:47:44] Board Type: Nvidia
[03:47:44] Core      :
[03:47:44] Preparing to commence simulation
[03:47:44] - Ensuring status. Please wait.
[03:47:53] - Looking at optimizations...
[03:47:53] - Working with standard loops on this execution.
[03:47:53] - Previous termination of core was improper.
[03:47:53] - Files status OK
[03:47:53] sizeof(CORE_PACKET_HDR) = 512 file=<>
[03:47:53] - Expanded 41690 -> 163067 (decompressed 391.1 percent)
[03:47:53] Called DecompressByteArray: compressed_data_size=41690 data_size=163067, decompre
ssed_data_size=163067 diff=0
[03:47:53] - Digital signature verified
[03:47:53]
[03:47:53] Project: 10627 (Run 50, Clone 59, Gen 0)
[03:47:53]
[03:47:53] Entering M.D.
Reading file work/wudata_07.tpr, VERSION 4.0.5 (single precision)
Note: tpx file_version 58, software version 59
wine: Call from 0x484768 to unimplemented function cudart32_30_14.dll.cudaSetDeviceFlags, ab
orting
Run: exception thrown during GuardedRun
Network already in use.
[03:47:59] Will resume from checkpoint file work/wudata_07.ckp
[03:47:59] Tpr hash work/wudata_07.tpr:  3786786508 4242169704 3844488788 4058928834 3985951
195
[03:47:59] Working on 582 p2750_N68H_AM03
[03:47:59] Client config found, loading data.
[03:47:59] Run: exception thrown during GuardedRun
[03:47:59] Run: exception thrown in GuardedRun -- Gromacs cannot continue further.
[03:47:59] GoingStarting GUI Server
[03:47:59] ave done -- stepsTotalG=0
[03:47:59] Work fraction=0.0000 steps=0.
[03:48:03] ned results file
[03:48:03] - Writing 278728 bytes of core data to disk...
[03:48:04] Done: 278216 -> 15273 (compressed to 5.4 percent)
[03:48:04]   ... Done.
[03:48:04] DeleteFrameFiles: successfully deleted file=work/wudata_07.ckp
[03:48:04]
[03:48:04] Folding@home Core Shutdown: UNSTABLE_MACHINE
[03:48:06] CoreStatus = 7A (122)


Downloaded and tried to run Core15 but it looks like we're missing a function in the wrapper... We are very very close...

Edit: Yea, function was commented out of the wrapper.. Uncommented it and now it doesn't compile.. Hrm...

[jimerickson]

three cheers for sidicas!!

[Sidicas]

Fixed that..
Uncommented a whole bunch more functions that I found..
They didn't compile either..
Some of them were missing wine_ before the function name in the wrapper.. Fixed that... Compiled fine...
One of them didn't match the prototype given in the CUDA Toolkit... Fixed that...

One more error I'm working on and I should be good to go for another test...

[Sidicas]

OK folks,
The GPU3 Linux wrapper is now ready for Testing...


Install recommended drivers: 256 Beta (64-bit drivers used for testing)


ALL users: Must install CUDA 32-bit ToolKit 3.0

Install toolkit 3.0

Edit /etc/ld.so.conf.d/cuda.conf to include:
/usr/local/cuda/lib

Run:
sudo ldconfig

All of the below commands should be executed in the GPU3 client folder.
Cleaning Up The Mess From Earlier Testing

Code: Select all

rm *.so
rm *.dll

Getting Wrapper Files

Code: Select all

wget http://mysite.verizon.net/vzeq1z17/temp/FAH/cufft3/cufft_gpu3.dll.so
wget http://mysite.verizon.net/vzeq1z17/temp/FAH/cudart3/cudart_gpu3.dll.so

Setting Up Symlinks

Code: Select all

ln -s cudart_gpu3.dll.so cudart32_30_14.dll
ln -s cufft_gpu3.dll.so cufft32_30_14.dll
ln -s cudart_gpu3.dll.so cudart.dll
ln -s cudart_gpu3.dll.so nvcuda.dll

Executing Folding@Home
YOU MUST USE EITHER -forcegpu nvidia_g80 OR -forcegpu nvidia_fermi

Code: Select all

wine ./Folding@home-Win32-GPU.exe -forcegpu nvidia_g80 


Tested & Compiled With:
GPU: GTS 250
FAH Client:GPU3
Cores Tested: Both Core11 and Core15 run fine
Kernel: 2.6.26-2-amd64
OS: Debian Lenny 64-bit
Wine: 32-bit 1.0.1 (1.2r2 may be required for GTX400 series users)



KNOWN ISSUES
NONE


Developer Links for compiling it yourself:
cudart made with CUDA ToolKit 3.0 => cudart_gpu3.dll.so

Code: Select all

wget http://mysite.verizon.net/vzeq1z17/temp/FAH/cudart3/cudart.c
wget http://mysite.verizon.net/vzeq1z17/temp/FAH/cudart3/cudart.h
wget http://mysite.verizon.net/vzeq1z17/temp/FAH/cudart3/cudart.dll.spec
wget http://mysite.verizon.net/vzeq1z17/temp/FAH/cudart3/Makefile

cufft made with CUDA ToolKit 3.0 => cufft_gpu3.dll.so

Code: Select all

wget http://mysite.verizon.net/vzeq1z17/temp/FAH/cufft3/cufft.c
wget http://mysite.verizon.net/vzeq1z17/temp/FAH/cufft3/cufft.dll.spec
wget http://mysite.verizon.net/vzeq1z17/temp/FAH/cufft3/Makefile

[jimerickson]

three more cheers for sidicas!!! actually on my machine core 15 only takes 73% of a core...
but it makes my fermi run at 84 degrees C!!
had to use the vista/win7 client. the xp client won't work with the fermi card.
once again many thanks sidicas!! i have been waiting for this for quite awhile.

currently using:
nvidia 256.25
cudatoolkit 3.0
gpu3
gtx480
wine 1.2rc2

[RAH]

Though I have no use for this, I have followed it from the start. Just want to give you guys,
a BIG SALUTE FOR A JOB WELL DONE

[sarkeizen]

I can confirm that sidicas's config works for me my system has the following differences:

Wine 1.2rc2
256.25 BETA
I'm running the 64 bit versions of both CUDA toolkits (ubutnu versions) but I only have the 32 bit versions (i.e. /lib) in ld.so.config
64 bit Gentoo OS
XP GPU3 client

Currently running my GTX480 and my GTS250 simultaneously.
GTX 480 nails about 99% of one of my four cores. The GTS250 only takes about 10% - not sure why of if I'm suffering some penalty because of it.

[Shelnutt2]

You shouldn't need both toolkits, just use 3.0

I also updated my files.

Can you confirm if core_15 takes 100% of a cpu or just core_11? We know core_11 does this, but core_14 didn't, so I'm just wondering if core_15 will or will not require the delay (If it doesn't then we'll phase out the delay when core_11 goes away).

[jimerickson]

sarkeizen

did the xp client work for you or did you have to use the vista client??
nevermind i see it now...

[jimerickson]

Shelnutt2

core 15 takes about 73% of a core on average.
thats on an 8 core system.
with a smp bigadv client running simultaneously.