Configuring Multiple/Additional GPUs [Continuous Giude]

[DownshiftArtist]

ISSUE:

GPU detection and config issues on some multiple GPU systems still aren't fixed.

I started with an issue where the client could not correctly identify which card was what. It would somehow fold on the GTX 260 while the GTX 570 was selected and vice-versa. I found help on the forums about setting the indices and I was able to correct the config to be able to fold on the correct GPU (included in the walkthrough below). The problem I'm having now is that the GPU index ALWAYS reverts back to -1 every time I restart the client for any reason wiping out any I've tried manually editing the config file, but it just overwrites what I do as soon as folding starts. And I'm using 7.4.4 if that makes any difference.

THE SOLUTION THAT WORKED FOR ME

Note: This may not work for you, or may need some modification to work for you, if you have AMD GPUs, Mixed AMD & NVidia GPUs, or certain OpenCL Drivers. This is not a guaranteed 'one-size-fits-all' solution but a guide that may give you alternative you have not tried. We will continue to update the guide as new information is presented in an effort to assist as many as possible. Thank you.

WALKTHROUGH

1. Make sure to do a clean removal and re-install first to see if this will fix the issue. If not proceed to step 2.

2. Install and/or open GPU-Z and make a note of the order of the GPUs in the drop down list at the bottom, 0 for the first GPU, 1 for the second and so on; this can be 'List A'

GPU-Z, for "List A". GTX 570 will be 0, and the 260 will be 1



3. Use command prompt (Start button, 'cmd' in the search bar and press enter) to run C:\Program Files (x86)\FAHClient\fachclient.exe --lspci and note the order the GPUs come up on this list; 'List B'


C:\Program Files (x86)\FAHClient\fachclient.exe --lspci

Code: Select all

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\Family>cd\

C:\>cd program files (x86)

C:\Program Files (x86)>cd fahclient

C:\Program Files (x86)\FAHClient>fahclient.exe --lspci
VendorID:DeviceID:Vendor Name:Description
0x104c:0x8023:Texas Instruments:Texas Instruments 1394 OHCI Compliant Host Contr
oller
0x1095:0x3114:Silicon Image, Inc.:Silicon Image SiI 3114 SoftRaid 5 Controller
0x10de:0x0360:NVIDIA Corporation:PCI standard ISA bridge
0x10de:0x0368:NVIDIA Corporation:NVIDIA nForce PCI System Management
0x10de:0x0369:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x036c:NVIDIA Corporation:Standard OpenHCD USB Host Controller
0x10de:0x036d:NVIDIA Corporation:Standard Enhanced PCI to USB Host Controller
0x10de:0x036e:NVIDIA Corporation:Standard Dual Channel PCI IDE Controller
0x10de:0x0370:NVIDIA Corporation:PCI standard PCI-to-PCI bridge
0x10de:0x0371:NVIDIA Corporation:High Definition Audio Controller
0x10de:0x0373:NVIDIA Corporation:NVIDIA nForce Networking Controller
0x10de:0x0373:NVIDIA Corporation:NVIDIA nForce Networking Controller
0x10de:0x0376:NVIDIA Corporation:PCI Express standard Root Port
0x10de:0x037f:NVIDIA Corporation:NVIDIA nForce Serial ATA Controller
0x10de:0x037f:NVIDIA Corporation:NVIDIA nForce Serial ATA Controller
0x10de:0x037f:NVIDIA Corporation:NVIDIA nForce Serial ATA Controller
0x10de:0x03a1:NVIDIA Corporation:PCI standard host CPU bridge
0x10de:0x03a8:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03a9:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03aa:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03ab:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03ac:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03ad:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03ae:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03af:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b0:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b1:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b2:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b3:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b4:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b5:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b6:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b7:NVIDIA Corporation:PCI Express standard Root Port
0x10de:0x03ba:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03bc:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x05b1:NVIDIA Corporation:PCI Express standard Downstream Switch Port
0x10de:0x05b1:NVIDIA Corporation:PCI Express standard Downstream Switch Port
0x10de:0x05b1:NVIDIA Corporation:PCI Express standard Downstream Switch Port
0x10de:0x05e2:NVIDIA Corporation:GT200 [GeForce GTX 260]
0x10de:0x0e09:NVIDIA Corporation:High Definition Audio Controller
0x10de:0x1081:NVIDIA Corporation:GF110 [GeForce GTX 570]

C:\Program Files (x86)\FAHClient>

0x10de:0x05e2:NVIDIA Corporation:GT200 [GeForce GTX 260]
0x10de:0x0e09:NVIDIA Corporation:High Definition Audio Controller
0x10de:0x1081:NVIDIA Corporation:GF110 [GeForce GTX 570]

This is what you're looking for, and this will be "List B". In this case, the 260 will be 0 and the 570 will be 1

4. Open F@H Client and clear all the automatically assigned slots for the GPUs

5. Create new GPU slots and set the each GPU index according to "list B', and the core indices according to 'list A'. Do this for all GPUs present
Ex. gpu-index = 0, CUDA-index = 1, OpenCL-index = 1 will fold on GTX 260



6. For any GPU you DO NOT want to fold, under 'Extra Slot Options' add Name: paused Value: true

7. Save your configuration then quit and restart your client to verify that config has been saved properly.

After you set up the GPU configs correctly, the config file should look something like this

Code: Select all

23:37:52:<config>
23:37:52:  <!-- Network -->
23:37:52:  <proxy v=':8080'/>
23:37:52:
23:37:52:  <!-- Slot Control -->
23:37:52:  <power v='full'/>
23:37:52:
23:37:52:  <!-- User Information -->
23:37:52:  <passkey v='********************************'/>
23:37:52:  <team v='37726'/>
23:37:52:  <user v='DownshiftArtist'/>
23:37:52:
23:37:52:  <!-- Folding Slots -->
23:37:52:  <slot id='0' type='GPU'>
23:37:52:    <cuda-index v='1'/>
23:37:52:    <gpu-index v='0'/>
23:37:52:    <opencl-index v='1'/>
23:37:52:    <paused v='true'/>
23:37:52:  </slot>
23:37:52:  <slot id='2' type='GPU'>
23:37:52:    <client-type v='beta'/>
23:37:52:    <cuda-index v='0'/>
23:37:52:    <gpu-index v='1'/>
23:37:52:    <opencl-index v='0'/>
23:37:52:  </slot>
23:37:52:</config>


edit: NOTE: Your slot id may be different as it is based on whatever number your client assigns to it.


8. Clear any left over WUs if they are any (which, if done correctly there shouldn't be) and FOLD!!

Added 5/23/14:

... my experience with a 2 and a 3 GPU system is that FAHControl simply lists the GPUs inside out.
When I manually assign the first GPU the openl-index/cuda-index 2, the second GPU openl-index/cuda-index 1 and the third GPU openl-index/cuda-index 0 everything is correct.
In a 2 GPU system the assignment for the first GPU with openl-index/cuda-index 1 and for the second GPU with openl-index/cuda-index 0 solves the problem.

Heiko

[7im]

GPU detection and config issues on multiple GPU systems still aren't fixed.

Sorry, you'll have to provide a little more detail. It's a different story if you add a card versus install with both cards present.

[7im]

Having a similar issue, but I I fixed the config for the slot. The problem I'm having now is that the GPU index ALWAYS reverts back to -1 every time I restart the client for any reason. I've tried manually editing the config file, but it just overwrites what I do as soon as folding starts. ANd I'm using 7.4.4 if that makes any difference.

What instructions say to change the GPU index?

[DownshiftArtist]

Starting folding with only one card present. Installed a second card, which somehow took over the '0' spot as far as the client was concerned, despite it showing up as the second card in everything else including GPU-Z. Did a clean removal and reinstall, client still can't see them correctly. Thanks to the help I received in another thread, I manually configured the slot to use the card I wanted. Everything works great until client needs to be restarted for any reason after which it defaults back to '-1' status and grabs the wrong card. Doesn't seem to make any difference if I manually edit the config file; I tried adding in a line <gpu-index v='1'/> but as soon as I click on fold and just overwrites the changed config file. I searched and found a thread about this issue on 7.4.2 but it ended with "problem solved in new version, 7.4.3" but that wasn't the case either because I also tried using .3 when the issues first began and it was the same.

below is my current config file:

<config>
<!-- Network -->
<proxy v=':8080'/>

<!-- Slot Control -->
<power v='full'/>

<!-- User Information -->
<passkey v='9b28b98b95984a7******************'/>
<team v='37726'/>
<user v='DownshiftArtist'/>

<!-- Folding Slots -->
<slot id='1' type='GPU'>
<client-type v='beta'/>
<cuda-index v='0'/>
<opencl-index v='0'/>
</slot>
</config>

[DownshiftArtist]

Having a similar issue, but I I fixed the config for the slot. The problem I'm having now is that the GPU index ALWAYS reverts back to -1 every time I restart the client for any reason. I've tried manually editing the config file, but it just overwrites what I do as soon as folding starts. ANd I'm using 7.4.4 if that makes any difference.

What instructions say to change the GPU index?

<gpu-index v='1'/> or so I've read. That was supposed to tell the client which gpu index to use but it seems to just be ignored and erased.

[7im]

Follow this: https://foldingforum.org/viewtopic.php?p=199379#p199379

Also note that default settings are not shown in the log by default. If the default gpu-index is already 1, adding that setting will not cause it to appear in the log.

[P5-133XL]

The client automatically removes any settings that are the same as the default. So if -1 produces a gpu-index of 0 for a specific slot then inserting gpu-index = 0 into that slot will always be removed. My guess is that you need to do more fiddling to find the value(s) that actually work.

For multiple GPU's, there are 3 settings you need to be concerned with when manually assigning those values. gpu-index, cuda-index, and opencl-index. -1 means and choose and they all start with 0 when being enumerated. It is hard for me to give you values since I do not know the HW but if you are systematic you can probably figure out a set of working values for each of your slots/GPU's that do not disappear and do work. Do not duplicate values between slots for you may end up with a single WU that is being run by multiple slots which will corrupt the WU or a single slot working multiple WU's drop your PPD precipitously.

[DownshiftArtist]

You need to make the changes in Fahcontrol and save it there.

Also, could we get the full output from lspci? This will write the data to that text file.

Fahclient.exe --lspci > lspci.txt

I always save after making the changes in the client but it doesn't save the gpu-index. I tried deleting the slot and creating a new one with the GPU index set to 1 but no matter what I do it changes back to -1.

here's the lspci output:

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.



C:\Users\Family>cd\

C:\>cd program files (x86)


C:\Program Files (x86)>cd FAHclient


C:\Program Files (x86)\FAHClient>fahclient.exe --lspci
VendorID:DeviceID:Vendor Name:Description
0x104c:0x8023:Texas Instruments:Texas Instruments 1394 OHCI Compliant Host Contr
oller
0x1095:0x3114:Silicon Image, Inc.:Silicon Image SiI 3114 SoftRaid 5 Controller
0x10de:0x0360:NVIDIA Corporation:PCI standard ISA bridge
0x10de:0x0368:NVIDIA Corporation:NVIDIA nForce PCI System Management
0x10de:0x0369:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x036c:NVIDIA Corporation:Standard OpenHCD USB Host Controller
0x10de:0x036d:NVIDIA Corporation:Standard Enhanced PCI to USB Host Controller
0x10de:0x036e:NVIDIA Corporation:Standard Dual Channel PCI IDE Controller
0x10de:0x0370:NVIDIA Corporation:PCI standard PCI-to-PCI bridge
0x10de:0x0371:NVIDIA Corporation:High Definition Audio Controller
0x10de:0x0373:NVIDIA Corporation:NVIDIA nForce Networking Controller
0x10de:0x0373:NVIDIA Corporation:NVIDIA nForce Networking Controller
0x10de:0x0376:NVIDIA Corporation:PCI Express standard Root Port
0x10de:0x037f:NVIDIA Corporation:NVIDIA nForce Serial ATA Controller
0x10de:0x037f:NVIDIA Corporation:NVIDIA nForce Serial ATA Controller
0x10de:0x037f:NVIDIA Corporation:NVIDIA nForce Serial ATA Controller
0x10de:0x03a1:NVIDIA Corporation:PCI standard host CPU bridge
0x10de:0x03a8:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03a9:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03aa:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03ab:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03ac:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03ad:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03ae:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03af:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b0:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b1:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b2:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b3:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b4:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b5:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b6:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03b7:NVIDIA Corporation:PCI Express standard Root Port
0x10de:0x03ba:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x03bc:NVIDIA Corporation:PCI standard RAM Controller
0x10de:0x05b1:NVIDIA Corporation:PCI Express standard Downstream Switch Port
0x10de:0x05b1:NVIDIA Corporation:PCI Express standard Downstream Switch Port
0x10de:0x05b1:NVIDIA Corporation:PCI Express standard Downstream Switch Port
0x10de:0x05e2:NVIDIA Corporation:GT200 [GeForce GTX 260]
0x10de:0x0e09:NVIDIA Corporation:High Definition Audio Controller
0x10de:0x1081:NVIDIA Corporation:GF110 [GeForce GTX 570]

C:\Program Files (x86)\FAHClient>

[7im]

I don't recall asking for the LSPCI output, but thanks. So if I get this right, you are trying to set the 570 as gpu index 1?

[DownshiftArtist]

Yes sir. If I set the GPU index to 1 and both core indices to 0 it folds on the 570. Otherwise it tries to fold on the the 260. If I leave everything at -1, selecting the "570" causes the client to fold on the 260.

[P5-133XL]

Do you have a second slot for the 260? The GPU-index enumerates the slots. With only one slot, the only valid value is 0 or -1.

What you need to be doing is adjusting cuda-index and opencl-index (probably to 1) and leave gpu-index at 0 or -1.

[DownshiftArtist]

No, actually I don't have a slot for the 260. It had one originally when I installed the client but since I didn't want it to use that GPU i removed that slot. Do you think creating a new slot for the 260 and just setting it as inactive would work?

[P5-133XL]

That may work, but it also makes manually assigning values more complex because there are more possible values but that still may be necessary so keep it in reserve. KISS principle or Ocam's razor! Try my suggestion of assigning gpu-index = -1 (or 0), cuda-index = 1 and opencl-index = 1 first.

[DownshiftArtist]

Will do. I'm going to wait until after I finish my current WU, which looks like won't be until early tomorrow AM. So I'll try it out first thing tomorrow and let you know how it goes. Thanks.

[bruce]

The GPUs must match the WUs that have been downloaded. Under normal circumstances, if you uninstall and reinstall everything, including any unfinsihed WUs (best to FINISH them first) there's nothing wrong with the slots being reconfigured in the opposite order. The freshly downloaded WUs will then match whichever GPU has been configured for that slot.

In other words, the only circumstances where I've seen this sort of thing with the latest client is when a second GPU is added after the first one is running. This has a 50% chance of messing up the order that V7 would have chosen if it had set up a fresh configuration with no history.

I don't remember seeing a warning about this and we should probably have one somewhere saying that the FAH client must have a clean reinstal if GPUs are added or replaced with a different type of device. The fundamental problem is that wherever that warning is placed, you wouldn't have seen it.