Installed second video card, but only one is recognized

[bruce]

Have you run memtestg80?

I was going to explain a theory and to tell you how to test it but discovered my own GPU has recently failed. My V7 log looks just like yours does .... a WU is downloaded for the GPU(s), it appears to start, but it never makes any progress, nor does it indicate any errors. It's quite interesting that FAH doesn't report the failure, but memtestg80 does. I'll see about getting this fixed.

[melephant]

7im or Bruce will solve this or get the code rewritten. I have seen them help before.

[melephant]

MemtestG80 produced no errors thank God.

[bruce]

Ok. Glad to hear it.

Reboot and restart FAH. That will reinitialize the log file.

In the log you posted, the SMP was spending about 4 minutes per percent. Wait 15 minutes or so until the log shows several lines showing increasing percentages.
If you're set to Novice mode (upper right hand corner of FAHControl), change it to Advanced.

In the center of the screen, under the Status tab, you'll see a section called Folding Slots and I believe you'll see one SMP slot and two GPU slots. Right-click on the SMP slot and choose Pause. (No, don't use the Pause button at the top.)
In the section below that called Work Queue, you should have seen three entries that said "Running" and now one of them will change to "Ready"

Wait half an hour and click either Fold function to resume work on the SMP Project.

Wait another 15 minutes or so and post the log file just like you did earlier.

[melephant]

Rebooted and started FAH. Started at 6% waited 13 minutes until SMP reached 12%. Right-clicked on SMP and clicked paused. SMP changed to ready. Waited 30 minutes. Clicked Fold on top. Waited 15 minutes then sent log. Log was reloaded before sending? Was that necesary? I noticed my percentage progress went back to 6%. Could that be because I have a lenghty checkpoint time? Is there a setting in which I can configure the forum to send an email to me when a reply has posted? That would speed up the process a lot. Is a solution available after reading this log? I assume I have no not made any setup errors if your cards malfunctioned too. Thanks tremndously for your help. I plan on buying 6 GTX 680 Hydrocopper's by June if this is operational.

Code: Select all

       *********************** Log Started 2011-11-14T18:14:00 ************************
18:14:00:************************* Folding@home Client *************************
18:14:00:      Website: http://folding.stanford.edu/
18:14:00:    Copyright: (c) 2009-2011 Stanford University
18:14:00:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:14:00:         Args: --lifeline 2740 --command-port=36330
18:14:00:       Config: C:/Users/Mel/AppData/Roaming/FAHClient/config.xml
18:14:00:******************************** Build ********************************
18:14:00:      Version: 7.1.38
18:14:00:         Date: Oct 6 2011
18:14:00:         Time: 19:57:04
18:14:00:      SVN Rev: 3080
18:14:00:       Branch: fah/trunk/client
18:14:00:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
18:14:00:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
18:14:00:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT
18:14:00:     Platform: win32 XP
18:14:00:         Bits: 32
18:14:00:         Mode: Release
18:14:00:******************************* System ********************************
18:14:00:          CPU: Intel(R) Core(TM) i7 CPU 980 @ 3.33GHz
18:14:00:       CPU ID: GenuineIntel Family 6 Model 44 Stepping 2
18:14:00:         CPUs: 12
18:14:00:       Memory: 11.99GiB
18:14:00:  Free Memory: 10.33GiB
18:14:00:      Threads: WINDOWS_THREADS
18:14:00:   On Battery: false
18:14:00:   UTC offset: -6
18:14:00:          PID: 4812
18:14:00:          CWD: C:/Users/Mel/AppData/Roaming/FAHClient
18:14:00:           OS: Windows 7 Home Premium
18:14:00:      OS Arch: AMD64
18:14:00:         GPUs: 2
18:14:00:        GPU 0: NVIDIA:1 G94 [GeForce 9600 GT]
18:14:00:        GPU 1: FERMI:1 GF100 [GeForce GTX 480]
18:14:00:         CUDA: 2.0
18:14:00:  CUDA Driver: 4010
18:14:00:Win32 Service: false
18:14:00:***********************************************************************
18:14:00:<config>
18:14:00:  <!-- Folding Slot Configuration -->
18:14:00:  <gpu v='true'/>
18:14:00:
18:14:00:  <!-- User Information -->
18:14:00:  <passkey v='********************************'/>
18:14:00:  <team v='12867'/>
18:14:00:  <user v='Mel_Stark'/>
18:14:00:
18:14:00:  <!-- Folding Slots -->
18:14:00:</config>
18:14:00:Trying to access database...
18:14:00:Successfully acquired database lock
18:14:00:Enabled folding slot 00: READY gpu:0:"G94 [GeForce 9600 GT]"
18:14:00:Enabled folding slot 01: READY gpu:1:"GF100 [GeForce GTX 480]"
18:14:00:Enabled folding slot 02: READY smp:12
18:14:00:Starting Unit 00
18:14:00:Running core: C:/Users/Mel/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 00 -suffix 01 -lifeline 4812 -version 701 -checkpoint 15 -gpu 0
18:14:00:Started core on PID 5040
18:14:00:FahCore 0x11 started
18:14:00:Starting Unit 01
18:14:00:Running core: C:/Users/Mel/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 01 -suffix 01 -lifeline 4812 -version 701 -checkpoint 15 -gpu 1
18:14:00:Started core on PID 5056
18:14:00:FahCore 0x15 started
18:14:00:Starting Unit 02
18:14:00:Running core: C:/Users/Mel/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 02 -suffix 01 -lifeline 4812 -version 701 -checkpoint 15 -np 12
18:14:00:Started core on PID 5076
18:14:00:FahCore 0xa4 started
18:14:00:Unit 00:
18:14:00:Unit 00:*------------------------------*
18:14:00:Unit 00:Folding@Home GPU Core
18:14:00:Unit 00:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
18:14:00:Unit 00:
18:14:00:Unit 00:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
18:14:00:Unit 00:Build host: amoeba
18:14:00:Unit 00:Board Type: Nvidia
18:14:00:Unit 00:Core      :
18:14:00:Unit 00:Preparing to commence simulation
18:14:00:Unit 00:- Ensuring status. Please wait.
18:14:01:Unit 01:
18:14:01:Unit 01:*------------------------------*
18:14:01:Unit 01:Folding@Home GPU Core
18:14:01:Unit 01:Version                2.20 (Tue Aug 2 12:06:37 PDT 2011)
18:14:01:Unit 01:Build host             SimbiosNvdWin7
18:14:01:Unit 01:Board Type             NVIDIA/CUDA
18:14:01:Unit 01:Core                   15
18:14:01:Unit 01:GPU device id          1
18:14:01:Unit 01:
18:14:01:Unit 01:Window's signal control handler registered.
18:14:01:Unit 01:Preparing to commence simulation
18:14:01:Unit 01:- Ensuring status. Please wait.
18:14:01:Unit 02:
18:14:01:Unit 02:*------------------------------*
18:14:01:Unit 02:Folding@Home Gromacs GB Core
18:14:01:Unit 02:Version 2.27 (Dec. 15, 2010)
18:14:01:Unit 02:
18:14:01:Unit 02:Preparing to commence simulation
18:14:01:Unit 02:- Ensuring status. Please wait.
18:14:03:Server connection id=1 on 0.0.0.0:36330 from 127.0.0.1
18:14:10:Unit 00:- Looking at optimizations...
18:14:10:Unit 00:- Working with standard loops on this execution.
18:14:10:Unit 00:- Previous termination of core was improper.
18:14:10:Unit 00:- Going to use standard loops.
18:14:10:Unit 00:- Files status OK
18:14:10:Unit 00:- Expanded 46755 -> 252912 (decompressed 540.9 percent)
18:14:10:Unit 00:Called DecompressByteArray: compressed_data_size=46755 data_size=252912, decompressed_data_size=252912 diff=0
18:14:10:Unit 00:- Digital signature verified
18:14:10:Unit 00:
18:14:10:Unit 00:Project: 5765 (Run 1, Clone 175, Gen 83)
18:14:10:Unit 00:
18:14:10:Unit 00:Entering M.D.
18:14:10:Unit 01:- Looking at optimizations...
18:14:10:Unit 02:- Looking at optimizations...
18:14:10:Unit 01:- Working with standard loops on this execution.
18:14:10:Unit 02:- Working with standard loops on this execution.
18:14:10:Unit 01:- Previous termination of core was improper.
18:14:10:Unit 02:- Previous termination of core was improper.
18:14:10:Unit 01:- Going to use standard loops.
18:14:10:Unit 02:- Going to use standard loops.
18:14:10:Unit 01:- Files status OK
18:14:10:Unit 02:- Files status OK
18:14:10:Unit 01:sizeof(CORE_PACKET_HDR) = 512 file=<>
18:14:10:Unit 02:- Expanded 1354037 -> 2552108 (decompressed 188.4 percent)
18:14:10:Unit 01:- Expanded 45149 -> 171827 (decompressed 380.5 percent)
18:14:10:Unit 02:Called DecompressByteArray: compressed_data_size=1354037 data_size=2552108, decompressed_data_size=2552108 diff=0
18:14:10:Unit 01:Called DecompressByteArray: compressed_data_size=45149 data_size=171827, decompressed_data_size=171827 diff=0
18:14:10:Unit 02:- Digital signature verified
18:14:10:Unit 01:- Digital signature verified
18:14:10:Unit 02:
18:14:10:Unit 01:
18:14:10:Unit 02:Project: 7905 (Run 120, Clone 10, Gen 0)
18:14:10:Unit 01:Project: 6801 (Run 8806, Clone 0, Gen 85)
18:14:10:Unit 02:
18:14:10:Unit 01:
18:14:10:Unit 02:Entering M.D.
18:14:10:Unit 01:Entering M.D.
18:14:12:Unit 01:Tpr hash 01/wudata_01.tpr:  651012101 731616396 4062098411 4001114977 2458156469
18:14:12:Unit 01:calling fah_main gpuDeviceId=1
18:14:12:Unit 01:Working on ALZHEIMER'S DISEASE AMYLOID
18:14:12:Unit 01:Client config unavailable.
18:14:12:Unit 01:Starting GUI Server
18:14:16:Unit 00:Tpr hash 00/wudata_01.tpr:  2462480482 4159901929 4222575972 2259098210 366859908
18:14:16:Unit 00:
18:14:16:Unit 00:Calling fah_main args: 14 usage=100
18:14:16:Unit 00:
18:14:16:Unit 02:Using Gromacs checkpoints
18:14:16:Unit 02:Mapping NT from 12 to 12
18:14:17:Unit 02:Resuming from checkpoint
18:14:17:Unit 02:Verified 02/wudata_01.log
18:14:17:Unit 02:Verified 02/wudata_01.trr
18:14:17:Unit 02:Verified 02/wudata_01.edr
18:14:17:Unit 02:Completed 32540 out of 500000 steps  (6%)
18:15:35:Unit 02:Completed 35000 out of 500000 steps  (7%)
18:17:59:Unit 02:Completed 40000 out of 500000 steps  (8%)
18:20:24:Unit 02:Completed 45000 out of 500000 steps  (9%)
18:22:48:Unit 02:Completed 50000 out of 500000 steps  (10%)
18:25:15:Unit 02:Completed 55000 out of 500000 steps  (11%)
18:27:38:Unit 02:Completed 60000 out of 500000 steps  (12%)
18:27:59:Slot 02 paused
18:28:00:Slot 02: shutting core down
18:28:00:WARNING: FahCore is known to not shutdown cleanly, killing
18:28:00:WARNING: Killing Unit 02
18:28:00:FahCore, running Unit 02, terminated.
18:59:44:Slot 02 unpaused
18:59:44:Starting Unit 02
18:59:44:Running core: C:/Users/Mel/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 02 -suffix 01 -lifeline 4812 -version 701 -checkpoint 15 -np 12
18:59:44:Started core on PID 4380
18:59:44:FahCore 0xa4 started
18:59:44:Unit 02:
18:59:44:Unit 02:*------------------------------*
18:59:44:Unit 02:Folding@Home Gromacs GB Core
18:59:44:Unit 02:Version 2.27 (Dec. 15, 2010)
18:59:44:Unit 02:
18:59:44:Unit 02:Preparing to commence simulation
18:59:44:Unit 02:- Ensuring status. Please wait.
18:59:54:Unit 02:- Looking at optimizations...
18:59:54:Unit 02:- Working with standard loops on this execution.
18:59:54:Unit 02:- Previous termination of core was improper.
18:59:54:Unit 02:- Going to use standard loops.
18:59:54:Unit 02:- Files status OK
18:59:54:Unit 02:- Expanded 1354037 -> 2552108 (decompressed 188.4 percent)
18:59:54:Unit 02:Called DecompressByteArray: compressed_data_size=1354037 data_size=2552108, decompressed_data_size=2552108 diff=0
18:59:54:Unit 02:- Digital signature verified
18:59:54:Unit 02:
18:59:54:Unit 02:Project: 7905 (Run 120, Clone 10, Gen 0)
18:59:54:Unit 02:
18:59:54:Unit 02:Entering M.D.
19:00:00:Unit 02:Using Gromacs checkpoints
19:00:00:Unit 02:Mapping NT from 12 to 12
19:00:00:Unit 02:Resuming from checkpoint
19:00:00:Unit 02:Verified 02/wudata_01.log
19:00:00:Unit 02:Verified 02/wudata_01.trr
19:00:00:Unit 02:Verified 02/wudata_01.edr
19:00:00:Unit 02:Completed 32540 out of 500000 steps  (6%)
19:01:10:Unit 02:Completed 35000 out of 500000 steps  (7%)
19:03:33:Unit 02:Completed 40000 out of 500000 steps  (8%)
19:05:56:Unit 02:Completed 45000 out of 500000 steps  (9%)
19:08:27:Saving configuration to config.xml
19:08:27:<config>
19:08:27:  <!-- FahCore Control -->
19:08:27:  <checkpoint v='3'/>
19:08:27:
19:08:27:  <!-- Folding Slot Configuration -->
19:08:27:  <gpu v='true'/>
19:08:27:
19:08:27:  <!-- Network -->
19:08:27:  <proxy v=':8080'/>
19:08:27:
19:08:27:  <!-- User Information -->
19:08:27:  <passkey v='********************************'/>
19:08:27:  <team v='12867'/>
19:08:27:  <user v='Mel_Stark'/>
19:08:27:
19:08:27:  <!-- Folding Slots -->
19:08:27:  <slot id='0' type='GPU'/>
19:08:27:  <slot id='1' type='GPU'/>
19:08:27:  <slot id='2' type='SMP'/>
19:08:27:</config>
19:08:31:Unit 02:Completed 50000 out of 500000 steps  (10%)
19:10:55:Unit 02:Completed 55000 out of 500000 steps  (11%)
19:13:18:Unit 02:Completed 60000 out of 500000 steps  (12%)
19:16:06:Unit 02:Completed 65000 out of 500000 steps  (13%)

Mod Edit: Added Code Tags - PantherX

[bruce]

Well, I had a theory about what was wrong, but your log shows my theory was wrong. I'm stumped, (or, as they say in engineering circles: "Back to the drawing board") but I'll keep thinking about it.

[melephant]

Any chance the code can be rewritten on this issue for a future version update?

[bruce]

And just how would you suggest the code should be rewritten?

V7 is doing exactly what it should be doing. It's creating slots based on the hardware that is identified and it's downloading WUs for those slots. Once a WU is downloaded, it's up to FahCore_11 to process WUs assigned to NVidia non-Fermi assignments and it's up to FahCore_15 to process Fermi assignments. Beyond that point it's no longer a client issue.

The options remaining are
A) Bad hardware
B) Bad drivers
C) Bad WUs or FahCores.

We've already discussed A and B. It seems rather unlikely that you'd be assigned two (or more?) bad WUs that would fail in the exact same way. The FahCores have been working fine for everyone else who have nVidia GPUs, whether they run V6 or V7.

You did say you reinstalled V7, but apparently you retained the data. The only thing I can think of that you didn't try was to discard the data which would cause your client to re-download WUs and FahCores. You can "Finish" the current SMP WU and then discard the data. You'll either need to save config.xml somewhere external to the data directory and restore it manually prior to starting the reinstalled client or you'll have to re-enter all the setup options.

[7im]

Uninstall everything, including the data, and then reinstall, using the (GPU+SMP) option.

And then post the log again.


Another problem we might be running in to at this point.... if you've had several different GPUs in this one system, you will likely have to run a Driver Cleaner utility to clean out the registry. Then install the latest GPU drivers again. That may be why we don't see the CUDA 2.1.

[melephant]

Bruce, Where can I find the data directory? When I uninstall it removes everything from program files and program files (x86).
7im, The driver cleaner will have to wait until I get paid in a few days. The log will follow after I find the data and delete it like Bruce suggested. I still have a WU running.

[codysluder]

Bruce must be busy with RL, but you can find that information two ways. FAHClient installs a shortcut to the "data directory" under FAHClient in the All Programs directory. You can also find it in the log that you posted:

*********************** Log Started 2011-11-14T18:14:00 ************************
18:14:00:************************* Folding@home Client *************************
18:14:00: Website: http://folding.stanford.edu/
18:14:00: Copyright: (c) 2009-2011 Stanford University
18:14:00: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:14:00: Args: --lifeline 2740 --command-port=36330
18:14:00: Config: C:/Users/Mel/AppData/Roaming/FAHClient/config.xml

There are more files that just config.xml in that directory but to do what he asked, they should all be discarded (except perhaps config.xml, itself.

Actually, whenever you install, if the installer detects a previous version it asks if you want to uninstall. All you have to do is click the box telling it NOT to keep the data.

[7im]

There are Driver Cleaner utils that are free, at least the last time I checked. Maybe someone familiar with a good free cleaner can make a few recommendations.

[fredfoobar]

I'd say you have an identical problem to what I just had, so let me post how I solved my problem:

1. Run cmd.
2. Run this command:
   

   Code: Select all

   "C:\Program Files (x86)\FAHClient\FAHClient.exe" --lspci

   
   
3. Take down the order in which your graphics cards appear. Call this List A.
4. Download GPU-Z and run it.
5. Look at the dropdown at the very bottom of the GPU-Z window.
6. Make a note of the order in which the graphics cards appear. This might be different from List A. Call this List B.
7. Open FAHControl, and click Configure. Go to the Slots tab.
8. For each GPU slot:
   
   1. Select a GPU you want in that slot.
   2. Click the slot, then click Edit.
   3. Set the gpu-index field to the position of the GPU in List A, starting from 0. (The first card on the list will be 0, the second will be 1, etc.)
   4. Set the opencl-index and cuda-index fields equal to the position of the GPU in List B, starting from 0.
   5. Click OK. Repeat for each card.
9. Click Save.
10. Click Pause in FAHControl, and wait for all slots to read "Paused".
11. Click Quit in FAHControl.
12. Restart FAHControl. You should now be up and running.

7im, feel free to put this in the FAQ somewhere. The symptom is hanging WUs, and it only seems to happen with mixed Fermi/non-Fermi GPUs. The order in which GPUs are being configured by FAHControl (and thus the GPU ID FAHClient passes to the cores) is not equal to the actual CUDA/OpenCL GPU ID, so Fermi WUs are trying to run on non-Fermi GPUs, and vice-versa. Hopefully this will be fixed in a future release - ensuring that GPUs are enumerated in CUDA/OpenCL ID order should do the trick.

Edit by Bruce: I'm editing this post since a reply will now be on the next page.

Set the opencl-index and cuda-index fields equal to the position of the GPU in List B, starting from 0.

This does not always work. (If the two values were always equal, Development would not have created two values.)

Suppose you have several CUDA-capable GPUs and several OpenCL-capable GPUs and some of them are only supported by one of those choices. The list of CUDA-capable GPUs isn't necessarily the same length as the list of OpenCL-capable GPUs but the numbers in each list will start from 0 and increase by consecutive values without any connection to the position of that GPU in the other list.

Suppose your first GPU is an AMD. It's not on the CUDA list but it's opencl-index is probably zero. Suppose the next GPU is NVidia. Its cuda-index will be 0 and it will probably have an opencl-index 1. This can change if you install OpenCL drivers for Intel.

[7im]

This needs to be fixed in the client, not put in an FAQ.

But this is very FAQ worthy, and I'm sure we'll refer to it often, at least until the client is fixed.

[melephant]

Both cards are up and working. I found that the last step (Quitting) was causing my slot ID 1 to constantly revert to -1 -1 on the bottom to two configurations and my slot ID 1 would not fold. So, I tried pausing and not quitting and bingo - success. Thank you guys!!!!!