Possible to 'rotate' WUs to different GPU?

[foldinghomealone2]

Hi there,

Right now I'm using a 1080 and a 1080 Ti.
It happens that a 'big' WU like 11409 is assigned to the 1080 and a 'small' WU like 94xx is assigned to the 1080 Ti.
I would make around 1.7 to 1.8 Mio PPD.

My experience is that if it is the other way round, meaning the 1080 gets the small WU and the 1080 Ti gets the big WU then I would make around 2 Mio PPD.
(Performance increase for big WUs on a 1080 Ti is much more than the performance loss with a small on the 1080).

According to PG's philosophy, more PPD mean more scientific value, isn't it?
Therefore it would be useful to have the possibility to rotate the WUs to the other GPUs in order to get always the max PPD.
(edit: rotation of WUs only in my own rig, no assignment servers or uploads needed)

Is there a way that I can rotate the WUs manually? Will there be a feature planned for future client versions?

Thank you.

[ChristianVirtual]

Unlikely, as it has an easy potential for cherry picking ... if done on donor’s side.

They only way might be if In the backend /assignment server some setting would react on optimized assignments; but not sure what in detail is possible and feasible.

Practically: don’t worry too much about it ; the smaller WUs also need to get done and the scientific impact here seems rather small

[foldinghomealone2]

Sure, cherrypicking is not my intention.
I was just interested if it can be done or there are some restriction once a WU has been assigned.
An I assume that performance increase can be high depending on rig configuration.

I mean only rotation of WUs within my own rig. No additional server assignment necessary. Or uploads of half-finished WUs or so.
'Only' a function in the client software to optimize the assignment of the rig's assignment internally.

[bruce]

First, we have to assume that the two GPUs are equivalent in FAH's eyes. (That's not always obvious but my guess would be that it's a pretty safe bet that the 1080 and the 1080Ti are functionally equivalent.)

Your pair of WUs are in different slots, so your question can be restated as: Is it possible to reconfigure FAHClient so that the hardware associated with slot M and N are switched to be associated with slots N and M? The answer to that question is YES.

First of all, pause both slots and give the system time to write any pending files. Next, examine the values of opencl-index and cuda-index. (If you're using some or all default values, you won't see them unless you increase the verbosity.) I have never tried this, so I could be wrong, but from what I understand, you can leave the existing values for slot number and for gpu-index unchanged and interchange the values for both cuda-index and opecl-index. That may be a little tricky, too.

Anyway, once you accomplished that change, DO NOT un-pause. FAHClient must be restated (or your system rebooted).

Before you start, make a backup copy of config.xml.

This may or may not work, and of course there's always the chance that you'll lose a WU or two before you make it work, but I do think it's possible.

[foldinghomealone2]

Thank you for your answer.
I'm very excited to test this tomorrow.

Edit:
"interchange the values for both cuda-index and opecl-index"
Where should I interchange the values?

[bruce]

It looks like you don't need what I said about verbosity.

Scroll through your log and find a section for each GPU that looks something like this:

22:28:13:WU00:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Gloria\AppData\Roaming\FAHClient\cores/fahwebx.stanford.edu/cores/Win32/AMD64/NVIDIA/Fermi/beta/Core_21.fah/FahCore_21.exe -dir 00 -suffix 01 -version 704 -lifeline 9088 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0

... and ...

22:31:59:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Gloria\AppData\Roaming\FAHClient\cores/fahwebx.stanford.edu/cores/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 704 -lifeline 9088 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1

(Yours will be different.)

I'm trying several alternatives to confirm they work. I may be wrong about it, but since it takes a restart of the client, it's not as simple to test as I would like.

The values can be set in FAHClient using FAHControl:
Configure + slots + select a GPU ... and you'll see the values are currently set to -1 which means the client should choose them. The actual values are shown in those two segments of the log above.

[foldinghomealone2]

I've read about verbosity but I didn't know where to set up the values for both cuda-index and opecl-index.
I was looking at all xml-files because I thought I have to change it there.

This helped:
"The values can be set in FAHClient using FAHControl:
Configure + slots + select a GPU ... and you'll see the values are currently set to -1 which means the client should choose them. The actual values are shown in those two segments of the log above."

I will try soon...

[foldinghomealone2]

It works and even w/o restart of the client

Just pause, leave GPU index, interchange openCL and CUDA index, save and fold.

You can see on my screenshot that after I interchanged openCL and CUDA index the GPU utilization of both GPUs changed.
https://imgur.com/a/sXQQq

Just for information: The designation of the GPU 'sticks' with the WU. After interchange, although the WU is processed in a different GPU, it still shows the same GPU designation as before.

System specs:
GPU1: 1080 Ti
GPU2: 1080 (non-Ti)
driver: 389.94
FAHclient: 7.4.4
i5-6600K
Win10

[ChristianVirtual]

Something unexpected learned; interesting proposal and thanks for testing ...

[bruce]

It works and even w/o restart of the client

Just pause, leave GPU index, interchange openCL and CUDA index, save and fold.

You can see on my screenshot that after I interchanged openCL and CUDA index the GPU utilization of both GPUs changed.
https://imgur.com/a/sXQQq

Just for information: The designation of the GPU 'sticks' with the WU. After interchange, although the WU is processed in a different GPU, it still shows the same GPU designation as before.

System specs:
GPU1: 1080 Ti
GPU2: 1080 (non-Ti)
driver: 389.94
FAHclient: 7.4.4
i5-6600K
Win10

Thanks for testing that and your report.

I expected that this would NOT change the GPU's description (as you observed). Once you accomplish this switch, you are responsible for remembering which index values are "correct" and which are switched ... along with all the other information associated with that GPU.

Most of the time, the default setup associates the right name with the right GPU, but there have been a number of bug reports where the default setup was interchanged and development has been improving the situation. (This method allowed folks to correct the naming.) We just took advantage of it to respond to your desire to "rotate" the configuration.

Are you running V7.4.4 or the beta V7.4.15+?

[foldinghomealone2]

V7.4.4

As there is no restart of the client required, a function could be added to the client for an 'automated internal rig assignment optimization'.
For sure, it makes sense in only a few configurations like: at least 2 different GPUs, different projects assigned for different GPU utilization
Otherwise interchange wouldn't make sense.
Testing with different rig configurations would be mandatory but I'm confident that you know a lot of willing testers.

Escpecially heavy lifters with mulit-GPU rigs could benefit from such a function.

[ChristianVirtual]

To be considered is the potential lost of already folded time due to pause/restart and the checkpoint used for restart. Depend on project setting it might be worst then the gain of a better card. But yeah; that’s one individual decision to make.
(Would be nice to see actual checkpoints in log file; or at least the planned frequency)

[foldinghomealone2]

Sure, it should be checked if a WU rotation would make sense direcly after a WU has been downloaded.

[Joe_H]

(Would be nice to see actual checkpoints in log file; or at least the planned frequency)

At least for the CPU cores, checkpoints are recorded in one of the work files used by the folding core. That may also hold for the GPU cores, but I don't have any system doing GPU folding to check that on. I have no idea if that information is communicated to the FAHClient process to be available for recording in the log.txt file.

Sure, it should be checked if a WU rotation would make sense direcly after a WU has been downloaded.

That is a feature that is unlikely to be added to the client as it would add complexity for little gain to the science. When assigned the WU would have been the best available match to the request, additional optimizations at the folder level have usually been left to the individuals operating the systems.

[foldinghomealone2]

That is a feature that is unlikely to be added to the client as it would add complexity for little gain to the science. When assigned the WU would have been the best available match to the request, additional optimizations at the folder level have usually been left to the individuals operating the systems.

I have seen up to 15% PPD difference depending on whether WU A is assigned to slot N and WU B is assigned to M compared to A assigned to M and B to N.
And I use a 1080 TI and a 1080. On 'more different' GPUs the PPD difference could be even higher, I assume.
And the more PPD a rig makes, the more scientific value there is.

When you have a rig with same GPUs then there won't be any benefit. Same when all GPUs process the 'same' WU.
Normal users with one GPU won't profit at all.
But all the top ranking folders use multi GPU rigs. The top 300 (per PPD ranking) use all multi GPU rigs and make more than 60% of all PPD.
And there are a lot more multi GPU rigs out there...

I understand that every change in the client has to be evaluated regarding improvent vs. money.