WUs Expiring (Slow GPU)

[technic58]

Hello all,

I recently joined the forum and made this post (wanted older AMD GPUs to be whitelisted, was informed why it wasn't possible):

viewtopic.php?f=83&t=36869

So, I jumped on eBay to see what was available for cheap that wouldn't run up the electricity bill. I narrowed the selection down to either the nVidia Quadro 2000 or the AMD FirePro W2100. I ended up going with the W2100 - they are less powerful than the Quadro, but seem to be more efficient. Just looking at the specs listed on TechPowerUp, the W2100 FP64/TDP is 27.2 GFLOPS/26W vs the Quadro 2000 40 GFLOPS/62W. I ended up with two W2100s for ~$15 a piece, plugged them into my system, and they are happily folding.

Now for the issue - although I am happy that I have a CPU and two GPUs folding and am sitting at ~120W at the wall, my W2100s seem to be constantly in danger of not finishing WUs before they time out or expire. I don't care about points, but I'd like my system to be doing useful work. Any advice for how to handle this situation? Are there flags I can add to make sure smaller WUs get downloaded? I'm assuming WUs that don't get finished by expiration get thrown out.

System:
Intel Core i5-6400 (currently happily folding 4 threads)
MSI B150M Mortar Motherboard
2x4GB Kingston HyperX DDR4-2400
2xAMD FirePro W2100 (GPU-z Device ID 1002 6608)
Corsair CX600 PSU
Windows 10 Pro x64 20H2

Current performance is as follows:
CPU Slot: Project 16927, TPF 5 mins 11 secs, PPD 3778
GPU 1 Slot: Project 17434, TPF 36 mins 8 secs, PPD 9963
GPU 2 Slot: Project 13438, TPF 50 mins 46 secs, PPD 15431

I have the latest drivers installed and I am not seeing any errors in the log. Just running a little more slowly on the GPUs than the deadlines allow for.

Thanks!!
technic58 (team 34106)

EDIT: I should note that due to a motherboard limitation, one of the cards is limited to PCIe x2 rather than the full x16. However, I don't know how limiting that would be on a low end card like this.

[JimboPalmer]

Intel Core i5-6400 (currently happily folding 4 threads)

F@H tries to reserve one thread for helping each GPU, so I think your CPU is overbooked. See if setting it to 2 gives more speed to your FirePro W2100s.

You may not be as limited by x2 PCIE, as the W2100 only uses x8 in the best of times.

https://www.techpowerup.com/gpu-specs/f ... 2100.c2612

" 2 x PCIe 3.0 x16 slots (support x16/x4)*
• 2 x PCIe 3.0 x1 slots
• 1 x M.2 Key E Socket supports type 2242 Wi-Fi devices
* The PCI_E4 slot will only run PCIe 3.0 x2 speed when the PCI_E3 slot is installed a device." - https://us.msi.com/Motherboard/B150M-MO ... cification

So you may be getting x4.

GPU-Z would say
https://www.techpowerup.com/gpuz/

[technic58]

Setting the CPU threads to "-1" caused the program to default to 1 thread, and that did indeed cause the GPU PPD to go up. Raising the CPU thread number to 2 seems like it caused it to fall a bit (may have something to do with the CPU boosting when one core is being used but not boosting when two cores are being used). I'll mess with that.

Also - good find! GPU-Z is telling me the card is running at PCIe x2. However, I wasn't aware that the board would select the speed of that PCIe slot based on whether or not a card is inserted into the 3rd slot. I currently have a WIFI card in the third slot that I can easily move to the other PCIe x1 slot. I agree, probably not bottlenecking the card, but might as well switch it. I'll report back.

Thanks!

[JimboPalmer]

I agree, if you cut back on CPU threads, to speed up on GPU threads, you may come out ahead on points.

I hope I am giving interesting advice even when it is not good advice!

[technic58]

Okay, setting CPU to 2 threads gives the following:

CPU Slot: Project 16927, 2438 PPD (formerly 3778 PPD) (now 2 threads vs 4)
GPU 1 Slot: Project 17434, 10942 PPD (formerly 9963 PPD)
GPU 2 Slot: Project 13438, 22783 PPD (formerly 15431 PPD)

So that's a total of 36,163 PPD vs the original 29,172 PPD. Definitely an improvement, and pulling less wattage at the wall as well. I will swap the PCIe card later to see if that does anything, although GPU-Z is showing both GPUs at ~100% load, so I don't expect much. I should also say, I have the Folding Power slider moved to "Full" for all slots.

So this is an improvement, but depending on the project, I may still be in danger of expiring/timing out WUs. Is there any flag I can set to get smaller WUs or more optimized WUs? I am planning to mess with overclocking/undervolting in the future as well, but I think my starting point is just very low.

Thanks again, good advice!

[Neil-B]

Do remember there are two different "deadlines" ... timeout is the shorter of the two and is the point at which the wu is put into the queue for re-release .. trying to stay within the is preferable but is not critical in any way .. expiration is the longer of the two and is the point at which the wu has no value and will be dumped by the client .. this second one is the one that if you cant meet you need to think about retiring the kit .. currently there is not much you can do to influence the assignment of wus .. how assignments work ard always under review and this may change .. but I reckon your cards should be meeting current timeouts even though they are quite old

[Neil-B]

Once you have a good feel for average performance you could try not using the cpu slot ... the extra overhead and letting the cpu boost as much as it can may give you even more out of you gpus ... it still feels as if you have a pcie issue you might be able to improve

[JimboPalmer]

Is there any flag I can set to get smaller WUs or more optimized WUs?

There is, but not the way you may expect. Back in dial up modem days, you could specify a smaller download, there is no direct correlation to a shorter WU.

gunarre says
"There has been an advanced option called "max-packet-size" which could be set to "small", "normal" or "big""

You can try that, but the size of the WU is not directly your issue, the length of execution is. Still, there may be some relationship, and it does not hurt.

[iero]

According to the FOLDING.LAR.SYSTEMS database, with a sample of 115 WUs the avg PPD for an W2100 is 12.000 [best 30.000/worst 4.000]. So everything would seem to be working as intended?

[technic58]

Thanks for explaining the difference between the timeout date/time and the expiration date/time.

Both GPUs are now folding on a similar project (13438 and 13440) and have comparable estimated PPD (22717 and 22861, respectively). So, I don't think the PCIe slot running at a slower speed is causing any bottlenecking. But I'll switch the WIFI card, anyway. But to iero's point - yes - it looks like I'm getting about as much performance as would be expected.

The situation I am finding myself in is the following:
Current date/time: 8PM 3/9/2021

GPU1:
ETA: 2.03 days
Expiration: 4PM 3/11/2021

GPU2:
ETA: 1.39 days
Expiration: 12AM 3/11/2021

Both GPU slots downloaded WUs that have expiration dates exactly 2 days after being assigned. At this point, it looks like neither one is going to complete a WU prior to the expiration date. So from the guidance above - it looks like I should be asking a different question. The size of the WU isn't the issue - the amount of time I have to finish the WU looks to be the issue. So is there a way to download WU's with longer allowable execution times? Or is there a way to push this up the flagpole - I have entry level cards from 2014 folding 100% capacity around the clock that don't seem to have enough time to complete WUs. Not sure if leadership is okay with that stat or not - I would think they would want F&H to be accessible to people with lower end hardware (or people with mid-range hardware who aren't folding around the clock) as well, although I understand wanting to get WUs done as quickly as possible, as well.

I will try the max-packet-size flag to see if that has some impact on allowable execution time.

Thanks again!

EDIT: Dates are month/day/year.

[technic58]

Well, I guess we won't find out if those WUs would have finished because switching the WIFI card caused them both to dump. However, similar situation, both are working on similar project WUs that expire in two days with ETA of 2.4 days.

[gunnarre]

Those are Covid Moonshot sprint WUs. They have particularly short timeouts and expirations, and most of them have a high atom count which aren't conducive to making the timeout on your cards. Some Covid Moonshot WUs have lower atom counts, and might finish, like project 13439 and 13443.

The Moonshot sprint already are trying to issue projects to cards that fit the WU, so each sprint has previously had two different project numbers for big and small cards, but it seems to me that the W2100 has too few cores to finish Moonshot sprints reliably. I have a Radeon HD 7770 card which seems to be set to the same GPU species as your W2100, but it has twice as many cores, faster memory etc.

Unless this can be run up the pole so the Chodera lab can blacklist the W2100 from Moonshot WUs, I see no other solution for you than to blacklist the Moonshot work servers in your firewall. (Setting the cause preference away from Covid would deprive you of WUs from other non-Moonshot Covid-related projects, and as it's a only a soft preference you might still get these expiring WUs issued.)

Edit: I've sent a PM to Dr. Chodera with a link to this thread.

[technic58]

Thanks for the reply! It makes sense now that moonshot WUs would have a quick turnaround time. Is there a list of the moonshot servers somewhere? It'd be nice to support the moonshots but as you say, it looks like my hardware isn't up to it.

[gunnarre]

This page lists the assignment server for each project as well as the atom counts (as Moonshot projects are unconventional they might not have a consistent atom count run-to-run): https://apps.foldingathome.org/psummary

If you just blacklist 18.188.125.154 in your PC or router that should prevent you from receiving Moonshot WUs. Once the lab has been able to reconfigure the assigment, you should be able to remove the blacklisting.

Question for readers: Would projects 13439 and 13445 make timeouts on the W2100 GPU?

[Neil-B]

Please note blacklisting of servers shouldn't be the "go to" solution for resolving assignment issues ... ... People who use this approach to shape their points production are cheating the fah system and intent of the eula ... therefore recommending the same action (even if for good reason) before the powers that be have a chance to sort the assignment server is basically suggesting people do something that will mean they look like "cheats" ... The traces of this type of activity will no doubt be able to be spotted in server logs and if it becomes "popular" may well have to be addressed by some form of remedial action? ... if that happens and innocent users have been being advised to do this type of thing in the forums they may well feel somewhat hard done by !!