Shortage of GPU work again?

If you think it might be a driver problem, see viewforum.php?f=79

Moderators: Site Moderators, FAHC Science Team

Post Reply
Juggy
Posts: 111
Joined: Fri Mar 20, 2020 12:07 pm
Hardware configuration: Intel i9 9900KF @4.9GHZ
MSI Z390M Gaming OC
64GB G-Skill Ripjaw 3200
Gigabyte RTX2060 Super Gaming OC
NZXT Kraken X63 AIO
Corsair RM850
Samsung 1TB Evo Pro NVMe
Location: Qatar

Shortage of GPU work again?

Post by Juggy »

I am struggling to get GPU work units, is there a shortage again? *.mskcc.org/
Image
Image
Joe_H
Site Admin
Posts: 7856
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Shortage of GPU work again?

Post by Joe_H »

Maybe. JohnChodera did announce within the last few weeks that the work for GPUs would tend to come in waves as each round of projects for the COVID Moonshot was released. Each round would take a period of time to gather data, and then then next round would need to be prepared for release.

Also, checking the Server Status page - https://apps.foldingathome.org/serverstats, some of the servers show as being low or out of work to assign. As they receive WUs back, the next Gen WU can be generated, but they appear to be going out about as fast as they come in.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
ajm
Posts: 754
Joined: Sat Mar 21, 2020 5:22 am
Location: Lucerne, Switzerland

Re: Shortage of GPU work again?

Post by ajm »

Also 140.163.4.231 seems to be struggling somehow. I get only connection errors from it and it doesn't load: http://plfah1-1.mskcc.org/
vmzy
Posts: 136
Joined: Wed Apr 16, 2008 6:25 am

Re: Shortage of GPU work again?

Post by vmzy »

ajm wrote:Also 140.163.4.231 seems to be struggling somehow. I get only connection errors from it and it doesn't load: http://plfah1-1.mskcc.org/
AS server keeping assign me to 140.163.4.231, but it mustbe down now.

Code: Select all

01:55:53:WU01:FS01:Connecting to assign1.foldingathome.org:80
01:55:54:WU01:FS01:Assigned to work server 140.163.4.231
01:55:55:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP108 [GeForce MX150 (GT 1030) Max-Q] from 140.163.4.231
01:55:55:WU01:FS01:Connecting to 140.163.4.231:8080
01:56:59:ERROR:WU01:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0
aetch
Posts: 447
Joined: Thu Jun 25, 2020 3:04 pm
Location: Between chair and keyboard

Re: Shortage of GPU work again?

Post by aetch »

The servers are waiting on a load of big units coming back from slow computers. :D

Seriously, I've had a big unit running on my laptop for almost 4 hours now with an ETA of another 18 hours. I'd love to be able to take the unit and just drop it into my desktop where it could be done in 4-5 hours.

Check the servers for yourself. https://apps.foldingathome.org/serverstats

Joe_H - the Public/Beta jobs on the server stats page, are these units currently running on peoples computers or in a queue waiting to be assigned?
Folding Rigs - None (25-Jun-2022)

ImageImage
JohnChodera
Pande Group Member
Posts: 470
Joined: Fri Feb 22, 2013 9:59 pm

Re: Shortage of GPU work again?

Post by JohnChodera »

Thanks for the heads up! We certainly don't have a WU shortage---there's over 1.5 million WUs ready to go.

We've found an issue with plfah1-1.mskcc.org and think we have it temporarily resolved, and are investigating the root cause.

We've also increased the max assign rate on aws3.foldingathome.org.

~ John Chodera // MSKCC
Juggy
Posts: 111
Joined: Fri Mar 20, 2020 12:07 pm
Hardware configuration: Intel i9 9900KF @4.9GHZ
MSI Z390M Gaming OC
64GB G-Skill Ripjaw 3200
Gigabyte RTX2060 Super Gaming OC
NZXT Kraken X63 AIO
Corsair RM850
Samsung 1TB Evo Pro NVMe
Location: Qatar

Re: Shortage of GPU work again?

Post by Juggy »

JohnChodera wrote:Thanks for the heads up! We certainly don't have a WU shortage---there's over 1.5 million WUs ready to go.

We've found an issue with plfah1-1.mskcc.org and think we have it temporarily resolved, and are investigating the root cause.

We've also increased the max assign rate on aws3.foldingathome.org.

~ John Chodera // MSKCC
Great, thank you John
Image
Image
bumbel123
Posts: 30
Joined: Fri Mar 20, 2020 12:17 pm

Re: Shortage of GPU work again?

Post by bumbel123 »

Hi,

any update here? Found my GPUs in a row doing almost nothing, since 2 days, just occasional 17200, 117xx ... rare spots here and there.

If there's nothing to do anymore and you have all of your results, I stop FAH client on my gear ... :D

Cheers
  • Ryzen 3 3100 (Zen2), Nv GTX 1660 Ti, Win10 Ent
  • Ryzen 5 3600 (Zen2), Nv RTX 2060, Win10 Ent
  • Ryzen 7 2700 (Zen+), Nv RTX 2060, Nv GTX 1660, Ubuntu LTS 20.04
  • Ryzen 5 3600 (Zen2), Nv RTX 2060, Nv GTX 1650, Ubuntu LTS 20.04
markdotgooley
Posts: 101
Joined: Tue Apr 21, 2020 11:46 am

Re: Shortage of GPU work again?

Post by markdotgooley »

I seem to have my two RTX2060 cards busy almost constantly, though the PPD has gone from ridiculously high levels (2.7 to 2.9 million) to lower than usual (1.7 million), to nearer normal (somewhat over 2 million up to 2.5). Nowadays I just shrug it off and wait for better times...
kiore
Posts: 931
Joined: Fri Jan 16, 2009 5:45 pm
Location: USA

Re: Shortage of GPU work again?

Post by kiore »

markdotgooley wrote:I seem to have my two RTX2060 cards busy almost constantly, though the PPD has gone from ridiculously high levels (2.7 to 2.9 million) to lower than usual (1.7 million), to nearer normal (somewhat over 2 million up to 2.5). Nowadays I just shrug it off and wait for better times...
Indeed the work is coming in bursts, particular the COVID-19 'moonshot' stuff. The PPD is all over the place with these, extremely high and extremely low, averaging for me on the low end. But this is cutting edge science in a hurry, and I think we all need to accept these are exceptional times and tolerate some randomness for a bit.
Image
i7 7800x RTX 3070 OS= win10. AMD 3700x RTX 2080ti OS= win10 .

Team page: http://www.rationalskepticism.org
bumbel123
Posts: 30
Joined: Fri Mar 20, 2020 12:17 pm

Re: Shortage of GPU work again?

Post by bumbel123 »

Yes, PRCG 13416 and all 117xx WUs are causing a drop in PPD. Especially with 117xx WUx I can see an overall drop of solid 30% monitored over a couple of days now.

They are even worse to the already discussed PRCG 13416 ... where I could confirm a longer term drop of "just" 15% over all.
  • Ryzen 3 3100 (Zen2), Nv GTX 1660 Ti, Win10 Ent
  • Ryzen 5 3600 (Zen2), Nv RTX 2060, Win10 Ent
  • Ryzen 7 2700 (Zen+), Nv RTX 2060, Nv GTX 1660, Ubuntu LTS 20.04
  • Ryzen 5 3600 (Zen2), Nv RTX 2060, Nv GTX 1650, Ubuntu LTS 20.04
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Shortage of GPU work again?

Post by bruce »

Some of the projects 134xx are known to produce lower PPDs on certain GPUs. To avoid those cases, assignments can be restricted at the server level. Then we get complaints that the GPUs can't get work. Right now, the projects that earn "normal" PPDs are in short supply so you have to choose between getting lower PPDs or shutting down those devices entirely.

I belive this to be a temporary condition but have no prediction when conditions may change.
Tabaluga
Posts: 1
Joined: Sun Jul 19, 2020 10:22 am

Re: Shortage of GPU work again?

Post by Tabaluga »

Since a few days i'm having problems getting GPU WU.


The log shows, no WU availble for your configuration, so i change the settings to allow all type of GPU WU.
Rebooted the win10 machine and i received some WU.
Now again same message.

What is the problem with the GPU WU assignment ?

rgds

Tabaluga
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Shortage of GPU work again?

Post by Neil-B »

Making the change you did wont have actually mage any difference to your likelihood of getting a WU as the choice of WU type is simply a preference and defaults to other types if none of your preference are available - pure luck something was available after you restarted ... If you post the first 200 lines of your log (see bruce sig block above for help) then someone may be able to assist ... otherwise any assistance will just be pure guesswork !!
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Shortage of GPU work again?

Post by bruce »

Yes, there's a shortage of GPU projects.

Science run in spurts. Each scientist constructs projects to study particular aspects of a particular protein and when enough data is collected, it has to be analyzed. When somebody else proposes a new theory, a new study is launched. It takes scientific minds to do all those steps and FAH doesn't waste your time with "busy work" doing unneeded calculations.
Post Reply