GPU WU Uploads Backing Up P14220 ~ P14250

If you think it might be a driver problem, see viewforum.php?f=79

Moderators: Site Moderators, FAHC Science Team

Post Reply
dfgirl12
Posts: 38
Joined: Fri Aug 21, 2009 8:34 am

GPU WU Uploads Backing Up P14220 ~ P14250

Post by dfgirl12 »

I'm starting to experience a different, but related problem of WU taking longer to upload (original posts from: viewtopic.php?f=18&t=31876&p=308997#p308997) than it takes a 2080Ti / 1080Ti to complete them.
It could just be my cable internet connection, but currently having 37+ concurrent 50MB-100MB uploads is choking it to death (fast or slow servers: once the connection is clogged, then none of them get uploaded for hours or days).
My uploads are 4+ WU deep on some machines, and getting worse today.

Example:
<<Image: Had 5 WUs waiting to send for a single folding slot>>

Are there any plans for the larger WU sizes and not enough return data channel capacity to handle it (like longer WU run times)?
Any suggestions, other than turning off half (~20) of the GPUs, or getting a fiber connection (not really available for me yet)?
Or, like a setting to limit only uploads to 1 upload per FAH Client?

Without running a speed test, I know I can typically upload 100-300 KB/s with my connection. But, all it takes is a few slow FAH uploads and it starts backing up when the WUs take ~30 minutes to fold.

Thanks
Last edited by dfgirl12 on Sat Nov 09, 2019 4:16 am, edited 1 time in total.
dfgirl12
Posts: 38
Joined: Fri Aug 21, 2009 8:34 am

Re: GPU WU Uploads Backing Up P14220 ~ P14250

Post by dfgirl12 »

I 'finished' all the current WUs so they were idle, and disconnected half of the PCs from the network. It took 8-12 hours for half of the PCs to clear out the completed WUs. I'm adding in the others, one every 1-2 hours.

This problem has only happened with these larger WUs and short folding times.

I'm going to try FAH setting of: max-queue=2 (which didn't help, or limit the upload queue). Hopefully, something like that will keep the upload queue from spiraling out of control. Any other ideas?
Last edited by dfgirl12 on Sat Sep 28, 2019 7:22 am, edited 1 time in total.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: GPU WU Uploads Backing Up P14220 ~ P14250

Post by bruce »

there are several topics discussing the same problem. There have been network problems at temple.edu. The network admins are aware of the problem. See the post here.

The two Work Servers vav3.ocis.temple.edu and vav4.ocis.temple.edu apparently have been throttled (perhaps by a shared campus router) and they require a certain minimum bandwidth to support FAH's communications needs..
dfgirl12
Posts: 38
Joined: Fri Aug 21, 2009 8:34 am

Re: GPU WU Uploads Backing Up P14220 ~ P14250

Post by dfgirl12 »

It looks like with those servers back online again, the congestion is being alleviated with more WU variety. Thanks! :)
Post Reply