Page 6 of 7

Re: 155.247.166.220 downloads stalled

PostPosted: Fri Jul 03, 2020 11:29 am
by vvoelz
Thanks Sparkly -- I'll pass that on to our IT folks

Re: 155.247.166.220 downloads stalled

PostPosted: Fri Jul 03, 2020 12:31 pm
by scott@bjorn3d
Good luck with it but for now I can not have 3 2080 TI's sitting idle half the time. So blocked in my firewall.

Re: 155.247.166.220 downloads stalled

PostPosted: Fri Jul 03, 2020 12:36 pm
by HaloJones
I've temporarily unblocked since I'm WFH in front of one of my computers so can sort if there is a problem. So far, have not been directed there.

Re: 155.247.166.220 downloads stalled

PostPosted: Fri Jul 03, 2020 3:26 pm
by HaloJones
Had a stall on a Linux box, ran the following:

sudo netstat -tunp | grep .220

that will give you a response that includes the PID and hopefully FAHClient as the process.

kill the PID.

That worked for me as the client then re-started that connection and got a unit elsewhere.

Re: 155.247.166.220 downloads stalled

PostPosted: Wed Jul 08, 2020 2:36 am
by parkut
stalled download, around 5 hours ago

21:22:29:WU01:FS02:Connecting to assign1.foldingathome.org:80
21:22:30:WU01:FS02:Assigned to work server 155.247.166.220
21:22:30:WU01:FS02:Requesting new work unit for slot 02: RUNNING gpu:0:GM206 [GeForce GTX 960] 2308 from 155.247.166.220
21:22:30:WU01:FS02:Connecting to 155.247.166.220:8080
21:22:30:WU01:FS02:Downloading 3.56MiB
21:22:38:WU01:FS02:Download 5.26%
21:23:53:WU01:FS02:Download 7.02%

Re: 155.247.166.220 downloads stalled

PostPosted: Fri Jul 10, 2020 4:51 am
by kc2lrc
Can confirm, things are still very sluggy on this server as of writing. For the past few days I've had lots of stalled downloads, and just reset 3 tonight between my 11 GPU slots.

My thoughts on this matter are that the work server seems as though it's out of bandwidth - a 5MB download from it took about 2 minutes. It's as if the server is being DDoS'd by the clients. Perhaps the assignment server could be reconfigured to direct fewer clients at this server? That could help with the network consternation here.

In addition, this situation is exercising the bug in FAHClient in which stalled downloads never reset - I've manually killed about 50 of them in the past week just myself - and I imagine this is resulting in substantial idled resources across the Folding@home project in its entirety since it requires manual user intervention before the GPU comes back online. So I think reducing the assignments frequency to this server, if reasonable, would benefit the entire program.

Cheers -
Sam

PS - those of you running into this can reset the connection to un-jam the stalled download. On Windows, use the Sysinternals tcpview utility. On Linux, the 'ss' command (as root) does the trick, as such: 'ss -K dst 155.247.166.220 dport = 8080'

Re: 155.247.166.220 downloads stalled

PostPosted: Fri Jul 10, 2020 5:39 am
by Ichbin3
kc2lrc wrote:'ss -K dst 155.247.166.220 dport = 8080'

The option -K is not working in my ss.

Re: 155.247.166.220 downloads stalled

PostPosted: Sun Jul 12, 2020 2:36 am
by kc2lrc
It's possible you need to be logged on as root, or you have a different distribution that does the command a bit differently. This is from Debian 10.

Re: 155.247.166.220 downloads stalled

PostPosted: Sun Jul 12, 2020 5:28 pm
by HendricksSA
I sure hope this problem will be resolved soon. I've lost half my folding time due to stalled downloads from this server ... 36 hours in the last 3 days. For me the quickest solution is to disable the network adaptor and then re-enable it. That does require the root password.

Re: 155.247.166.220 downloads stalled

PostPosted: Sun Jul 12, 2020 11:10 pm
by bruce
155.247.166.220 has been the target of cyber attacks on and off for the past several days. (I have to wonder why somebody has decided to do that.)

Re: 155.247.166.220 downloads stalled

PostPosted: Sun Jul 12, 2020 11:33 pm
by psaam0001
Let's see.... Classic denial of service via ping flooding? Malware on clients computers from sources other than the F@H client distribution servers? Infrastructure level intrusion to the router's IOS? Just my guesses.

Good luck getting to the bottom of this....

Paul

Re: 155.247.166.220 downloads stalled

PostPosted: Mon Jul 13, 2020 12:39 am
by bruce
That server is replying to pings but the server code isn't running. I understand it's shut down to off-load some accumulated data and make room for your uploads.

155.247.166.220 / 503 HTTP_SERVICE_UNAVAILABLE

Re: 155.247.166.220 downloads stalled

PostPosted: Mon Jul 13, 2020 2:35 am
by psaam0001
Thanks for the update Bruce... I can only try to deal with my clients that are running on what was part of the latest & greatest technology 10-12 years ago (and one recently built Win10 system).

Paul

Re: 155.247.166.220 downloads stalled

PostPosted: Mon Jul 13, 2020 9:49 pm
by bruce
Have you tried it today? It might be back up.

Re: 155.247.166.220 downloads stalled

PostPosted: Tue Jul 14, 2020 4:38 am
by HendricksSA
Just lost 8 hours after a hung download. I was assigned to .220 at 1900z so I am not a good test. I will try firing up the computer tomorrow if this problem is really fixed.