155.247.166.220 downloads stalled

Moderators: Site Moderators, FAHC Science Team

Re: 155.247.166.220 downloads stalled

Postby Ichbin3 » Wed Jul 01, 2020 5:51 pm

And again ...

Code: Select all
16:32:10:WU01:FS00:Connecting to 155.247.166.220:8080
16:32:10:WU01:FS00:Downloading 3.75MiB
16:32:17:WU01:FS00:Download 5.01%
16:32:26:WU01:FS00:Download 6.68%
16:32:47:WU01:FS00:Download 8.34%
16:33:05:WU01:FS00:Download 11.68%
MSI B450 Tomahawk, Ryzen 5 2600, RTX 2080Ti@180W, Mint 19.3
Ichbin3
 
Posts: 69
Joined: Thu May 28, 2020 9:06 am
Location: Germany

Re: 155.247.166.220 downloads stalled

Postby parkut » Wed Jul 01, 2020 5:53 pm

Stalled download, almost 3 hours now. Linux Client Version: 7.6.13, and Core Version: 0.0.11

Code: Select all
13:43:07:WU02:FS02:Connecting to assign1.foldingathome.org:80
13:43:07:WU02:FS02:Assigned to work server 155.247.166.220
13:43:07:WU02:FS02:Requesting new work unit for slot 02: READY gpu:0:GM206 [GeForce GTX 960] 2308 from 155.247.166.220
13:43:07:WU02:FS02:Connecting to 155.247.166.220:8080
13:43:08:WU02:FS02:Downloading 1.39MiB
User avatar
parkut
 
Posts: 345
Joined: Tue Feb 12, 2008 8:33 am
Location: SE Michigan, USA

Re: 155.247.166.220 downloads stalled

Postby _r2w_ben » Wed Jul 01, 2020 6:04 pm

On Windows, you can use TCPView to kill the stalled download. Find FAHClient.exe in the list, right click the connection with vav4.ocis.temple.edu as the Remote Address and then click Close Connection.
_r2w_ben
 
Posts: 277
Joined: Wed Apr 23, 2008 4:11 pm

Re: 155.247.166.220 downloads stalled

Postby Ichbin3 » Wed Jul 01, 2020 8:12 pm

I'm running linux.
Btw - the next stalled dl happend.
3 MB would have needed 1h hour to download.
I'm starting to hate that server.
Ichbin3
 
Posts: 69
Joined: Thu May 28, 2020 9:06 am
Location: Germany

Re: 155.247.166.220 downloads stalled

Postby HaloJones » Wed Jul 01, 2020 8:40 pm

Just applied a block on that IP to my firewall. In theory, the clients that are assigned to that server will fail to connect rather than connect but fail to download.
1x Titan X, 5x 1070, 1x 970, 1 x Ryzen 3600

Image
HaloJones
 
Posts: 816
Joined: Thu Jul 24, 2008 11:16 am

Re: 155.247.166.220 downloads stalled

Postby scott@bjorn3d » Wed Jul 01, 2020 9:55 pm

These servers are killing me doing work. Why don't they ever fix them?
scott@bjorn3d
 
Posts: 68
Joined: Tue Dec 19, 2017 1:19 pm

Re: 155.247.166.220 downloads stalled

Postby HaloJones » Thu Jul 02, 2020 8:34 am

server needs a frequently scheduled reboot. it's a simple cronjob
HaloJones
 
Posts: 816
Joined: Thu Jul 24, 2008 11:16 am

Re: 155.247.166.220 downloads stalled

Postby Sparkly » Thu Jul 02, 2020 9:36 am

And another 3, so I am seriously considering blocking this server in my firewall permanently.
Sparkly
 
Posts: 73
Joined: Sun Apr 19, 2020 12:01 pm

Re: 155.247.166.220 downloads stalled

Postby Ichbin3 » Thu Jul 02, 2020 10:10 am

I did now too
Ichbin3
 
Posts: 69
Joined: Thu May 28, 2020 9:06 am
Location: Germany

Re: 155.247.166.220 downloads stalled

Postby rickoic » Thu Jul 02, 2020 9:52 pm

Have been having small problem with hung downloads over the past week with this server, 1-3 times a day a reboot was required. Been living with it as it was a minor problem. Woke this morning and 3 of 4 pcs required a reboot. Total of 4 gpus hung. Just checked my pcs again and had 1 hung up so I rebooted. Got 220 server again and it hung with downloading 3.75gb. Sat for a few minutes with no progress, so I rebooted again. And again I was assigned to 220. Got to 1.8% downloaded this time and then it hung. Total of 6 reboots and on the final reboot it caused my other 3 gpus on the board to throw their wu's and redownload new ones. But at least the problem child got sent to another server and is working again.
Duel 2.8 3 250's Quad 2.4 285. 260, Quad 2.4 3 250 , i7 2.27 2 250 GPU's, i7 2.24 2 250 GPU's, i7 3.06 bigadv, duel Xeon 2.27 bigadv, AMD Phenom ][ 3 250 GPU's, Laptop GT 130M.
I'm folding because Dec 2005 I had radical prostrate surgery.
rickoic
 
Posts: 258
Joined: Sat May 23, 2009 5:49 pm
Location: Mississippi near Memphis, Tn

Re: 155.247.166.220 downloads stalled

Postby HaloJones » Thu Jul 02, 2020 10:12 pm

How is this not being sorted despite all this noise????
HaloJones
 
Posts: 816
Joined: Thu Jul 24, 2008 11:16 am

Re: 155.247.166.220 downloads stalled

Postby vvoelz » Thu Jul 02, 2020 10:28 pm

Sorry for the ongoing problems with vav4. We recently put up more GPU WUs and the connection issue has worsened. I just did a hard reboot of this machine, which we will continue to do every day from now on. Its unclear whether this will actually do the trick, so if you observe ANY amelioration of problem, please post.

I have said before that we are trying to retire this server, and we still are. Hopefully our new hardware will installed in 1-2 months.
User avatar
vvoelz
Pande Group Member
 
Posts: 485
Joined: Sun Dec 02, 2007 9:07 pm
Location: Temple University, Philadelphia PA

Re: 155.247.166.220 downloads stalled

Postby rickoic » Fri Jul 03, 2020 1:01 am

Just had another failure to download from 220.
rickoic
 
Posts: 258
Joined: Sat May 23, 2009 5:49 pm
Location: Mississippi near Memphis, Tn

Re: 155.247.166.220 downloads stalled

Postby vvoelz » Fri Jul 03, 2020 2:16 am

UGH - if we didn't have vital projects being served from vav4 I'd shut it down. (The server code does not let us easily migrate projects from one server to another). There must be some load balancing issues (collection server relays?) that are beyond our control, but perhaps we can set some better parameters. We'll continue to push on this.
User avatar
vvoelz
Pande Group Member
 
Posts: 485
Joined: Sun Dec 02, 2007 9:07 pm
Location: Temple University, Philadelphia PA

Re: 155.247.166.220 downloads stalled

Postby Sparkly » Fri Jul 03, 2020 9:43 am

vvoelz wrote:UGH - if we didn't have vital projects being served from vav4 I'd shut it down. (The server code does not let us easily migrate projects from one server to another). There must be some load balancing issues (collection server relays?) that are beyond our control, but perhaps we can set some better parameters. We'll continue to push on this.

I would be surprised if this issue was a load-balancing thing, since it is more likely that it is a TCP packet loss/corruption/retransmission thing.

Could be as simple as a security setting in your firewall, so if you have older Cisco FWSM stuff on your boarders, you can try turning the sequence randomisation off.

https://community.cisco.com/t5/security-documents/single-tcp-flow-performance-on-firewall-services-module-fwsm/ta-p/3126988#TCP_Sequence_Number_Randomization_and_SACK
Sparkly
 
Posts: 73
Joined: Sun Apr 19, 2020 12:01 pm

PreviousNext

Return to Issues with a specific server

Who is online

Users browsing this forum: No registered users and 3 guests

cron