155.247.166.219 & 155.247.166.220 are now hosed [Not Now]

Moderators: Site Moderators, FAHC Science Team

nsummy
Posts: 9
Joined: Fri Aug 11, 2017 1:46 pm

155.247.166.219 & 155.247.166.220 are now hosed [Not Now]

Post by nsummy »

Looks like there are bigger issues going on here. After not being able to download new WUs yesterday I changed my client-type to advanced, which prompted WUs to start downloading. I wake up this morning to see that none of my finished WUs will upload, and big surprise, none will download, since they are sending and recieving from the same server!

This isn't the subforum to launch a big discussion about updates, but I think this is a shining example of why the client needs more development work and a meaningful update. One (or even 4-5) server should not bring the whole folding operation to its knees.
Joe_H
Site Admin
Posts: 7854
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: 155.247.166.219 & 155.247.166.220 are now hosed

Post by Joe_H »

Both WS's have been restarted and should be accepting connections.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
scottsak
Posts: 2
Joined: Thu Jan 18, 2018 4:35 pm

Re: 155.247.166.219 & 155.247.166.220 are now hosed

Post by scottsak »

down again
acnash
Posts: 20
Joined: Sun Mar 11, 2018 4:34 pm
Hardware configuration: 32x Intel (R) Xeon (R) CPU E5-2687W 0 @ 3.10GHz
125.88GiB
1 Nvidia GeForce GTX 970
Mint Linux ver 18
Location: Oxford, UK
Contact:

Re: 155.247.166.219 & 155.247.166.220 are now hosed

Post by acnash »

I've just noticed this thread; I'm having the same issue on (Tues 13 March, 12:31pm GMT).
Joe_H
Site Admin
Posts: 7854
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: 155.247.166.219 & 155.247.166.220 are now hosed

Post by Joe_H »

Currently both servers are up, and checking the logs only one was down for part of an hour earlier this morning. Post with further details if your client is still having problems connecting to either of these servers.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
e3072743
Posts: 5
Joined: Thu Mar 29, 2018 5:12 am

Re: 155.247.166.219 & 155.247.166.220 are now hosed

Post by e3072743 »

I have been stuck in "Ready" and "Download" modes with my GPU for a few days already. Even after reboots it just won't start download.

My log is full of this:

Code: Select all

05:12:26:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104GL [xxx] from 155.247.166.219
05:12:26:WU01:FS01:Connecting to 155.247.166.219:8080
05:12:47:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
05:12:47:WU01:FS01:Connecting to 155.247.166.219:80
05:16:40:WU01:FS01:Connecting to 171.67.108.45:80
05:16:40:WU01:FS01:Assigned to work server 155.247.166.219
05:16:40:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104GL [xxx] from 155.247.166.219
05:16:40:WU01:FS01:Connecting to 155.247.166.219:8080
05:17:01:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
Is there any way I can force the client to try and assign to another server in cases like this? (Newbie here, using FAHControl)
Joe_H
Site Admin
Posts: 7854
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: 155.247.166.219 & 155.247.166.220 are now hosed

Post by Joe_H »

Welcome to the folding support forum e3072743.

If you could post the first section of your log file that shows the system and configuration info, we would be more able to figure out what is going on with your system. The WS shown is otherwise working and connecting to assign WU's to other folders.

Also, have you recently updated your system, added or updated anti-malware software? Changes such as these can block the connection from from downloading new WU's.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: 155.247.166.219 & 155.247.166.220 are now hosed

Post by bruce »

e3072743 wrote:My log is full of this:

Code: Select all

05:12:26:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104GL [xxx] from 155.247.166.219
05:12:26:WU01:FS01:Connecting to 155.247.166.219:8080
05:12:47:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
05:12:47:WU01:FS01:Connecting to 155.247.166.219:80
05:16:40:WU01:FS01:Connecting to 171.67.108.45:80
05:17:01:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
Is there any way I can force the client to try and assign to another server in cases like this? (Newbie here, using FAHControl)
You're probably right. There will be cases where no assignment was made, but most of the time, that's temporary. After a certain number of attempted connections, the client SHOULD give up and try another work server.
How many times in a row did the "Request for new work unit for slot 01" fail? (Ignore anything in the log not associated with that slot.) Which Work Servers were tried?

In the portion of your log that I quoted, it did try both 155.247.166.219 and MAYBE some other WS, but you trimmed off too much of your log, (Connections to AS like 171.67.108.45 don't count and port 80 and 8080 aren't important here.)

I can connect to http://155.247.166.219:8080/ with my browser. Are you connecting to the internet through a proxy server? (They're not common on home computers ... only in businesses)
e3072743
Posts: 5
Joined: Thu Mar 29, 2018 5:12 am

Re: 155.247.166.219 & 155.247.166.220 are now hosed

Post by e3072743 »

Can't find the log from where the last snippet was taken but here's a full log from the last couple of days when the GPU folding was also stuck: https://pastebin.com/AxVRp9w5. The CPU seems to have been folding just fine during all this time.

After my last post it got a few assignments after just waiting and re-attempting. But then the problem started again.

I haven't installed any new antivirus/firewall products lately, but using the same one I used before this started happening. Of course it has auto-updated though.

The address http://155.247.166.219:8080/ is not opening in my browser at the moment, neither is http://155.247.166.219:80/. Ping DOES work though. Tried also on my mobile (over 4G) and :80 didn't work but :8080 did! So it seems like a connectivity issue..
I'm not using any proxies and there shouldn't be any special blacklists/filtering set on my router.
tempally
Posts: 3
Joined: Wed Apr 04, 2018 8:22 am

Re: 155.247.166.219 & 155.247.166.220 are now hosed

Post by tempally »

Hello!
Yesterday I have found slow upload to 155.247.166.219 (it is assigned to me by 171.67.108.45).
I have from 5 to 9 seconds to download 900KiB WU and 23 to 26 (!) seconds to upload 1.8MiB result
I've got 60Mbit full duplex (tested) 7ms-latency channel to MSK-IX.
- Is this speed normal for 155.247.166.219 server?
- How can I use the closest server to MSK-IX?

Code: Select all

*********************** Log Started 2018-04-02T21:03:57Z ***********************
21:03:57:******************************** Build ********************************
21:03:57:      Version: 7.4.4
21:03:57:         Date: Mar 4 2014
21:03:57:         Time: 20:26:54
21:03:57:      SVN Rev: 4130
21:03:57:       Branch: fah/trunk/client
21:03:57:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
21:03:57:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
21:03:57:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
21:03:57:     Platform: win32 XP
21:03:57:         Bits: 32
21:03:57:         Mode: Release
...
21:03:57:<config>
21:03:57:  <!-- Client Control -->
21:03:57:  <exit-when-done v='true'/>
21:03:57:
21:03:57:  <!-- Folding Core -->
21:03:57:  <checkpoint v='30'/>
21:03:57:  <core-priority v='low'/>
21:03:57:
21:03:57:  <!-- Network -->
21:03:57:  <proxy v=':8080'/>
21:03:57:
21:03:57:  <!-- Slot Control -->
21:03:57:  <pause-on-battery v='false'/>
21:03:57:  <power v='full'/>
21:03:57:
21:03:57:  <!-- User Information -->
21:03:57:  <passkey v='********************************'/>
21:03:57:  <team v='**********'/>
21:03:57:  <user v='**********'/>
21:03:57:
21:03:57:  <!-- Folding Slots -->
21:03:57:  <slot id='1' type='GPU'>
21:03:57:    <client-type v='advanced'/>
21:03:57:  </slot>
21:03:57:</config>
21:03:57:Trying to access database...
21:03:57:Successfully acquired database lock
21:03:57:Enabled folding slot 01: READY gpu:1:GP104 [GeForce GTX 1070] 6463
21:03:57:WU00:FS01:Connecting to 171.67.108.45:80
21:03:58:WU00:FS01:Assigned to work server 155.247.166.219
...
21:03:58:WU00:FS01:Connecting to 155.247.166.219:8080
21:04:01:WU00:FS01:Downloading 902.05KiB
21:04:03:WU00:FS01:Download complete
...
22:19:37:WU00:FS01:Uploading 1.79MiB to 155.247.166.219
22:19:37:WU00:FS01:Connecting to 155.247.166.219:8080
22:19:43:WU00:FS01:Upload 24.43%
22:19:49:WU00:FS01:Upload 52.35%
22:19:55:WU00:FS01:Upload 76.77%
22:20:00:WU00:FS01:Upload complete
22:20:00:WU00:FS01:Server responded WORK_ACK (400)
...
22:18:51:WU01:FS01:Connecting to 155.247.166.219:8080
22:18:54:WU01:FS01:Downloading 902.24KiB
22:19:00:WU01:FS01:Download 78.03%
22:19:01:WU01:FS01:Download complete
...
22:19:37:WU00:FS01:Uploading 1.79MiB to 155.247.166.219
22:19:37:WU00:FS01:Connecting to 155.247.166.219:8080
22:19:43:WU00:FS01:Upload 24.43%
22:19:49:WU00:FS01:Upload 52.35%
22:19:55:WU00:FS01:Upload 76.77%
22:20:00:WU00:FS01:Upload complete
22:20:00:WU00:FS01:Server responded WORK_ACK (400)
...
23:33:34:WU00:FS01:Connecting to 155.247.166.219:8080
23:33:36:WU00:FS01:Downloading 903.63KiB
23:33:41:WU00:FS01:Download complete
...
23:34:19:WU01:FS01:Uploading 1.79MiB to 155.247.166.219
23:34:19:WU01:FS01:Connecting to 155.247.166.219:8080
23:34:25:WU01:FS01:Upload 27.95%
23:34:31:WU01:FS01:Upload 52.40%
23:34:37:WU01:FS01:Upload 76.85%
23:34:43:WU01:FS01:Upload 100.00%
23:34:43:WU01:FS01:Upload complete
23:34:43:WU01:FS01:Server responded WORK_ACK (400)
...
02:04:03:WU00:FS01:Connecting to 155.247.166.219:8080
02:04:05:WU00:FS01:Downloading 902.91KiB
02:04:11:WU00:FS01:Download 63.79%
02:04:12:WU00:FS01:Download complete
...
02:04:48:WU01:FS01:Uploading 1.80MiB to 155.247.166.219
02:04:48:WU01:FS01:Connecting to 155.247.166.219:8080
02:04:54:WU01:FS01:Upload 27.78%
02:05:00:WU01:FS01:Upload 52.09%
02:05:06:WU01:FS01:Upload 83.34%
02:05:10:WU01:FS01:Upload complete
02:05:11:WU01:FS01:Server responded WORK_ACK (400)
...
03:19:34:WU01:FS01:Connecting to 155.247.166.219:8080
03:19:36:WU01:FS01:Downloading 902.67KiB
03:19:42:WU01:FS01:Download 56.72%
03:19:45:WU01:FS01:Download complete
...
03:20:19:WU00:FS01:Uploading 1.80MiB to 155.247.166.219
03:20:19:WU00:FS01:Connecting to 155.247.166.219:8080
03:20:25:WU00:FS01:Upload 20.84%
03:20:31:WU00:FS01:Upload 48.63%
03:20:37:WU00:FS01:Upload 72.94%
03:20:43:WU00:FS01:Upload 93.78%
03:20:45:WU00:FS01:Upload complete
03:20:45:WU00:FS01:Server responded WORK_ACK (400)
...
04:35:37:WU01:FS01:Uploading 1.79MiB to 155.247.166.219
04:35:37:WU01:FS01:Connecting to 155.247.166.219:8080
04:35:43:WU01:FS01:Upload 24.40%
04:35:49:WU01:FS01:Upload 52.28%
04:35:55:WU01:FS01:Upload 80.16%
04:35:59:WU01:FS01:Upload complete
04:35:59:WU01:FS01:Server responded WORK_ACK (400)
Joe_H
Site Admin
Posts: 7854
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: 155.247.166.219 & 155.247.166.220 are now hosed

Post by Joe_H »

@tempally I can not say whether this speed is normal or not, but under half a minute to upload is a good time under any circumstances. The two servers at Temple, 155.247.166.219 & 155.247.166.220, probably are the closest servers to you anyways as they are on the east coast of the US. As a practical matter you have to accept assignments as they are available, currently the bulk of them are coming from just a few servers.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
tempally
Posts: 3
Joined: Wed Apr 04, 2018 8:22 am

Re: 155.247.166.219 & 155.247.166.220 are now hosed

Post by tempally »

Joe_H wrote:...under half a minute to upload is a good time... 155.247.166.219 ... probably are the closest servers to you...
I see. I have to consider this server is fast enough:
today I had a WU from 171.67.108.157 - upload of 7.75MiB was 2 min 14 sec (59kiB/s, less than 71kiB/s to 155.247.166.219).
Does it mean I shouldn't activate big packet-size?
(The same speed of 500MB means more than 2 hours upload time...)
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: 155.247.166.219 & 155.247.166.220 are now hosed [Not Now

Post by bruce »

Support for the packet-size options has been in a bit of a turmoil. The choices based on words has changed; the numerical values still work the way the used to. Nevertheless, it's up to the project owner to enable that option.

If you have a 56kb dial-up modem (or slower) you'll probably need to use a packet-size setting ... that's what it was originally designed for many years ago. :D

The slow server speeds you're reporting are unusual. I suspect there have been some temporary events contributing to those numbers. I recommend trying traceroute (Linux) or tracert (Windows). this often shows if there are detectable problems between you and a server.
tempally
Posts: 3
Joined: Wed Apr 04, 2018 8:22 am

Re: 155.247.166.219 & 155.247.166.220 are now hosed [Not Now

Post by tempally »

bruce wrote:The slow server speeds you're reporting are unusual. I suspect there have been some temporary events contributing to those numbers. I recommend trying traceroute (Linux) or tracert (Windows).
Still having the same speed

Code: Select all

02:13:19:WU00:FS01:Uploading 1.79MiB to 155.247.166.219
02:13:42:WU00:FS01:Upload complete

03:57:33:WU01:FS01:Uploading 7.80MiB to 171.67.108.157
03:59:49:WU01:FS01:Upload complete
So, I've traced some routes:

Code: Select all

TRACERT 171.67.108.157
...
  4     4 ms     4 ms     4 ms  mag9-cr01-be12.51.msk.stream-internet.net [212.188.1.5]
  5    16 ms    17 ms    16 ms  oct-cr03-be1.78.spb.stream-internet.net [212.188.2.37]
  6    22 ms    22 ms    22 ms  kivi-cr01-ae8.78.hel.stream-internet.net [212.188.54.2]
  7    29 ms    28 ms    28 ms  bro-cr01-be7.135.stk.stream-internet.net [195.34.50.146]
  8    29 ms    28 ms    28 ms  10gigabitethernet1-3.core1.sto1.he.net [194.68.128.187]
  9    52 ms    49 ms    49 ms  100ge12-1.core1.fra1.he.net [184.105.64.105]
 10    62 ms    83 ms    97 ms  100ge5-2.core1.par2.he.net [72.52.92.13]
 11   122 ms   122 ms   122 ms  100ge14-1.core1.nyc4.he.net [184.105.81.77]
 12   204 ms   204 ms   204 ms  100ge13-1.core4.fmt2.he.net [184.105.81.61]
 13   204 ms   204 ms   204 ms  100ge9-1.core1.pao1.he.net [184.105.222.90]
 14   205 ms   205 ms   205 ms  stanford-university.10gigabitethernet1-4.core1.pao1.he.net [216.218.209.118]
 15   193 ms   193 ms   193 ms  woa-west-rtr-vl2.SUNet [171.64.255.132]
 16   192 ms   192 ms   192 ms  vspd3.stanford.edu [171.67.108.157]
 
TRACERT 155.247.166.219
...
  4     4 ms     4 ms     4 ms  mag9-cr01-be12.51.msk.stream-internet.net [212.188.1.5]
  5     4 ms     4 ms     4 ms  a197-cr04-be10.77.msk.stream-internet.net [195.34.50.73]
  6    39 ms    39 ms    39 ms  anc-cr01-be6.77.ff.stream-internet.net [212.188.2.189]
  7     *        *        *  
  8     *        *        *   
  9    46 ms    46 ms    46 ms  Cogent-level3-200G.Frankfurt1.Level3.net [4.68.111.178]
 10    46 ms    46 ms    46 ms  be2845.ccr41.fra03.atlas.cogentco.com [154.54.56.189]
 11    49 ms    49 ms    49 ms  be2813.ccr41.ams03.atlas.cogentco.com [130.117.0.121]
 12   127 ms   126 ms   127 ms  be12194.ccr41.lon13.atlas.cogentco.com [154.54.56.93]
 13   125 ms   125 ms   125 ms  be2317.ccr41.jfk02.atlas.cogentco.com [154.54.30.185]
 14   128 ms   128 ms   128 ms  be2364.rcr21.phl01.atlas.cogentco.com [154.54.3.142]
 15   128 ms   128 ms   128 ms  be3163.rcr21.phl03.atlas.cogentco.com [154.54.7.2]
 16   128 ms   128 ms   128 ms  38.140.148.75
 17     *        *        * 
 18   129 ms   129 ms   129 ms  vav3.ocis.temple.edu [155.247.166.219]
Then I've traced route from another country in Europe (ping.eu), result was even worse:

Code: Select all

traceroute to 155.247.166.219 (155.247.166.219), 30 hops max, 60 byte packets
1	 	 	 	*	*	*		
2	core22.fsn1.hetzner.com	213.239.245.241	de	0.199 ms	0.206 ms	core21.fsn1.hetzner.com	213.239.245.237	de	0.240 ms
3	core4.fra.hetzner.com	213.239.245.18	de	4.927 ms  core4.fra.hetzner.com	213.239.245.14	de	15.886 ms	15.889 ms
4	ffm-b4-link.telia.net	213.248.70.2	 	4.933 ms	4.932 ms	4.923 ms
5	be3031.agr41.fra03.atlas.cogentco.com	130.117.14.197	 	5.308 ms	5.460 ms	5.587 ms
6	be3186.ccr41.fra03.atlas.cogentco.com	130.117.0.1	 	94.037 ms	5.428 ms	93.994 ms
7	be2813.ccr41.ams03.atlas.cogentco.com	130.117.0.121	 	91.335 ms	91.447 ms	11.647 ms
8	be12194.ccr41.lon13.atlas.cogentco.com	154.54.56.93	us	91.258 ms	91.385 ms	91.073 ms
9	be2317.ccr41.jfk02.atlas.cogentco.com	154.54.30.185	us	95.019 ms	95.411 ms	97.563 ms
10	be3471.ccr41.jfk02.atlas.cogentco.com	154.54.40.154	us	91.990 ms	 be2359.ccr42.jfk02.atlas.cogentco.com	154.54.43.110	us	90.935 ms	93.651 ms
11	be2333.rcr22.phl01.atlas.cogentco.com	154.54.5.2	us	93.662 ms	 be3163.rcr21.phl03.atlas.cogentco.com	154.54.7.2	us	96.194 ms	96.303 ms
12	be3164.rcr21.phl03.atlas.cogentco.com	154.54.25.142	us	96.494 ms 		ru		
13	be2364.rcr21.phl01.atlas.cogentco.com	154.54.3.142	us	102.949 ms	 	 ru	
14	 	 	 	*	*	*
15	 	 	 	*	*	*
16	 	 	 	*	*	*
17	 	 	 	*	*	*
18	 	 	 	*	*	*
No reply for 5 hops. Assuming we reached firewall.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: 155.247.166.219 & 155.247.166.220 are now hosed [Not Now

Post by bruce »

tempally wrote:

Code: Select all

02:13:19:WU00:FS01:Uploading 1.79MiB to 155.247.166.219
02:13:42:WU00:FS01:Upload complete

03:57:33:WU01:FS01:Uploading 7.80MiB to 171.67.108.157
03:59:49:WU01:FS01:Upload complete
It does take some time for the server and the client to set up the appropriate path to the right place on the server's disk and to check the validity of the WU's results That has to happen before the connection can be successfully closed. There's a return-code from that processing that needs to be returned to your system. The actual transmission time is a small fraction of the approval's processing time on the server.

I don't understand your concern that the first upload took 23 seconds! How long was that WU on your system and what percentage of the total time was spent communicating and validating? The second WU took longer because the WU was more complex ... i'll bet it tt took longer on your system and the validation is proportionally longer. Again, what percentage of the total WU processing time did the 2 minutes 16 seconds take?.

The tract-route's look about as I'd expect them to.
Post Reply