Azure servers down 40.121.152.108 / 52.224.109.74

Moderators: Site Moderators, FAHC Science Team

Azure servers down 40.121.152.108 / 52.224.109.74

Postby comixgoddess » Fri Jul 17, 2020 8:11 pm

I have been getting the following string of log entries for the last 20 minutes or so --

19:00:34:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14570 run:0 clone:1384 gen:222 core:0xa7 unit:0x00000103287234c95e7eea1b6620dfda
19:00:34:WU01:FS00:Uploading 6.82MiB to 40.114.52.201
19:00:34:WU01:FS00:Connecting to 40.114.52.201:8080
19:00:34:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
19:00:34:WU01:FS00:Connecting to 40.114.52.201:80
19:00:34:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: Connection refused
19:00:34:WU01:FS00:Trying to send results to collection server
19:00:34:WU01:FS00:Uploading 6.82MiB to 52.224.109.74
19:00:34:WU01:FS00:Connecting to 52.224.109.74:8080
19:00:34:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
19:00:34:WU01:FS00:Connecting to 52.224.109.74:80
19:00:34:ERROR:WU01:FS00:Exception: Failed to connect to 52.224.109.74:80: Connection refused

Looking at the server stats page, 40.114.52.201 is showing as "Down" while 52.224.109.74 is showing that it should be accepting returned results. Is there anything I should be doing on my end to help this along? Thanks!
Image
User avatar
comixgoddess
 
Posts: 44
Joined: Wed Apr 08, 2020 10:57 pm
Location: Pacific Northwest

Azure servers down 40.121.152.108 / 52.224.109.74

Postby itskieran » Fri Jul 17, 2020 8:32 pm

I noticed I was stuck sending with the next CPU WU on 5% already, so I investigated.

The WS is 40.114.52.201 and the CS is 52.224.109.74

It seems that a lot of servers (8) are down according to the server stats

Someone might want to take a look.
itskieran
 
Posts: 4
Joined: Sat Mar 14, 2020 4:43 pm

Re: Large number of servers down

Postby matrix1999 » Fri Jul 17, 2020 9:06 pm

Yes, same here. My WU is failed to upload to 52.224.109.74. And according to the server stats as you posted, there are many servers being down at the moment, namely eastus.cloudapp.azure.com, seas.wustl.edu, temple.edu and some others. Can someone look into it, please?
matrix1999
 
Posts: 4
Joined: Wed Apr 22, 2020 2:18 am

Re: Large number of servers down

Postby bollix47 » Fri Jul 17, 2020 9:12 pm

Apparently the azure servers are experiencing a problem and development is currently looking into said problem ... hopefully it will be fixed 'soon'.

I too have a few WUs that I can't return so I will be keeping an 'eye' on events and will let you know if anything new develops.
bollix47
 
Posts: 2871
Joined: Sun Dec 02, 2007 6:04 am
Location: Canada

Re: Cannot upload to 40.114.52.201

Postby bollix47 » Fri Jul 17, 2020 9:20 pm

bollix47
 
Posts: 2871
Joined: Sun Dec 02, 2007 6:04 am
Location: Canada

Re: Large number of servers down

Postby Foxbat » Sat Jul 18, 2020 12:51 am

The UV index must be 10 because there isn't a working Cloud in the Azure Sky…

(sorry)

Glad to see someone is working on this. So far I have just the one WU trying to upload.
Image
Foxbat
 
Posts: 92
Joined: Wed Dec 05, 2007 11:23 pm
Location: Michiana, USA

WU not sending (40.114.52.201 and 52.224.109.74)

Postby Familyman_19 » Sat Jul 18, 2020 3:24 am

I have a completed work unit that has been stuck for several hours. The log shows the following errors:

02:04:46:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: No connection could be made because the target machine actively refused it.
02:04:51:ERROR:WU01:FS00:Exception: Failed to connect to 52.224.109.74:80: No connection could be made because the target machine actively refused it.

It keeps doing this over and over. Other WUs have completed and have been sent back just fine. Any ideas?
Familyman_19
 
Posts: 13
Joined: Sat Jul 18, 2020 3:20 am

Re: WU not sending

Postby comixgoddess » Sat Jul 18, 2020 6:39 am

Same here; mine has been "stuck" for 7 hours now. Please see this thread - viewtopic.php?f=18&t=35812.
User avatar
comixgoddess
 
Posts: 44
Joined: Wed Apr 08, 2020 10:57 pm
Location: Pacific Northwest

Re: WU not sending

Postby RichieDoubleU » Sat Jul 18, 2020 9:14 pm

Same with me: this WU doesn't get sent since over 12 hours now, while another WU has been processed and sent successfully. So right now I got stuck with 13851.
Here one sample of the meanwhile very lengthy log.
project:13851 run:0 clone:8229 gen:208 core:0xa7 unit:0x000000fe287234c95e72ea9026ea9b9b
20:01:40:WU00:FS00:Uploading 2.47MiB to 40.114.52.201
20:01:40:WU00:FS00:Connecting to 40.114.52.201:8080
20:01:40:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
20:01:40:WU00:FS00:Connecting to 40.114.52.201:80
20:01:40:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: Connection refused

Question: Can I do anything to solve this problem myself? I'd rather think not...
For the time being I've stopped folding and would prefer to have this thing solved before I start folding again.
RichieDoubleU
 
Posts: 3
Joined: Sat Jul 18, 2020 9:03 pm
Location: Germany

Re: Large number of servers down

Postby Joe_H » Sat Jul 18, 2020 9:44 pm

Foxbat wrote:The UV index must be 10 because there isn't a working Cloud in the Azure Sky…


One of the five servers on Azure is up and running, waiting on information as to when others will be back.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Joe_H
Site Admin
 
Posts: 6675
Joined: Tue Apr 21, 2009 5:41 pm
Location: W. MA

Re: WU not sending

Postby Neil-B » Sat Jul 18, 2020 10:26 pm

There are a number of servers down at the moment so until they are up again completed WUs for those servers will be unable to upload .. since they are down they wont be issuing any more WUs - let your client handle this (it will retry until the server is up and it uploads or until it passes expiration and is dumped by the client) and keeping folding from the servers that are up would be the normal approach (the client is designed to work this way) .. but if you wish to put a hold on folding until the WU clears that is obviously a perfectly ok choice - whether you fold or not wont make any difference to how quickly the completed WU clears.
1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro, Quadro M1000M 2GB, FAH 7.6.21
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro, GTX 750Ti 2GB, FAH 7.6.21
Neil-B
 
Posts: 1490
Joined: Sun Mar 22, 2020 6:52 pm
Location: UK

Re: WU not sending

Postby RichieDoubleU » Sat Jul 18, 2020 11:29 pm

Neil, thanks for the info. I understand it better now.
RichieDoubleU
 
Posts: 3
Joined: Sat Jul 18, 2020 9:03 pm
Location: Germany

Re: WU not sending

Postby Neil-B » Sat Jul 18, 2020 11:47 pm

It is a real pain (for everyone, folders, researchers, devs) when this happens cause it holds up the science and everyone gets frustrated as it in effect "wastes" effort and slows progress ... but issues happen - believe me, the researchers and devs behind the scenes will be doing the best they can to get the issues resolved asap - however that doesn't make it any less annoying ... in time one either has to be patient (which I am really bad at) or learn to look at the logs/control interfaces less often and have faith things are working/will sort themselves out !! ... I spotted in another thread that they have got one of the servers back up (hopefully functioning properly) but when the others will follow is anyones guess - and as usual it is a weekend so trying to fix stuff is harder/slower :(
Neil-B
 
Posts: 1490
Joined: Sun Mar 22, 2020 6:52 pm
Location: UK

Re: WU not sending (40.114.52.201 and 52.224.109.74)

Postby bruce » Sun Jul 19, 2020 12:36 am

There are reports that foreign hackers are targeting COVID research. Subject: Cozy Bear (APT-29) claws Coronavirus research from the West.

Yes, there are several servers down and people are working on fixing them. I don't know if there's any connection with the hackers, but it would not surpise me to learn that there's a connection.
bruce
 
Posts: 20122
Joined: Thu Nov 29, 2007 11:13 pm
Location: So. Cal.

Re: WU not sending (40.114.52.201 and 52.224.109.74)

Postby psaam0001 » Sun Jul 19, 2020 2:10 am

I know I have 4 WU's (so far) that are waiting to go to a collection server...

May the ultimate social distancing regulator separate these uncouth hackers from their tools--permanently!

Paul
psaam0001
 
Posts: 133
Joined: Mon May 18, 2020 3:02 am

Next

Return to Issues with a specific server

Who is online

Users browsing this forum: Ton80 and 3 guests

cron