Stuck "Send" to 52.224.109.74

Moderators: Site Moderators, FAHC Science Team

Stuck "Send" to 52.224.109.74

Postby tirreus » Tue Apr 07, 2020 4:49 pm

Hi,

I am stuck at 100% in status "Send" for a day already in project 14369 (1207, 3, 4).

System log:
15:45:53:WU01:FS00:Trying to send results to collection server
15:45:53:WU01:FS00:Uploading 8.27MiB to 52.224.109.74
15:45:53:WU01:FS00:Connecting to 52.224.109.74:8080
15:46:14:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
15:46:14:WU01:FS00:Connecting to 52.224.109.74:80
15:46:36:ERROR:WU01:FS00:Exception: Failed to connect to 52.224.109.74:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

And repeats over and over and over...
tirreus
 
Posts: 5
Joined: Fri Mar 20, 2020 7:31 pm

Re: Stuck at 100% in status "Send"

Postby Jesse_V » Tue Apr 07, 2020 4:54 pm

That work server is up for me. Can you get to 52.224.109.74:80 in your web browser?
F@h is now the top computing platform on the planet and nothing unites people like a dedicated fight against a common enemy. This virus affects all of us. Lets end it together.
Jesse_V
Site Moderator
 
Posts: 2851
Joined: Mon Jul 18, 2011 5:44 am
Location: Western Washington

Re: Stuck at 100% in status "Send"

Postby bruce » Tue Apr 07, 2020 4:59 pm

The 100% notation isn't really applicable. It reports the status of the WU ... i.e.- that computations have been completed.

As far as http://52.224.109.74 is concerned, it seems to be working now. I'm not sure what happened earlier. What range of times is denoted by "over and over and over"?
bruce
 
Posts: 20004
Joined: Thu Nov 29, 2007 11:13 pm
Location: So. Cal.

Re: Stuck at 100% in status "Send"

Postby uyaem » Tue Apr 07, 2020 8:55 pm

The server is reachable for me, but only after a significant amount of time. Load-related (on azure? hmmm)?
It's showing a warning on the https://apps.foldingathome.org/serverstats page.
Image
CPU: Ryzen 9 3900X (1x21 CPUs) ~ GPU: nVidia GeForce GTX 1660 Super (Asus)
uyaem
 
Posts: 222
Joined: Sat Mar 21, 2020 8:35 pm
Location: Esslingen, Germany

Re: Stuck at 100% in status "Send"

Postby aol » Wed Apr 08, 2020 2:27 am

I have this issue with 52.224.109.74 as well.
Many attempts to send results have failed to upload to this collection server.
http : // fah4 . eastus . cloudapp . azure . com:80 takes at least a minute to load, but always fails with a connection reset error first before Chrome tries again.

From the FAHControl Log:
Code: Select all
01:12:29:WU01:FS01:Uploading 197.17MiB to 13.82.98.119
01:12:29:WU01:FS01:Connecting to 13.82.98.119:8080
01:12:29:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
01:12:29:WU01:FS01:Trying to send results to collection server
01:12:29:WU01:FS01:Uploading 197.17MiB to 52.224.109.74
01:12:29:WU01:FS01:Connecting to 52.224.109.74:8080
01:13:12:WU01:FS01:Upload 0.10%
01:13:12:ERROR:WU01:FS01:Exception: Transfer failed


So fah3 . eastus . cloudapp . azure . com (the "Work Server" listed in FAHControl) seems to only assign and is rejecting answers.
Then it tries to send the work to fah4, but it only gets 0.10% of the way before too many connections to fah4 (I assume) make the server drop the connection, resulting in the "Transfer failed" exception.
From the server stats page, it looks like fah1, fah2, fah3, and fah5 on the Azure Cloud are all assign-only and are all directing completed work to this one poor fah4 server...
Why the stark contrast in number of servers listed as "Assign" to the number of servers listed as "Accept" (not to mention many of which seem to be down or error'd out)?
Last edited by aol on Wed Apr 08, 2020 9:22 pm, edited 1 time in total.
aol
 
Posts: 2
Joined: Wed Apr 08, 2020 1:58 am

Re: Stuck at 100% in status "Send"

Postby Neil-B » Wed Apr 08, 2020 7:45 am

Assign means 'assign and accept' whereas Accept means 'accept only'
1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent, Quadro K420 1GB, FAH 7.6.13
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro, Quadro M1000M 2GB, FAH 7.6.13
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro, GTX 750Ti 2GB, FAH 7.6.13
Neil-B
 
Posts: 1405
Joined: Sun Mar 22, 2020 6:52 pm
Location: UK

Re: Stuck at 100% in status "Send"

Postby tirreus » Wed Apr 08, 2020 10:09 am

The problem still persists.
I basically wrote this post in a hope that FAH admins or Project owner is informed and make correction. Apparently, this did not happen.
The run will expire in few hours, so the work will be discarded and lost (and it is not nice, we are giving our HW for use and this way it is wasted, I would expect somebody cares about the system and projects).
Then, new WU will be downloaded, I guess, hopefully I will be luckier and this time I will get work for project owner who needs the results and cares about work being done for him... ;)
tirreus
 
Posts: 5
Joined: Fri Mar 20, 2020 7:31 pm

Re: Stuck at 100% in status "Send"

Postby MatthewM » Wed Apr 08, 2020 6:20 pm

I now too have 2 WUs stuck in send, both are to 52.224.109.74 - I have been fine for weeks, then it started late yesterday - I just can't see this being on my end as nothing changed and internet works fine.

16:51:09:WARNING:WU02:FS01:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
16:52:14:ERROR:WU02:FS01:Exception: Transfer failed
16:52:26:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
16:53:30:ERROR:WU02:FS01:Exception: Transfer failed
MatthewM
 
Posts: 5
Joined: Fri Mar 27, 2020 8:43 pm

Re: Stuck at 100% in status "Send"

Postby PantherX » Thu Apr 09, 2020 10:57 am

tirreus wrote:...I would expect somebody cares about the system and projects...

The F@H Team do care for all the donors and the work everyone has put in.

If we step back and consider the following:
1) Global Pandemic that impacts everyone in 180+ countries. Keep in mind that a large portion of the F@H Team is based in USA and currently, they have a pandemic situation.
2) The F@H Team is understaffed from day 1. That's the nature of research so it's an up-hill battle for most of the days.
3) The F@H Team is staffed by humans... they need to eat, sleep, rest, think and have lives outside of "work" (families, responsibilities, etc.).
4) There's an Easter break in USA which typically means that people take time off work for R&R and focus on themselves.
5) The F@H Team's focus is to meet the the supply of WUs by increasing the F@H Server capacity... not an easy job as they have mostly on-premise hardware and are not 100% cloud.
6) The F@H Team has a single developer that looks after all the code for everything... a single human can't do several tasks at the time.
7) The F@H Team have some amazing corporate support and are working with them to increase the F@H Server capacity.

Looking at the bigger picture, I hope you understand that there's a lot that's happening in the background to meet the demands for WUs, you keep the donors up-to-date and to manage their own lives during the pandemic. If you're keen to see how you can help F@H, why not join in for Developer Fireside Chat: viewtopic.php?f=16&t=34136
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
User avatar
PantherX
Site Moderator
 
Posts: 6746
Joined: Wed Dec 23, 2009 10:33 am
Location: Land Of The Long White Cloud

Re: Stuck at 100% in status "Send"

Postby iceman1992 » Thu Apr 09, 2020 11:32 am

PantherX wrote:6) The F@H Team has a single developer that looks after all the code for everything... a single human can't do several tasks at the time

This to me jumps out as a big problem. A single developer for everything? Including clients, controls, cores? Do they not have the resources to have a small dev team?
iceman1992
 
Posts: 527
Joined: Fri Mar 23, 2012 6:16 pm

Re: Stuck at 100% in status "Send"

Postby PantherX » Thu Apr 09, 2020 12:11 pm

iceman1992 wrote:
PantherX wrote:6) The F@H Team has a single developer that looks after all the code for everything... a single human can't do several tasks at the time

This to me jumps out as a big problem. A single developer for everything? Including clients, controls, cores? Do they not have the resources to have a small dev team?

Yep, a single developer Joseph Coffland who has been working with the F@H Team for several years. He looks after V7 clients (Linux, Mac, Windows), the code for all F@H Servers (AS, WS, CS) and some of the websites. Keep in mind that he also has other clients and projects to juggle so isn't an exclusive F@H developer. The budget that the F@H Team operates on is very slim.

Other F@H Team members might help out a bit in bits and peices but keep in mind that they aren't a developer, instead their background is post-doc and grads who study protein folding.
User avatar
PantherX
Site Moderator
 
Posts: 6746
Joined: Wed Dec 23, 2009 10:33 am
Location: Land Of The Long White Cloud

Re: Stuck at 100% in status "Send"

Postby iceman1992 » Thu Apr 09, 2020 12:26 pm

PantherX wrote:Yep, a single developer Joseph Coffland who has been working with the F@H Team for several years. He looks after V7 clients (Linux, Mac, Windows), the code for all F@H Servers (AS, WS, CS) and some of the websites. Keep in mind that he also has other clients and projects to juggle so isn't an exclusive F@H developer. The budget that the F@H Team operates on is very slim.

I can probably help out a little bit with the web stuff, I'm a webapp developer. Might be able to help with fahcontrol, looks like it's written in Python? Not sure what the servers are using
iceman1992
 
Posts: 527
Joined: Fri Mar 23, 2012 6:16 pm

Re: Stuck at 100% in status "Send"

Postby jcoffland » Mon Apr 13, 2020 5:41 am

PantherX wrote:a single human can't do several tasks at the time

Of course I can.

In all seriousness. Folding@home needs more money. My income for the year in scraped together by four different universities. COVID-19 has blown this years budget to hell. We've had lots of companies helping with hardware and many volunteers but getting cash is much harder. For the record, you can donate to WUSTL via https://foldingathome.org/donate and 100% of it goes to Folding@home.
Cauldron Development LLC
http://cauldrondevelopment.com/
User avatar
jcoffland
Pande Group Member
 
Posts: 1017
Joined: Fri Oct 10, 2008 7:42 pm
Location: San Jose, CA

Re: Stuck at 100% in status "Send"

Postby anandhanju » Mon Apr 13, 2020 6:06 am

Can you please post the section of the log that has the Project, Run, Clone and Gen numbers when it is trying to upload as well as the line that has the upload size? This will help the researchers get to the root of the problem sooner.

For example,
12:35:55:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:13878 run:0 clone:622 gen:17 core:0x22 unit:0x0000001334e06d4a5e80cfe592ad2446
12:35:55:WU01:FS01:Uploading 48.05MiB to 52.224.109.74
anandhanju
 
Posts: 517
Joined: Mon Dec 03, 2007 5:33 am
Location: Australia

Re: Stuck at 100% in status "Send"

Postby iceman1992 » Mon Apr 13, 2020 6:15 am

jcoffland wrote:For the record, you can donate to WUSTL via https://foldingathome.org/donate and 100% of it goes to Folding@home.

Do I need to fill in my complete address information there? I don't live in the US so it shouldn't be relevant, right?
iceman1992
 
Posts: 527
Joined: Fri Mar 23, 2012 6:16 pm

Next

Return to Issues with a specific server

Who is online

Users browsing this forum: No registered users and 3 guests

cron