Stuck "Send" to 52.224.109.74

Moderators: Site Moderators, FAHC Science Team

tirreus
Posts: 5
Joined: Fri Mar 20, 2020 6:31 pm

Stuck "Send" to 52.224.109.74

Post by tirreus »

Hi,

I am stuck at 100% in status "Send" for a day already in project 14369 (1207, 3, 4).

System log:
15:45:53:WU01:FS00:Trying to send results to collection server
15:45:53:WU01:FS00:Uploading 8.27MiB to 52.224.109.74
15:45:53:WU01:FS00:Connecting to 52.224.109.74:8080
15:46:14:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
15:46:14:WU01:FS00:Connecting to 52.224.109.74:80
15:46:36:ERROR:WU01:FS00:Exception: Failed to connect to 52.224.109.74:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

And repeats over and over and over...
Jesse_V
Site Moderator
Posts: 2851
Joined: Mon Jul 18, 2011 4:44 am
Hardware configuration: OS: Windows 10, Kubuntu 19.04
CPU: i7-6700k
GPU: GTX 970, GTX 1080 TI
RAM: 24 GB DDR4
Location: Western Washington

Re: Stuck at 100% in status "Send"

Post by Jesse_V »

That work server is up for me. Can you get to 52.224.109.74:80 in your web browser?
F@h is now the top computing platform on the planet and nothing unites people like a dedicated fight against a common enemy. This virus affects all of us. Lets end it together.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Stuck at 100% in status "Send"

Post by bruce »

The 100% notation isn't really applicable. It reports the status of the WU ... i.e.- that computations have been completed.

As far as http://52.224.109.74 is concerned, it seems to be working now. I'm not sure what happened earlier. What range of times is denoted by "over and over and over"?
uyaem
Posts: 222
Joined: Sat Mar 21, 2020 7:35 pm
Location: Esslingen, Germany

Re: Stuck at 100% in status "Send"

Post by uyaem »

The server is reachable for me, but only after a significant amount of time. Load-related (on azure? hmmm)?
It's showing a warning on the https://apps.foldingathome.org/serverstats page.
Image
CPU: Ryzen 9 3900X (1x21 CPUs) ~ GPU: nVidia GeForce GTX 1660 Super (Asus)
aol
Posts: 2
Joined: Wed Apr 08, 2020 12:58 am

Re: Stuck at 100% in status "Send"

Post by aol »

I have this issue with 52.224.109.74 as well.
Many attempts to send results have failed to upload to this collection server.
http : // fah4 . eastus . cloudapp . azure . com:80 takes at least a minute to load, but always fails with a connection reset error first before Chrome tries again.

From the FAHControl Log:

Code: Select all

01:12:29:WU01:FS01:Uploading 197.17MiB to 13.82.98.119
01:12:29:WU01:FS01:Connecting to 13.82.98.119:8080
01:12:29:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
01:12:29:WU01:FS01:Trying to send results to collection server
01:12:29:WU01:FS01:Uploading 197.17MiB to 52.224.109.74
01:12:29:WU01:FS01:Connecting to 52.224.109.74:8080
01:13:12:WU01:FS01:Upload 0.10%
01:13:12:ERROR:WU01:FS01:Exception: Transfer failed
So fah3 . eastus . cloudapp . azure . com (the "Work Server" listed in FAHControl) seems to only assign and is rejecting answers.
Then it tries to send the work to fah4, but it only gets 0.10% of the way before too many connections to fah4 (I assume) make the server drop the connection, resulting in the "Transfer failed" exception.
From the server stats page, it looks like fah1, fah2, fah3, and fah5 on the Azure Cloud are all assign-only and are all directing completed work to this one poor fah4 server...
Why the stark contrast in number of servers listed as "Assign" to the number of servers listed as "Accept" (not to mention many of which seem to be down or error'd out)?
Last edited by aol on Wed Apr 08, 2020 8:22 pm, edited 1 time in total.
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Stuck at 100% in status "Send"

Post by Neil-B »

Assign means 'assign and accept' whereas Accept means 'accept only'
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
tirreus
Posts: 5
Joined: Fri Mar 20, 2020 6:31 pm

Re: Stuck at 100% in status "Send"

Post by tirreus »

The problem still persists.
I basically wrote this post in a hope that FAH admins or Project owner is informed and make correction. Apparently, this did not happen.
The run will expire in few hours, so the work will be discarded and lost (and it is not nice, we are giving our HW for use and this way it is wasted, I would expect somebody cares about the system and projects).
Then, new WU will be downloaded, I guess, hopefully I will be luckier and this time I will get work for project owner who needs the results and cares about work being done for him... ;)
MatthewM
Posts: 5
Joined: Fri Mar 27, 2020 7:43 pm

Re: Stuck at 100% in status "Send"

Post by MatthewM »

I now too have 2 WUs stuck in send, both are to 52.224.109.74 - I have been fine for weeks, then it started late yesterday - I just can't see this being on my end as nothing changed and internet works fine.

16:51:09:WARNING:WU02:FS01:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
16:52:14:ERROR:WU02:FS01:Exception: Transfer failed
16:52:26:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
16:53:30:ERROR:WU02:FS01:Exception: Transfer failed
PantherX
Site Moderator
Posts: 7020
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: Stuck at 100% in status "Send"

Post by PantherX »

tirreus wrote:...I would expect somebody cares about the system and projects...
The F@H Team do care for all the donors and the work everyone has put in.

If we step back and consider the following:
1) Global Pandemic that impacts everyone in 180+ countries. Keep in mind that a large portion of the F@H Team is based in USA and currently, they have a pandemic situation.
2) The F@H Team is understaffed from day 1. That's the nature of research so it's an up-hill battle for most of the days.
3) The F@H Team is staffed by humans... they need to eat, sleep, rest, think and have lives outside of "work" (families, responsibilities, etc.).
4) There's an Easter break in USA which typically means that people take time off work for R&R and focus on themselves.
5) The F@H Team's focus is to meet the the supply of WUs by increasing the F@H Server capacity... not an easy job as they have mostly on-premise hardware and are not 100% cloud.
6) The F@H Team has a single developer that looks after all the code for everything... a single human can't do several tasks at the time.
7) The F@H Team have some amazing corporate support and are working with them to increase the F@H Server capacity.

Looking at the bigger picture, I hope you understand that there's a lot that's happening in the background to meet the demands for WUs, you keep the donors up-to-date and to manage their own lives during the pandemic. If you're keen to see how you can help F@H, why not join in for Developer Fireside Chat: viewtopic.php?f=16&t=34136
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
iceman1992
Posts: 527
Joined: Fri Mar 23, 2012 5:16 pm

Re: Stuck at 100% in status "Send"

Post by iceman1992 »

PantherX wrote:6) The F@H Team has a single developer that looks after all the code for everything... a single human can't do several tasks at the time
This to me jumps out as a big problem. A single developer for everything? Including clients, controls, cores? Do they not have the resources to have a small dev team?
PantherX
Site Moderator
Posts: 7020
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: Stuck at 100% in status "Send"

Post by PantherX »

iceman1992 wrote:
PantherX wrote:6) The F@H Team has a single developer that looks after all the code for everything... a single human can't do several tasks at the time
This to me jumps out as a big problem. A single developer for everything? Including clients, controls, cores? Do they not have the resources to have a small dev team?
Yep, a single developer Joseph Coffland who has been working with the F@H Team for several years. He looks after V7 clients (Linux, Mac, Windows), the code for all F@H Servers (AS, WS, CS) and some of the websites. Keep in mind that he also has other clients and projects to juggle so isn't an exclusive F@H developer. The budget that the F@H Team operates on is very slim.

Other F@H Team members might help out a bit in bits and peices but keep in mind that they aren't a developer, instead their background is post-doc and grads who study protein folding.
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
iceman1992
Posts: 527
Joined: Fri Mar 23, 2012 5:16 pm

Re: Stuck at 100% in status "Send"

Post by iceman1992 »

PantherX wrote:Yep, a single developer Joseph Coffland who has been working with the F@H Team for several years. He looks after V7 clients (Linux, Mac, Windows), the code for all F@H Servers (AS, WS, CS) and some of the websites. Keep in mind that he also has other clients and projects to juggle so isn't an exclusive F@H developer. The budget that the F@H Team operates on is very slim.
I can probably help out a little bit with the web stuff, I'm a webapp developer. Might be able to help with fahcontrol, looks like it's written in Python? Not sure what the servers are using
jcoffland
Site Admin
Posts: 1019
Joined: Fri Oct 10, 2008 6:42 pm
Location: Helsinki, Finland
Contact:

Re: Stuck at 100% in status "Send"

Post by jcoffland »

PantherX wrote:a single human can't do several tasks at the time
Of course I can.

In all seriousness. Folding@home needs more money. My income for the year in scraped together by four different universities. COVID-19 has blown this years budget to hell. We've had lots of companies helping with hardware and many volunteers but getting cash is much harder. For the record, you can donate to WUSTL via https://foldingathome.org/donate and 100% of it goes to Folding@home.
Cauldron Development LLC
http://cauldrondevelopment.com/
anandhanju
Posts: 526
Joined: Mon Dec 03, 2007 4:33 am
Location: Australia

Re: Stuck at 100% in status "Send"

Post by anandhanju »

Can you please post the section of the log that has the Project, Run, Clone and Gen numbers when it is trying to upload as well as the line that has the upload size? This will help the researchers get to the root of the problem sooner.

For example,
12:35:55:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:13878 run:0 clone:622 gen:17 core:0x22 unit:0x0000001334e06d4a5e80cfe592ad2446
12:35:55:WU01:FS01:Uploading 48.05MiB to 52.224.109.74
iceman1992
Posts: 527
Joined: Fri Mar 23, 2012 5:16 pm

Re: Stuck at 100% in status "Send"

Post by iceman1992 »

jcoffland wrote:For the record, you can donate to WUSTL via https://foldingathome.org/donate and 100% of it goes to Folding@home.
Do I need to fill in my complete address information there? I don't live in the US so it shouldn't be relevant, right?
Post Reply