Send Errors - 155.247.164.213 & .214

Moderators: Site Moderators, FAHC Science Team

Re: Send Errors - 155.247.164.213 & .214

Postby tsunami2311 » Sat Mar 21, 2020 6:15 pm

I was going crazy other night wonder why the WU isnt being sent Then I cam here and read same thing..

Code: Select all
*********************** Log Started 2020-03-21T17:00:12Z ***********************
17:00:12:************************* Folding@home Client *************************
17:00:12:        Website: https://foldingathome.org/
17:00:12:      Copyright: (c) 2009-2018 foldingathome.org
17:00:12:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:00:12:           Args: --open-web-control
17:00:12:         Config: C:\Users\****\AppData\Roaming\FAHClient\config.xml
17:00:12:******************************** Build ********************************
17:00:12:        Version: 7.5.1
17:00:12:           Date: May 11 2018
17:00:12:           Time: 13:06:32
17:00:12:     Repository: Git
17:00:12:       Revision: 4705bf53c635f88b8fe85af7675557e15d491ff0
17:00:12:         Branch: master
17:00:12:       Compiler: Visual C++ 2008
17:00:12:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
17:00:12:       Platform: win32 10
17:00:12:           Bits: 32
17:00:12:           Mode: Release
17:00:12:******************************* System ********************************
17:00:12:            CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
17:00:12:         CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
17:00:12:           CPUs: 8
17:00:12:         Memory: 15.95GiB
17:00:12:    Free Memory: 13.19GiB
17:00:12:        Threads: WINDOWS_THREADS
17:00:12:     OS Version: 6.2
17:00:12:    Has Battery: false
17:00:12:     On Battery: false
17:00:12:     UTC Offset: -4
17:00:12:            PID: 1016
17:00:12:            CWD: C:\Users\*****\AppData\Roaming\FAHClient
17:00:12:             OS: Windows 10 Enterprise
17:00:12:        OS Arch: AMD64
17:00:12:           GPUs: 1
17:00:12:          GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1070 Ti] 8186
17:00:12:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:10.2
17:00:12:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:442.59
17:00:12:  Win32 Service: false
17:00:12:***********************************************************************
17:00:12:<config>
17:00:12:  <!-- Folding Core -->
17:00:12:  <gpu-usage v='50'/>
17:00:12:
17:00:12:  <!-- Network -->
17:00:12:  <proxy v=':8080'/>
17:00:12:
17:00:12:  <!-- User Information -->
17:00:12:  <team v='69411'/>
17:00:12:  <user v='tsunami231'/>
17:00:12:
17:00:12:  <!-- Work Unit Control -->
17:00:12:  <max-units v='3'/>
17:00:12:  <next-unit-percentage v='98'/>
17:00:12:
17:00:12:  <!-- Folding Slots -->
17:00:12:  <slot id='0' type='CPU'>
17:00:12:    <cpus v='3'/>
17:00:12:    <paused v='true'/>
17:00:12:  </slot>
17:00:12:  <slot id='1' type='GPU'>
17:00:12:    <paused v='true'/>
17:00:12:  </slot>
17:00:12:</config>
17:00:12:Trying to access database...
17:00:12:Successfully acquired database lock
17:00:12:Enabled folding slot 00: PAUSED cpu:3 (by user)
17:00:12:Enabled folding slot 01: PAUSED gpu:0:GP104 [GeForce GTX 1070 Ti] 8186 (by user)
17:00:12:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:2325 gen:0 core:0x22 unit:0x000000069bf7a4d55e6d771534b82be1
17:00:12:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
17:00:12:WU00:FS01:Connecting to 155.247.164.213:8080
17:00:13:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
17:00:13:WU00:FS01:Trying to send results to collection server
17:00:13:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
17:00:13:WU00:FS01:Connecting to 155.247.164.214:8080
17:00:13:ERROR:WU00:FS01:Exception: Transfer failed
17:00:13:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:2325 gen:0 core:0x22 unit:0x000000069bf7a4d55e6d771534b82be1
17:00:13:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
17:00:13:WU00:FS01:Connecting to 155.247.164.213:8080
17:00:13:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
17:00:13:WU00:FS01:Trying to send results to collection server
17:00:13:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
17:00:13:WU00:FS01:Connecting to 155.247.164.214:8080
17:00:13:ERROR:WU00:FS01:Exception: Transfer failed
17:00:14:13:127.0.0.1:New Web connection
17:01:13:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:2325 gen:0 core:0x22 unit:0x000000069bf7a4d55e6d771534b82be1
17:01:13:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
17:01:13:WU00:FS01:Connecting to 155.247.164.213:8080
17:01:13:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
17:01:13:WU00:FS01:Trying to send results to collection server
17:01:13:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
17:01:13:WU00:FS01:Connecting to 155.247.164.214:8080
17:01:13:ERROR:WU00:FS01:Exception: Transfer failed
17:02:50:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:2325 gen:0 core:0x22 unit:0x000000069bf7a4d55e6d771534b82be1
17:02:50:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
17:02:50:WU00:FS01:Connecting to 155.247.164.213:8080
17:02:50:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
17:02:50:WU00:FS01:Trying to send results to collection server
17:02:50:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
17:02:50:WU00:FS01:Connecting to 155.247.164.214:8080
17:02:51:ERROR:WU00:FS01:Exception: Transfer failed
17:05:28:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:2325 gen:0 core:0x22 unit:0x000000069bf7a4d55e6d771534b82be1
17:05:28:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
17:05:28:WU00:FS01:Connecting to 155.247.164.213:8080
17:05:28:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
17:05:28:WU00:FS01:Trying to send results to collection server
17:05:28:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
17:05:28:WU00:FS01:Connecting to 155.247.164.214:8080
17:05:28:ERROR:WU00:FS01:Exception: Transfer failed
17:09:42:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:2325 gen:0 core:0x22 unit:0x000000069bf7a4d55e6d771534b82be1
17:09:42:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
17:09:42:WU00:FS01:Connecting to 155.247.164.213:8080
17:09:44:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
17:09:44:WU00:FS01:Trying to send results to collection server
17:09:44:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
17:09:44:WU00:FS01:Connecting to 155.247.164.214:8080
17:09:48:ERROR:WU00:FS01:Exception: Transfer failed


I been turn off the restarting the client and once the "retry attempts start to take 10+ minutes, I dont mind doing WU on my GPU or CPU but sit there for hours doing the work with both cpu and gpu pushing 70c just to not be able to send the results then it was special if it times out. Either Turn the server off till it fixed there is not point assigning work if it cant be collected. just be comes waste on time and engery on both parties
tsunami2311
 
Posts: 5
Joined: Sat Mar 21, 2020 6:08 pm

Re: Send Errors - 155.247.164.213 & .214

Postby vnicolici » Sat Mar 21, 2020 7:57 pm

vangli wrote:Found just the same as Whittle . Transmission ends with HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE analyzing with wireshark. One possibility seems to be that the collector isn't able to receive a chunk of 55 Mbyte . Not very uncommon in a web server setup. Could it be that a reconfiguration to allow chunks of 64 Mbyte or something like that will help? Wireshark analyze show that a connection is established, but then disconnected a part out in transmission


Issue persists for me as well. I'm unable to upload two units finished Monday and Tuesday, and Wireshark shows the same error:

Code: Select all
HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE\r\n
<html><head><title>413 HTTP_REQUEST_ENTITY_TOO_LARGE</title></head><body><h1>413 HTTP_REQUEST_ENTITY_TOO_LARGE</h1></body></html>


Corresponding log entry:

Code: Select all
18:34:52:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:5213 gen:0 core:0x22 unit:0x000000009bf7a4d55e6e8abff76953ed
18:34:52:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
18:34:52:WU00:FS01:Connecting to 155.247.164.213:8080
18:34:53:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
18:34:53:WU00:FS01:Trying to send results to collection server
18:34:53:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
18:34:53:WU00:FS01:Connecting to 155.247.164.214:8080
18:34:55:ERROR:WU00:FS01:Exception: Transfer failed


A list of the most recent failure replies from .213 and .214 when trying to upload, as seen with Wireshark:

Code: Select all
No.   Timestamp   Source   Destination   Length   Info   Src port   Dest port
93   2020-03-21 18:30:38.018503   155.247.164.213   192.168.32.135   60   HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE  (text/html)   8080   54675
155   2020-03-21 18:30:38.282700   155.247.164.214   192.168.32.135   60   HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE  (text/html)   8080   54676
185   2020-03-21 18:30:38.292086   155.247.164.213   192.168.32.135   60   HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE  (text/html)   8080   54677
216   2020-03-21 18:30:38.908294   155.247.164.214   192.168.32.135   60   HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE  (text/html)   8080   54680
653   2020-03-21 18:32:15.336840   155.247.164.213   192.168.32.135   60   HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE  (text/html)   8080   54729
742   2020-03-21 18:32:15.627243   155.247.164.213   192.168.32.135   60   HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE  (text/html)   8080   54731
874   2020-03-21 18:32:16.442598   155.247.164.214   192.168.32.135   60   HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE  (text/html)   8080   54732
876   2020-03-21 18:32:16.526702   155.247.164.214   192.168.32.135   316   HTTP/1.1 503 HTTP_SERVICE_UNAVAILABLE  (text/html)   8080   54733
3488   2020-03-21 18:34:52.556879   155.247.164.213   192.168.32.135   60   HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE  (text/html)   8080   54842
3522   2020-03-21 18:34:52.809037   155.247.164.214   192.168.32.135   60   HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE  (text/html)   8080   54843
3538   2020-03-21 18:34:52.839153   155.247.164.213   192.168.32.135   60   HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE  (text/html)   8080   54844
3632   2020-03-21 18:34:55.006346   155.247.164.214   192.168.32.135   60   HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE  (text/html)   8080   54845


When I asked in a separate thread for the server to send the clients more details about the problem, I got a reply that it's not a good idea, because some people would attempt to cheat if the server would give more error details to the clients.

While that kind of makes sense, although I'm not a big fan of security through obscurity, the problem is the client is not even displaying all the information the servers actually send to it. That doesn't make sense, since any potential cheater could easily find that information using Wireshark.

If the information returned by the server can't be made more verbose for security reasons, at the very least the client should be modified to show all the information the server actually sends it in reply, there is no good reason to hide it in my opinion.

I mean it's almost a week of wasting people's time troubleshooting this because somebody didn't add a 2-3 lines of code needed to log all the information sent by the server. That's not cool.

If anyone needs the full Wireshark capture of the communication to troubleshoot this further I can send it to him / her, unfortunately it seems this forum doesn't seem to support attaching files to a comment, but I can send a link to the capture file to anyone that needs it.
vnicolici
 
Posts: 15
Joined: Sun Mar 15, 2020 1:10 am

Re: Send Errors - 155.247.164.213 & .214

Postby ap1978 » Sat Mar 21, 2020 8:17 pm

I've got 3 completed WUs waiting.
ap1978
 
Posts: 6
Joined: Wed Nov 19, 2008 5:53 pm

Re: Send Errors - 155.247.164.213 & .214

Postby misha-mzs » Sat Mar 21, 2020 8:22 pm

I've got 5 already :e)
misha-mzs
 
Posts: 2
Joined: Sun Mar 15, 2020 7:18 pm

Re: Send Errors - 155.247.164.213 & .214

Postby vangli » Sat Mar 21, 2020 9:50 pm

According to the server status page, the software has just been upgraded from version 9.5.2 to 9.5.6. for both 213 and 214. However same sad result for mine part. Will start a new wireshark session to see if it any changes in communication.

Edit: Same result looking in with wireshark, however some data seems to be transfered before disconnecting with same message, 413 HTTP_REQUEST_ENTITY_TOO_LARGE
Regards
Bent Vangli, Oslo, Norway
vangli
 
Posts: 12
Joined: Thu Mar 19, 2020 11:35 am

Re: Send Errors - 155.247.164.213 & .214

Postby tsunami2311 » Sat Mar 21, 2020 11:41 pm

I now have 3 to send that wont send, The server should stop Assigning work if CS cant accept it for these job, it just waste if they wind up timing out for both sides
tsunami2311
 
Posts: 5
Joined: Sat Mar 21, 2020 6:08 pm

Re: Send Errors - 155.247.164.213 & .214

Postby vnicolici » Sun Mar 22, 2020 12:15 am

Now we know that, unless the servers lie "to prevent cheating", the problem is most likely the result size, since they return HTTP_REQUEST_ENTITY_TOO_LARGE when attempting to upload the result.

I looked through my logs from this week to try to get more insight about the problem. This is what I found so far:

- out of the 58 units I processed so far 56 results have uploaded successfully;
- the 2 WU results that fail uploading are both 55.24 MiB in size and are both from project 11758 and the attempts are to servers 155.247.164.213 & .214;
- I didn't process any other WUs from project 11758;
- the largest WU results accepted by either .213 or .214 has been a unit of 21.92 MiB from project 11753, which was accepted after 3 failed attempts.
- I found 2 other WU results that are also both 55.24 MiB in size, from a different project, 11764, but those were successfully uploaded to 128.252.203.10;
- the remaining WU results from the logs have been smaller, the largest ones being 5 WU results of 49.92 MiB accepted by 128.252.203.10 and 128.252.203.2

So this evidence suggests the issue is related to the servers .213 / .214 or project 11758 not being configured properly to accept large WU results, and that the configured result limit is somewhere between 21.92 MiB and 55.24MiB.

I hope this helps the people in charge of the server/project fix the issue.
vnicolici
 
Posts: 15
Joined: Sun Mar 15, 2020 1:10 am

Re: Send Errors - 155.247.164.213 & .214

Postby Empie » Sun Mar 22, 2020 12:55 am

Getting "Failed to send results to work server: Transfer failed" on both of these too.
Empie
 
Posts: 17
Joined: Sat Dec 27, 2008 4:56 pm
Location: Dutch Power Cows

Re: Send Errors - 155.247.164.213 & .214

Postby L.shawcross » Sun Mar 22, 2020 1:10 am

Still having problems sending results to 155.247.164.213
And 155.247.164.214

I have two WU 11758 ready to send but fails

If I ping either server I receive a response within 40ms or so

Im not interested in points I just want the data to get through and do the job I have my PC running flat out for.

Any update on this? anything I can do from my end?
L.shawcross
 
Posts: 1
Joined: Sat Mar 21, 2020 8:20 pm

Re: Send Errors - 155.247.164.213 & .214

Postby _r2w_ben » Sun Mar 22, 2020 2:29 am

vnicolici wrote:- I found 2 other WU results that are also both 55.24 MiB in size, from a different project, 11764, but those were successfully uploaded to 128.252.203.10;
- the remaining WU results from the logs have been smaller, the largest ones being 5 WU results of 49.92 MiB accepted by 128.252.203.10 and 128.252.203.2

I check my logs and have uploaded large results to other servers. Whatever request size related setting is different on those needs to be applied to .213/.214.
Code: Select all
112.77MiB to 128.252.203.10
 86.85MiB to 128.252.203.4
 86.96MiB to 140.163.4.241
_r2w_ben
 
Posts: 277
Joined: Wed Apr 23, 2008 4:11 pm

Re: Send Errors - 155.247.164.213 & .214

Postby vangli » Sun Mar 22, 2020 9:38 am

After a week, without any substantial information or solution from the administrators of those two servers, either directly or through moderators of this forum, the only conclusion is to close down those two servers until the problem is fixed or they are able to stop sending out new WUs which cannot be collected. I am almost there to stop supporting F&H and return to BOINC projects. This neglicting of supporters are not good.
vangli
 
Posts: 12
Joined: Thu Mar 19, 2020 11:35 am

Re: Send Errors - 155.247.164.213 & .214

Postby davidcoton » Sun Mar 22, 2020 11:16 am

@vangli I understand your frustration. The issues are known to the team, and are regularly passed up to them. We could post platitudes in reply, but I prefer to say nothing until there is some news coming back. Please understand that the team is very small, AIUI only one full-time tech person for IT, the rest is done by the researchers who also have to take care of the science side and the vast step-change in the size of the operation -- not to mention that COVID-19 projects are entirely new work.
Image
davidcoton
 
Posts: 1102
Joined: Wed Nov 05, 2008 4:19 pm
Location: Cambridge, UK

Re: Send Errors - 155.247.164.213 & .214

Postby vangli » Sun Mar 22, 2020 11:51 am

davidcoton wrote:@vangli I understand your frustration. The issues are known to the team, and are regularly passed up to them. We could post platitudes in reply, but I prefer to say nothing until there is some news coming back. Please understand that the team is very small, AIUI only one full-time tech person for IT, the rest is done by the researchers who also have to take care of the science side and the vast step-change in the size of the operation -- not to mention that COVID-19 projects are entirely new work.


This is such simple information I ask for. Nothing more. I quiet well understand the situation with this information. It tells me and many other that they are seriously working on it with small resources, and aware of the situation. Thanks :)
vangli
 
Posts: 12
Joined: Thu Mar 19, 2020 11:35 am

Re: Send Errors - 155.247.164.213 & .214

Postby vangli » Sun Mar 22, 2020 7:46 pm

Just out of pure curiosity, when I read what shown when i hoover over the CS tag in server status, the green Yes:

On server vav16.ocis.temple.edu (155.247.164.214):
Accepted: 155.247.166.219 155.247.166.220 128.252.203.2 128.252.203.9 40.114.52.201 13.90.152.57 37.187.12.48
Failed: 13.59.169.226 155.247.164.213 155.247.164.214 140.163.4.231 140.163.4.232 140.163.4.241 128.174.73.74 128.252.203.10 128.252.203.4 13.82.98.119

and on server vav15.ocis.temple.edu (155.247.164.213):
Accepted: 155.247.164.214
Failed:

The WU's in question has 155.247.164.213 as work server and 155.247.164.214 as collection server. No other possibilities.

Am I right in that a collection server accept WU's originated from those Accepted addresses only? If failed or not mentioned, they will not? If this is true, a WU from 155.247.164.213 which only mention a single collection server, namely 155.247.164.214, cannot be uploaded. The 213 work server block it self for upload, and 214 does not accept anything from itself. Am I right? :?
vangli
 
Posts: 12
Joined: Thu Mar 19, 2020 11:35 am

Re: Send Errors - 155.247.164.213 & .214

Postby davidcoton » Mon Mar 23, 2020 1:05 am

vangli wrote:Am I right? :?

Not entirely. Work servers accept their own work (usually) without using the CS mechanism. It is only when that fails that the client calls the CS that was set when the WU was downloaded.
Every WS should specify a CS (AFAIK it can only specify one), but this is not always done. The CS may be another WS or a separate server.
davidcoton
 
Posts: 1102
Joined: Wed Nov 05, 2008 4:19 pm
Location: Cambridge, UK

PreviousNext

Return to Issues with a specific server

Who is online

Users browsing this forum: Google [Bot] and 2 guests

cron