140.163.4.245 / 140.163.4.241 upload failed

[ChristianVirtual]

Work is done, but both WS and CS can't finish the transfers ...

Code: Select all

02:03:34:Completed 990000 out of 1000000 steps (99%)
02:06:37:Completed 1000000 out of 1000000 steps (100%)
02:07:00:Saving result file logfile_01.txt
02:07:00:Saving result file checkpointState.xml
02:07:05:Saving result file checkpt.crc
02:07:05:Saving result file log.txt
02:07:05:Saving result file positions.xtc
02:07:08:Folding@home Core Shutdown: FINISHED_UNIT
02:07:08:FahCore returned: FINISHED_UNIT (100 = 0x64)
02:07:08:Sending unit results: id:01 state:SEND error:NO_ERROR project:10494 run:10 clone:35 gen:2 core unit:0x000000038ca304f555de9472fb27da37
02:07:08:Uploading 35.88MiB to 140.163.4.245
02:07:08:Connecting to 140.163.4.245:8080
02:07:14:Upload 9.06%
02:07:20:Upload 18.29%
02:07:36:Upload 31.70%
02:10:04:Upload 33.27%
02:10:42:Upload 33.62%
02:10:42:WARNING:Exception: Failed to send results to work server: Transfer failed

02:10:42:Trying to send results to collection server
02:10:42:Uploading 35.88MiB to 140.163.4.241
02:10:42:Connecting to 140.163.4.241:8080
02:10:51:Upload 4.01%
02:10:57:Upload 13.41%
02:11:12:Upload 24.73%
02:14:15:Upload 25.78%
02:14:16:ERROR:Exception: Transfer failed

02:14:17:Sending unit results: id:01 state:SEND error:NO_ERROR project:10494 run:10 clone:35 gen:2 core unit:0x000000038ca304f555de9472fb27da37
02:14:17:Uploading 35.88MiB to 140.163.4.245
02:14:17:Connecting to 140.163.4.245:8080
02:14:24:Upload 5.75%
02:14:30:Upload 12.37%
02:17:15:Upload 26.65%
02:17:53:Upload 27.35%
02:17:53:WARNING:Exception: Failed to send results to work server: Transfer failed
02:17:53:Trying to send results to collection server
02:17:53:Uploading 35.88MiB to 140.163.4.241
02:17:53:Connecting to 140.163.4.241:8080
02:17:59:Upload 1.39%
02:20:43:Upload 16.20%
02:21:19:Upload 16.90%
02:21:19:ERROR:Exception: Transfer failed
02:21:20:Sending unit results: id:01 state:SEND error:NO_ERROR project:10494 run:10 clone:35 gen:2 core unit:0x000000038ca304f555de9472fb27da37
02:21:20:Uploading 35.88MiB to 140.163.4.245
02:21:20:Connecting to 140.163.4.245:8080
02:21:26:Upload 1.57%
02:24:23:Upload 17.59%
02:24:52:Upload 18.29%
02:24:52:WARNING:Exception: Failed to send results to work server: Transfer failed
02:24:52:Trying to send results to collection server
02:24:52:Uploading 35.88MiB to 140.163.4.241
02:24:52:Connecting to 140.163.4.241:8080
02:24:58:Upload 5.57%
02:25:09:Upload 6.62%
02:25:15:Upload 26.82%
02:26:52:Upload 30.83%
02:28:27:Upload 32.22%
02:28:27:ERROR:Exception: Transfer failed
02:28:27:Sending unit results: id:01 state:SEND error:NO_ERROR project:10494 run:10 clone:35 gen:2 core unit:0x000000038ca304f555de9472fb27da37
02:28:27:Uploading 35.88MiB to 140.163.4.245
02:28:27:Connecting to 140.163.4.245:8080
02:28:34:Upload 4.70%
02:28:40:Upload 10.28%
02:31:18:Upload 25.78%
02:31:50:Upload 26.30%
02:31:50:WARNING:Exception: Failed to send results to work server: Transfer failed
02:31:50:Trying to send results to collection server
02:31:50:Uploading 35.88MiB to 140.163.4.241
02:31:50:Connecting to 140.163.4.241:8080
02:31:56:Upload 1.92%
02:32:02:Upload 19.33%
02:35:08:Upload 20.38%
02:35:08:ERROR:Exception: Transfer failed
02:35:08:Sending unit results: id:01 state:SEND error:NO_ERROR project:10494 run:10 clone:35 gen:2 core unit:0x000000038ca304f555de9472fb27da37
02:35:08:Uploading 35.88MiB to 140.163.4.245
02:35:08:Connecting to 140.163.4.245:8080
02:38:02:Upload 6.44%
02:38:18:Upload 6.62%
02:38:18:WARNING:Exception: Failed to send results to work server: Transfer failed
02:38:18:Trying to send results to collection server
02:38:18:Uploading 35.88MiB to 140.163.4.241
02:38:18:Connecting to 140.163.4.241:8080
02:38:24:Upload 1.57%
02:38:30:Upload 19.68%
02:38:36:Upload 21.95%
02:38:43:Upload 27.52%
02:38:49:Upload 36.93%
02:42:10:Upload 38.15%
02:42:10:ERROR:Exception: Transfer failed
02:42:10:Sending unit results: id:01 state:SEND error:NO_ERROR project:10494 run:10 clone:35 gen:2 core unit:0x000000038ca304f555de9472fb27da37
02:42:10:Uploading 35.88MiB to 140.163.4.245
02:42:10:Connecting to 140.163.4.245:8080
02:42:16:Upload 5.57%

[ChristianVirtual]

Just after posting it here the servers accepted the transfer ... But still 40min delay

[Kebast]

I've been having a lot of upload problems the last couple of days too. I restarted my machines tonight and things seem to be running smoothly again.

[ChristianVirtual]

actually I just restarted my boxes too, after some dust removal and change of hardware ... incl. router and firewall. and other WS/CS are ok ...

[ChristianVirtual]

I continue having this issue with the server in segment number range 140.xx.xx.xx

Other servers from FAH don't have that issue; also in general other connections from my home network are stable.

Code: Select all

06:49:26:WU01:FS01:0x21:Completed 990000 out of 1000000 steps (99%)
06:52:29:WU01:FS01:0x21:Completed 1000000 out of 1000000 steps (100%)
06:52:36:WU01:FS01:0x21:Saving result file logfile_01.txt
06:52:36:WU01:FS01:0x21:Saving result file checkpointState.xml
06:52:41:WU01:FS01:0x21:Saving result file checkpt.crc
06:52:41:WU01:FS01:0x21:Saving result file log.txt
06:52:41:WU01:FS01:0x21:Saving result file positions.xtc
06:52:44:WU01:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
06:52:44:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
06:52:44:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10494 run:0 clone:11 gen:25 core:0x21 unit:0x000000218ca304f555de9284d02d4508
06:52:44:WU01:FS01:Uploading 35.89MiB to 140.163.4.245
06:52:44:WU01:FS01:Connecting to 140.163.4.245:8080
06:55:20:WU01:FS01:Upload 10.97%
06:55:52:WU01:FS01:Upload 11.32%
06:55:52:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
06:55:52:WU01:FS01:Trying to send results to collection server
06:55:52:WU01:FS01:Uploading 35.89MiB to 140.163.4.241
06:55:52:WU01:FS01:Connecting to 140.163.4.241:8080
06:58:34:WU01:FS01:Upload 6.62%
06:59:04:WU01:FS01:Upload 7.49%
06:59:04:ERROR:WU01:FS01:Exception: Transfer failed
06:59:04:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10494 run:0 clone:11 gen:25 core:0x21 unit:0x000000218ca304f555de9284d02d4508
06:59:04:WU01:FS01:Uploading 35.89MiB to 140.163.4.245
06:59:04:WU01:FS01:Connecting to 140.163.4.245:8080
06:59:14:WU01:FS01:Upload 5.05%
06:59:20:WU01:FS01:Upload 9.75%
07:02:00:WU01:FS01:Upload 17.42%
07:02:35:WU01:FS01:Upload 17.59%
07:02:35:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
07:02:35:WU01:FS01:Trying to send results to collection server
07:02:35:WU01:FS01:Uploading 35.89MiB to 140.163.4.241
07:02:35:WU01:FS01:Connecting to 140.163.4.241:8080
07:05:20:WU01:FS01:Upload 6.10%
07:06:00:WU01:FS01:Upload 6.97%
07:06:00:ERROR:WU01:FS01:Exception: Transfer failed
07:06:00:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10494 run:0 clone:11 gen:25 core:0x21 unit:0x000000218ca304f555de9284d02d4508
07:06:00:WU01:FS01:Uploading 35.89MiB to 140.163.4.245
07:06:00:WU01:FS01:Connecting to 140.163.4.245:8080
07:06:07:WU01:FS01:Upload 5.22%
07:06:13:WU01:FS01:Upload 21.42%
07:08:53:WU01:FS01:Upload 28.21%
07:09:28:WU01:FS01:Upload 28.56%
07:09:28:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
07:09:28:WU01:FS01:Trying to send results to collection server
07:09:28:WU01:FS01:Uploading 35.89MiB to 140.163.4.241
07:09:28:WU01:FS01:Connecting to 140.163.4.241:8080
07:09:34:WU01:FS01:Upload 1.57%
07:09:44:WU01:FS01:Upload 18.81%

it seems taking very much time to get the initial connection into a stable status
06:52:44:WU01:FS01:Connecting to 140.163.4.245:8080
06:55:20:WU01:FS01:Upload 10.97%

My connection is a 200Mb/s / 100Mb/s fiber-optical line; normally not a bottleneck.I know, its holiday season and not much will/can happen. But just to document the event.

Update: finally worked (but bite away 20 minutes from QRB )

Code: Select all

07:25:09:WU01:FS01:Upload complete
07:25:09:WU01:FS01:Server responded WORK_ACK (400)
07:25:09:WU01:FS01:Final credit estimate, 130215.00 points
07:25:09:WU01:FS01:Cleaning up

at least not dumped as in other cases I had ...

[Ricky]

I had one fail to upload on this server, but another server quickly took the WU.

Code: Select all

11:59:32:WU00:FS01:0x21:Completed 5000000 out of 5000000 steps (100%)
11:59:32:WU01:FS01:Connecting to 171.67.108.45:80
11:59:32:WU01:FS01:Assigned to work server 140.163.4.243
11:59:32:WU01:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GM200 [GeForce GTX 980 Ti] from 140.163.4.243
11:59:32:WU01:FS01:Connecting to 140.163.4.243:8080
11:59:33:WU01:FS01:Downloading 5.11MiB
11:59:35:WU00:FS01:0x21:Saving result file logfile_01.txt
11:59:35:WU00:FS01:0x21:Saving result file checkpointState.xml
11:59:36:WU00:FS01:0x21:Saving result file checkpt.crc
11:59:36:WU00:FS01:0x21:Saving result file log.txt
11:59:36:WU00:FS01:0x21:Saving result file positions.xtc
11:59:37:WU01:FS01:Download complete
11:59:37:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:11703 run:0 clone:61 gen:8 core:0x21 unit:0x000000088ca304f3568961ba92c70727
11:59:37:WU00:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
11:59:38:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
11:59:38:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11704 run:0 clone:53 gen:15 core:0x21 unit:0x0000000f8ca304f35689623553f619ef
11:59:38:WU00:FS01:Uploading 6.98MiB to 140.163.4.243
11:59:38:WU01:FS01:Starting
11:59:38:WU00:FS01:Connecting to 140.163.4.243:8080
11:59:38:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win2/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/beta/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 704 -lifeline 1564 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
11:59:38:WU01:FS01:Started FahCore on PID 1392
11:59:38:WU01:FS01:Core PID:4804
11:59:38:WU01:FS01:FahCore 0x21 started
11:59:38:WU01:FS01:0x21:*********************** Log Started 2016-01-23T11:59:38Z ***********************
11:59:38:WU01:FS01:0x21:Project: 11703 (Run 0, Clone 61, Gen 8)
11:59:38:WU01:FS01:0x21:Unit: 0x000000088ca304f3568961ba92c70727
11:59:38:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:59:38:WU01:FS01:0x21:Machine: 1
11:59:38:WU01:FS01:0x21:Reading tar file core.xml
11:59:38:WU01:FS01:0x21:Reading tar file system.xml
11:59:39:WU01:FS01:0x21:Reading tar file integrator.xml
11:59:39:WU01:FS01:0x21:Reading tar file state.xml
11:59:40:WU01:FS01:0x21:Digital signatures verified
11:59:40:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:59:40:WU01:FS01:0x21:Version 0.0.17
11:59:44:WU00:FS01:Upload 12.53%
11:59:50:WU00:FS01:Upload 29.54%
11:59:51:WU01:FS01:0x21:Completed 0 out of 5000000 steps (0%)
11:59:51:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
11:59:56:WU00:FS01:Upload 43.86%
12:00:02:WU00:FS01:Upload 54.61%
12:00:25:WU00:FS01:Upload 60.87%
12:00:25:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
12:00:25:WU00:FS01:Trying to send results to collection server
12:00:25:WU00:FS01:Uploading 6.98MiB to 128.252.203.2
12:00:25:WU00:FS01:Connecting to 128.252.203.2:8080
12:00:31:WU00:FS01:Upload complete
12:00:31:WU00:FS01:Server responded WORK_ACK (400)
12:00:31:WU00:FS01:Final credit estimate, 108221.00 points
12:00:31:WU00:FS01:Cleaning up
12:03:20:WU01:FS01:0x21:Completed 50000 out of 5000000 steps (1%)

[bruce]

Your upload to 140.163.4.243 failed (so I conclude it's probably a problem with 140.163.4.*)

Nevertheless, the Collection Server, 128.252.203.2. dutifully accepted the upload as it should have.

In notice that at a that moment there was a tremendous difference it upload speed. Does traceroute from your location give any indications? From here, I get as far as 67.151.24.149 to 74.8.57.6 and then everything times out. (windstream.net)