Upload stuck at 100% | 140.163.4.241

Moderators: Site Moderators, FAHC Science Team

foldinghomealone2
Posts: 148
Joined: Sun Jul 30, 2017 8:40 pm

Re: Upload stuck at 100% | 140.163.4.241

Post by foldinghomealone2 »

Vodafone is is one of the major internet providers. In my experience it offers reliable services.
I made tests when the upload hung and it offered max download and upload bandwith.
Joe_H
Site Admin
Posts: 7857
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Upload stuck at 100% | 140.163.4.241

Post by Joe_H »

From where you are to where the test site for use speeds is important. But your connection is going from Europe to the US, and once in the US there are routers that get congested, and both of the servers you were attempting to connect to are in large urban areas. Add heavy loads on the servers themselves, and your connection to them will be slow at times, and possibly interrupted if packet ACK's are dropped often enough.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Upload stuck at 100% | 140.163.4.241

Post by bruce »

The real bug is that whatever condition is causing the slow-down is not being detected by either the the client on your system or the server on the other end ... including the "hang" No error is recognized and no message is issued so no error recovery is undertaken. If the error had been recognized, something could have been done to correct it. Moreover, I expect that there's no record in either your client or in the server that was contacted that anything untoward happened.
foldinghomealone2
Posts: 148
Joined: Sun Jul 30, 2017 8:40 pm

Re: Upload stuck at 100% | 140.163.4.241

Post by foldinghomealone2 »

Bruce, I totally agree.

I think it would be easy to implement a routine to detect hanging uploads in the client.
Like: In the logs you can see the time stamps for the upload progress (let's call it 'upload update'). It's easy to calculate the average upload speed and to estimate the necessary time for the next 'upload update'. If this time is exceeded (let's say) 5 times then the upload should be redirected to a new server.

I know that resources for client developement are limited, though.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Upload stuck at 100% | 140.163.4.241

Post by bruce »

Those updates seem to happen on a regular schedule most of the time although the amount of data transferred changes. I've been thinking of dividing data by time to calculate a speed but I don't think that's the best way to do it because my ISP connection is different than yours. You're suggesting that the client should

From your first log, no error is detected in ether stream because each update is ~6 seconds
21:43:47:WU01:FS01:Upload 1.08%
21:43:53:WU01:FS01:Upload 1.80%
21:43:59:WU01:FS01:Upload 3.74%
21:44:05:WU01:FS01:Upload 5.75%
21:44:11:WU01:FS01:Upload 7.47%
21:44:17:WU01:FS01:Upload 9.41%
21:44:23:WU01:FS01:Upload 10.49%
------
21:43:47:WU02:FS01:Upload 6.81%
21:43:53:WU02:FS01:Upload 11.92%
21:43:59:WU02:FS01:Upload 23.84%
21:44:05:WU02:FS01:Upload 45.41%
21:44:11:WU02:FS01:Upload 64.71%
21:44:17:WU02:FS01:Upload 89.12%
21:44:23:WU02:FS01:Upload 97.63%

but second log shows an error because it simply stops.
22:33:39:WU01:FS01:Upload 93.99%
22:33:45:WU01:FS01:Upload 94.93%
22:33:51:WU01:FS01:Upload 95.86%
22:33:57:WU01:FS01:Upload 96.58%
22:34:03:WU01:FS01:Upload 97.59%
22:34:09:WU01:FS01:Upload 98.23%
22:34:15:WU01:FS01:Upload 99.24%
(but missing any more updates at or after 22:34:21)

After a suitable timeout of TBD, the client should abort the upload and redirect it to a different server. Right?
And the percent changes (or an equivalent in KiB) should be ignored. Right?

And after a reboot, you show it completing as
22:48:30:WU01:FS01:Upload 97.37%
22:48:36:WU01:FS01:Upload 98.09%
22:48:42:WU01:FS01:Upload 98.81%
22:48:48:WU01:FS01:Upload 99.74%
22:48:51:WU01:FS01:Upload complete
22:48:52:WU01:FS01:Server responded WORK_ACK (400)
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Upload stuck at 100% | 140.163.4.241

Post by bruce »

In another of your examples, you show
21:50:05:WU01:FS01:Upload 94.35%
21:50:11:WU01:FS01:Upload 95.14%
21:50:17:WU01:FS01:Upload 96.80%
22:22:37:ERROR:Receive error: 10054: Eine vorhandene Verbindung wurde vom Remotehost geschlossen.

In this case, what happened after the error was reported? Did the client restart the upload without your intervention or was this another case that needs the same change?

I think the key here is that the client detected the error and in the previous example, it did not detect the error.
foldinghomealone2
Posts: 148
Joined: Sun Jul 30, 2017 8:40 pm

Re: Upload stuck at 100% | 140.163.4.241

Post by foldinghomealone2 »

After 21:50:17 there was no progress anymore and I restarted the PC at 22:22:37 manually.
I guess this caused the log entry.

I agree, after a certain timeout " the client should abort the upload and redirect it to a different server".
rickoic
Posts: 322
Joined: Sat May 23, 2009 4:49 pm
Hardware configuration: eVga x299 DARK 2070 Super, eVGA 2080, eVga 1070, eVga 2080 Super
MSI x399 eVga 2080, eVga 1070, eVga 1070, GT970
Location: Mississippi near Memphis, Tn

Re: Upload stuck at 100% | 140.163.4.241

Post by rickoic »

Had a slow upload to 140.163.4.241

Code: Select all

15:48:08:WU01:FS00:Uploading 19.45MiB to 140.163.4.241
15:48:08:WU01:FS00:Connecting to 140.163.4.241:8080
15:48:08:WU02:FS00:Starting
15:48:08:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\ricko\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/beta/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 705 -lifeline 8840 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:48:08:WU02:FS00:Started FahCore on PID 46824
15:48:09:WU02:FS00:Core PID:52208
15:48:09:WU02:FS00:FahCore 0x22 started
15:48:09:WU02:FS00:0x22:*********************** Log Started 2020-02-29T15:48:09Z ***********************
15:48:09:WU02:FS00:0x22:*************************** Core22 Folding@home Core ***************************
15:48:09:WU02:FS00:0x22:       Type: 0x22
15:48:09:WU02:FS00:0x22:       Core: Core22
15:48:09:WU02:FS00:0x22:    Website: https://foldingathome.org/
15:48:09:WU02:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
15:48:09:WU02:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
15:48:09:WU02:FS00:0x22:             <rafal.wiewiora@choderalab.org>
15:48:09:WU02:FS00:0x22:       Args: -dir 02 -suffix 01 -version 705 -lifeline 46824 -checkpoint 15
15:48:09:WU02:FS00:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
15:48:09:WU02:FS00:0x22:             0 -gpu 0
15:48:09:WU02:FS00:0x22:     Config: <none>
15:48:09:WU02:FS00:0x22:************************************ Build *************************************
15:48:09:WU02:FS00:0x22:    Version: 0.0.2
15:48:09:WU02:FS00:0x22:       Date: Dec 6 2019
15:48:09:WU02:FS00:0x22:       Time: 21:30:31
15:48:09:WU02:FS00:0x22: Repository: Git
15:48:09:WU02:FS00:0x22:   Revision: abeb39247cc72df5af0f63723edafadb23d5dfbe
15:48:09:WU02:FS00:0x22:     Branch: HEAD
15:48:09:WU02:FS00:0x22:   Compiler: Visual C++ 2008
15:48:09:WU02:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
15:48:09:WU02:FS00:0x22:   Platform: win32 10
15:48:09:WU02:FS00:0x22:       Bits: 64
15:48:09:WU02:FS00:0x22:       Mode: Release
15:48:09:WU02:FS00:0x22:************************************ System ************************************
15:48:09:WU02:FS00:0x22:        CPU: AMD A8-9600 RADEON R7, 10 COMPUTE CORES 4C+6G
15:48:09:WU02:FS00:0x22:     CPU ID: AuthenticAMD Family 21 Model 101 Stepping 1
15:48:09:WU02:FS00:0x22:       CPUs: 4
15:48:09:WU02:FS00:0x22:     Memory: 15.95GiB
15:48:09:WU02:FS00:0x22:Free Memory: 10.43GiB
15:48:09:WU02:FS00:0x22:    Threads: WINDOWS_THREADS
15:48:09:WU02:FS00:0x22: OS Version: 6.2
15:48:09:WU02:FS00:0x22:Has Battery: false
15:48:09:WU02:FS00:0x22: On Battery: false
15:48:09:WU02:FS00:0x22: UTC Offset: -6
15:48:09:WU02:FS00:0x22:        PID: 52208
15:48:09:WU02:FS00:0x22:        CWD: C:\Users\ricko\AppData\Roaming\FAHClient\work
15:48:09:WU02:FS00:0x22:         OS: Windows 10 Home
15:48:09:WU02:FS00:0x22:    OS Arch: AMD64
15:48:09:WU02:FS00:0x22:********************************************************************************
15:48:09:WU02:FS00:0x22:Project: 11738 (Run 0, Clone 463, Gen 55)
15:48:09:WU02:FS00:0x22:Unit: 0x000000438ca304f15e0fb949ff1974c0
15:48:09:WU02:FS00:0x22:Reading tar file core.xml
15:48:09:WU02:FS00:0x22:Reading tar file integrator.xml
15:48:09:WU02:FS00:0x22:Reading tar file state.xml
15:48:10:WU02:FS00:0x22:Reading tar file system.xml
15:48:11:WU02:FS00:0x22:Digital signatures verified
15:48:11:WU02:FS00:0x22:Folding@home GPU Core22 Folding@home Core
15:48:11:WU02:FS00:0x22:Version 0.0.2
15:48:17:WU00:FS01:0x22:Completed 680000 out of 2000000 steps (34%)
15:48:18:WU01:FS00:Upload 17.36%
15:48:24:WU01:FS00:Upload 31.50%
15:48:30:WU01:FS00:Upload 45.96%
15:48:36:WU01:FS00:Upload 61.07%
15:48:58:WU02:FS00:0x22:Completed 0 out of 1000000 steps (0%)
15:48:58:WU02:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
15:48:58:WU01:FS00:Upload 70.07%
15:50:28:WU01:FS00:Upload 70.39%
15:50:32:WU00:FS01:0x22:Completed 700000 out of 2000000 steps (35%)
15:50:51:WU02:FS00:0x22:Completed 10000 out of 1000000 steps (1%)
15:51:42:WU01:FS00:Upload 70.71%
15:52:46:WU02:FS00:0x22:Completed 20000 out of 1000000 steps (2%)
15:52:57:WU00:FS01:0x22:Completed 720000 out of 2000000 steps (36%)
15:54:32:WU02:FS00:0x22:Completed 30000 out of 1000000 steps (3%)
15:55:06:WU00:FS01:0x22:Completed 740000 out of 2000000 steps (37%)
15:56:15:WU02:FS00:0x22:Completed 40000 out of 1000000 steps (4%)
15:57:11:WU00:FS01:0x22:Completed 760000 out of 2000000 steps (38%)
15:57:57:WU02:FS00:0x22:Completed 50000 out of 1000000 steps (5%)
15:59:18:WU00:FS01:0x22:Completed 780000 out of 2000000 steps (39%)
15:59:56:WU02:FS00:0x22:Completed 60000 out of 1000000 steps (6%)
16:01:25:WU00:FS01:0x22:Completed 800000 out of 2000000 steps (40%)
16:01:35:WU02:FS00:0x22:Completed 70000 out of 1000000 steps (7%)
16:03:13:WU02:FS00:0x22:Completed 80000 out of 1000000 steps (8%)
16:03:31:WU00:FS01:0x22:Completed 820000 out of 2000000 steps (41%)
16:04:51:WU02:FS00:0x22:Completed 90000 out of 1000000 steps (9%)
16:05:34:WU00:FS01:0x22:Completed 840000 out of 2000000 steps (42%)
16:06:00:WU01:FS00:Upload 72.32%
16:06:29:WU02:FS00:0x22:Completed 100000 out of 1000000 steps (10%)
16:07:39:WU00:FS01:0x22:Completed 860000 out of 2000000 steps (43%)
16:08:23:WU02:FS00:0x22:Completed 110000 out of 1000000 steps (11%)
16:09:42:WU00:FS01:0x22:Completed 880000 out of 2000000 steps (44%)
16:10:02:WU02:FS00:0x22:Completed 120000 out of 1000000 steps (12%)
16:11:41:WU02:FS00:0x22:Completed 130000 out of 1000000 steps (13%)
16:11:45:WU00:FS01:0x22:Completed 900000 out of 2000000 steps (45%)
16:12:26:FS01:Paused
16:12:26:FS00:Paused
16:12:26:FS01:Shutting core down
16:12:26:FS00:Shutting core down
16:12:27:WU00:FS01:0x22:WARNING:Console control signal 1 on PID 46208
16:12:27:WU00:FS01:0x22:Exiting, please wait. . .
16:12:27:WU02:FS00:0x22:WARNING:Console control signal 1 on PID 52208
16:12:27:WU02:FS00:0x22:Exiting, please wait. . .
16:12:27:WU00:FS01:0x22:Folding@home Core Shutdown: INTERRUPTED
16:12:27:WU02:FS00:0x22:Folding@home Core Shutdown: INTERRUPTED
16:12:27:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
16:12:28:WU02:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
16:12:37:FS01:Unpaused
16:12:37:FS00:Unpaused
16:12:37:WU00:FS01:Starting
16:12:37:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\ricko\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/beta/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 705 -lifeline 8840 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
16:12:37:WU00:FS01:Started FahCore on PID 37944
16:12:37:WU00:FS01:Core PID:50700
16:12:37:WU00:FS01:FahCore 0x22 started
16:12:38:WU00:FS01:0x22:*********************** Log Started 2020-02-29T16:12:38Z ***********************
16:12:38:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
16:12:38:WU00:FS01:0x22:       Type: 0x22
16:12:38:WU00:FS01:0x22:       Core: Core22
16:12:38:WU00:FS01:0x22:    Website: https://foldingathome.org/
16:12:38:WU00:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
16:12:38:WU00:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
16:12:38:WU00:FS01:0x22:             <rafal.wiewiora@choderalab.org>
16:12:38:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 705 -lifeline 37944 -checkpoint 15
16:12:38:WU00:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
16:12:38:WU00:FS01:0x22:             1 -gpu 1
16:12:38:WU00:FS01:0x22:     Config: <none>
16:12:38:WU00:FS01:0x22:************************************ Build *************************************
16:12:38:WU00:FS01:0x22:    Version: 0.0.2
16:12:38:WU00:FS01:0x22:       Date: Dec 6 2019
16:12:38:WU00:FS01:0x22:       Time: 21:30:31
16:12:38:WU00:FS01:0x22: Repository: Git
16:12:38:WU00:FS01:0x22:   Revision: abeb39247cc72df5af0f63723edafadb23d5dfbe
16:12:38:WU00:FS01:0x22:     Branch: HEAD
16:12:38:WU00:FS01:0x22:   Compiler: Visual C++ 2008
16:12:38:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
16:12:38:WU00:FS01:0x22:   Platform: win32 10
16:12:38:WU00:FS01:0x22:       Bits: 64
16:12:38:WU00:FS01:0x22:       Mode: Release
16:12:38:WU00:FS01:0x22:************************************ System ************************************
16:12:38:WU00:FS01:0x22:        CPU: AMD A8-9600 RADEON R7, 10 COMPUTE CORES 4C+6G
16:12:38:WU00:FS01:0x22:     CPU ID: AuthenticAMD Family 21 Model 101 Stepping 1
16:12:38:WU00:FS01:0x22:       CPUs: 4
16:12:38:WU00:FS01:0x22:     Memory: 15.95GiB
16:12:38:WU00:FS01:0x22:Free Memory: 12.00GiB
16:12:38:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
16:12:38:WU00:FS01:0x22: OS Version: 6.2
16:12:38:WU00:FS01:0x22:Has Battery: false
16:12:38:WU00:FS01:0x22: On Battery: false
16:12:38:WU00:FS01:0x22: UTC Offset: -6
16:12:38:WU00:FS01:0x22:        PID: 50700
16:12:38:WU00:FS01:0x22:        CWD: C:\Users\ricko\AppData\Roaming\FAHClient\work
16:12:38:WU00:FS01:0x22:         OS: Windows 10 Home
16:12:38:WU00:FS01:0x22:    OS Arch: AMD64
16:12:38:WU00:FS01:0x22:********************************************************************************
16:12:38:WU00:FS01:0x22:Project: 11739 (Run 18, Clone 21, Gen 150)
16:12:38:WU00:FS01:0x22:Unit: 0x000000a88ca304f15e14e7fec12c1415
16:12:38:WU00:FS01:0x22:Digital signatures verified
16:12:38:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
16:12:38:WU00:FS01:0x22:Version 0.0.2
16:12:38:WU00:FS01:0x22:  Found a checkpoint file
16:12:38:WU02:FS00:Starting
16:12:38:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\ricko\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/beta/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 705 -lifeline 8840 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
16:12:39:WU02:FS00:Started FahCore on PID 50548
16:12:39:WU02:FS00:Core PID:55036
16:12:39:WU02:FS00:FahCore 0x22 started
16:12:39:WU02:FS00:0x22:*********************** Log Started 2020-02-29T16:12:39Z ***********************
16:12:39:WU02:FS00:0x22:*************************** Core22 Folding@home Core ***************************
16:12:39:WU02:FS00:0x22:       Type: 0x22
16:12:39:WU02:FS00:0x22:       Core: Core22
16:12:39:WU02:FS00:0x22:    Website: https://foldingathome.org/
16:12:39:WU02:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
16:12:39:WU02:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
16:12:39:WU02:FS00:0x22:             <rafal.wiewiora@choderalab.org>
16:12:39:WU02:FS00:0x22:       Args: -dir 02 -suffix 01 -version 705 -lifeline 50548 -checkpoint 15
16:12:39:WU02:FS00:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
16:12:39:WU02:FS00:0x22:             0 -gpu 0
16:12:39:WU02:FS00:0x22:     Config: <none>
16:12:39:WU02:FS00:0x22:************************************ Build *************************************
16:12:39:WU02:FS00:0x22:    Version: 0.0.2
16:12:39:WU02:FS00:0x22:       Date: Dec 6 2019
16:12:39:WU02:FS00:0x22:       Time: 21:30:31
16:12:39:WU02:FS00:0x22: Repository: Git
16:12:39:WU02:FS00:0x22:   Revision: abeb39247cc72df5af0f63723edafadb23d5dfbe
16:12:39:WU02:FS00:0x22:     Branch: HEAD
16:12:39:WU02:FS00:0x22:   Compiler: Visual C++ 2008
16:12:39:WU02:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
16:12:39:WU02:FS00:0x22:   Platform: win32 10
16:12:39:WU02:FS00:0x22:       Bits: 64
16:12:39:WU02:FS00:0x22:       Mode: Release
16:12:39:WU02:FS00:0x22:************************************ System ************************************
16:12:39:WU02:FS00:0x22:        CPU: AMD A8-9600 RADEON R7, 10 COMPUTE CORES 4C+6G
16:12:39:WU02:FS00:0x22:     CPU ID: AuthenticAMD Family 21 Model 101 Stepping 1
16:12:39:WU02:FS00:0x22:       CPUs: 4
16:12:39:WU02:FS00:0x22:     Memory: 15.95GiB
16:12:39:WU02:FS00:0x22:Free Memory: 11.93GiB
16:12:39:WU02:FS00:0x22:    Threads: WINDOWS_THREADS
16:12:39:WU02:FS00:0x22: OS Version: 6.2
16:12:39:WU02:FS00:0x22:Has Battery: false
16:12:39:WU02:FS00:0x22: On Battery: false
16:12:39:WU02:FS00:0x22: UTC Offset: -6
16:12:39:WU02:FS00:0x22:        PID: 55036
16:12:39:WU02:FS00:0x22:        CWD: C:\Users\ricko\AppData\Roaming\FAHClient\work
16:12:39:WU02:FS00:0x22:         OS: Windows 10 Home
16:12:39:WU02:FS00:0x22:    OS Arch: AMD64
16:12:39:WU02:FS00:0x22:********************************************************************************
16:12:39:WU02:FS00:0x22:Project: 11738 (Run 0, Clone 463, Gen 55)
16:12:39:WU02:FS00:0x22:Unit: 0x000000438ca304f15e0fb949ff1974c0
16:12:39:WU02:FS00:0x22:Digital signatures verified
16:12:39:WU02:FS00:0x22:Folding@home GPU Core22 Folding@home Core
16:12:39:WU02:FS00:0x22:Version 0.0.2
16:12:39:WU02:FS00:0x22:  Found a checkpoint file
16:12:49:WU00:FS01:0x22:Completed 900000 out of 2000000 steps (45%)
16:12:49:WU00:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
16:13:24:WU02:FS00:0x22:Completed 100000 out of 1000000 steps (10%)
16:13:24:WU02:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
16:15:51:WU01:FS00:Upload 72.64%
16:15:57:WU01:FS00:Upload 87.75%
16:16:03:WU01:FS00:Upload complete
16:16:03:WU01:FS00:Server responded WORK_ACK (400)
16:16:03:WU01:FS00:Final credit estimate, 89680.00 points
16:16:03:WU01:FS00:Cleaning up
Did a pause command and then a fold command and upload picked up like it was susposed to.
I'm folding because Dec 2005 I had radical prostate surgery.
Lost brother to spinal cancer, brother-in-law to prostate cancer.
Several 1st cousins lost and a few who have survived.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Upload stuck at 100% | 140.163.4.241

Post by bruce »

rickoic wrote:Had a slow upload to 140.163.4.241wu01

15:48:08:WU01:FS00:Uploading 19.45MiB to 140.163.4.241
15:48:08:WU01:FS00:Connecting to 140.163.4.241:8080
15:48:18:WU01:FS00:Upload 17.36%
15:48:24:WU01:FS00:Upload 31.50%
15:48:30:WU01:FS00:Upload 45.96%
15:48:36:WU01:FS00:Upload 61.07%
15:48:58:WU01:FS00:Upload 70.07%
15:50:28:WU01:FS00:Upload 70.39%
15:51:42:WU01:FS00:Upload 70.71%
16:06:00:WU01:FS00:Upload 72.32%
16:15:51:WU01:FS00:Upload 72.64%
16:15:57:WU01:FS00:Upload 87.75%
16:16:03:WU01:FS00:Upload complete
16:16:03:WU01:FS00:Server responded WORK_ACK (400)
16:16:03:WU01:FS00:Final credit estimate, 89680.00 points
16:16:03:WU01:FS00:Cleaning up

Did a pause command and then a fold command and upload picked up like it was susposed to.
Apparently the intermixing of messages from concurrent tasks has confused you. Looking only at the upload of WU01, it looks like it completed normally.

Uploads and downloads are processed independently from the active wUs that are being computed. Pausing and unpausing are unrelated to the WUs being calculated. Note that these are WU00 and WU02.

Code: Select all

16:12:26:FS01:Paused
16:12:26:FS00:Paused
16:12:26:FS01:Shutting core down
16:12:26:FS00:Shutting core down
16:12:27:WU00:FS01:0x22:WARNING:Console control signal 1 on PID 46208
16:12:27:WU00:FS01:0x22:Exiting, please wait. . .
16:12:27:WU02:FS00:0x22:WARNING:Console control signal 1 on PID 52208
16:12:27:WU02:FS00:0x22:Exiting, please wait. . .
16:12:27:WU00:FS01:0x22:Folding@home Core Shutdown: INTERRUPTED
16:12:27:WU02:FS00:0x22:Folding@home Core Shutdown: INTERRUPTED
16:12:27:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
16:12:28:WU02:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
16:12:37:FS01:Unpaused
16:12:37:FS00:Unpaused
16:12:37:WU00:FS01:Starting
The only thing I find strange is that either some of the upload messages for WU01 may be missing or the upload suddenly became faster. That may be an important clue to solving this mystery.

15:48:58:WU01:FS00:Upload 70.07%
15:50:28:WU01:FS00:Upload 70.39%
15:51:42:WU01:FS00:Upload 70.71%
16:15:51:WU01:FS00:Upload 72.64%
16:15:57:WU01:FS00:Upload 87.75%
16:16:03:WU01:FS00:Upload complete
rickoic
Posts: 322
Joined: Sat May 23, 2009 4:49 pm
Hardware configuration: eVga x299 DARK 2070 Super, eVGA 2080, eVga 1070, eVga 2080 Super
MSI x399 eVga 2080, eVga 1070, eVga 1070, GT970
Location: Mississippi near Memphis, Tn

Re: Upload stuck at 100% | 140.163.4.241

Post by rickoic »

Yes upload slowed to a crawl if not stopped. After I paused the folding, it picked back up to normal speed quickly.

Also I had the same problem on another computer with the upload to the same server. Did the pause and then fold on it and the upload picked back up to normal speed also.

Don't know if it has anything to do with it, but, had the first computer begin uploading to the server and slowed down and stopped as soon as the gpu reached 1% on the next work unit.
Did the pause and fold procedure and the gpu restarted at 0% and as soon as it reached 1% again the previous upload began again as if it had never slowed down?
I'm folding because Dec 2005 I had radical prostate surgery.
Lost brother to spinal cancer, brother-in-law to prostate cancer.
Several 1st cousins lost and a few who have survived.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Upload stuck at 100% | 140.163.4.241

Post by bruce »

I referred this discussion to Development. Maybe they can see something I didn't see ... and can examine to code to see why Pause/Resume seems to work for you. (I'd be happy to be wrong... especially if it points a way toward a fix for the the problem.)
rickoic
Posts: 322
Joined: Sat May 23, 2009 4:49 pm
Hardware configuration: eVga x299 DARK 2070 Super, eVGA 2080, eVga 1070, eVga 2080 Super
MSI x399 eVga 2080, eVga 1070, eVga 1070, GT970
Location: Mississippi near Memphis, Tn

Re: Upload stuck at 100% | 140.163.4.241

Post by rickoic »

Been doing some thinking about this and I may be out of date as I haven't done any programing since my Apple II+ then may IBM 286 when programming was done using 8 bits.

It appears that there is no problem IF the upload completes prior to the 1% completion point of the next work unit. As work units have gotten bigger and bigger and uploads have increased accordingly whey are not completing in time to prevent this problem from happening.

So my thoughts are that the accumulate date to save and the upload sub routines are using the same buffer and the more the date the save data routine puts there prior to saving it the smaller the available buffer for the upload routine becomes until it reaches a critical point and stops until much later in the work unit were something frees up enough of the buffer for it to pick up and complete. Once the upload is complete there is no more problem.

This probably wouldn't show up in testing as the testing machine and server were close enough to each other so that uploads would complete before the 1% check point happened.

As I say it's been a long time since I did any programming so maybe I'm all wet.
I'm folding because Dec 2005 I had radical prostate surgery.
Lost brother to spinal cancer, brother-in-law to prostate cancer.
Several 1st cousins lost and a few who have survived.
toTOW
Site Moderator
Posts: 6296
Joined: Sun Dec 02, 2007 10:38 am
Location: Bordeaux, France
Contact:

Re: Upload stuck at 100% | 140.163.4.241

Post by toTOW »

If upload or download is stuck because of this bug : viewtopic.php?f=18&t=32169 the pause/unpause will have no effect and it will remain stuck. Only a restart of the client will unblock it.

If upload or download was interrupted (transfer error or server timeout) and the client is waiting for a retry, pausing/resuming the slot will clear the timer and trigger an immediate retry.
Image

Folding@Home beta tester since 2002. Folding Forum moderator since July 2008.
rickoic
Posts: 322
Joined: Sat May 23, 2009 4:49 pm
Hardware configuration: eVga x299 DARK 2070 Super, eVGA 2080, eVga 1070, eVga 2080 Super
MSI x399 eVga 2080, eVga 1070, eVga 1070, GT970
Location: Mississippi near Memphis, Tn

Re: Upload stuck at 100% | 140.163.4.241

Post by rickoic »

As indicated it doesn't trigger an immediate retry. It seems to wait until the wu finishes the next percentage update which may take a minute or two.

Don't believe its the same problem. In fact I'm not convinced its a server issue at all, but a fah client issue.
I'm folding because Dec 2005 I had radical prostate surgery.
Lost brother to spinal cancer, brother-in-law to prostate cancer.
Several 1st cousins lost and a few who have survived.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Upload stuck at 100% | 140.163.4.241

Post by bruce »

FAHCore_xx is running as an independent process from FAHClient. When the FAHCore reaches the next 1% it writes a message to the log file which is merged with messages from other FAHCores as well as with the output from FAHClient. The handling of the download stream SHOULDN'T be able to tell that the FAHCore put out a message (...altough the code that adds the datestamp and merges the streams might be a productive place to look for a bug.)
Post Reply