Recent WUs are too large to upload

Moderators: Site Moderators, FAHC Science Team

Joe_H
Site Admin
Posts: 7856
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Recent WUs are too large to upload

Post by Joe_H »

I have checked my logs for Project 17219, the upload size has varied from about 83 MB to about 92 MB. This is without any errors showing up in the runs, and whether or not the WU was paused. In fact the WU processed on my laptop where it was paused during the daytime sent one of those 83 MB results files. The download size is just over 31 MB.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
comixgoddess
Posts: 88
Joined: Wed Apr 08, 2020 9:57 pm
Location: Pacific Northwest

Re: Recent WUs are too large to upload

Post by comixgoddess »

Neil-B wrote:I'd check what size my uploads for that project are but my server is down for rebuild at the moment so haven't got access to those logs ... If someone else spots this ad is running 17219 wus then perhaps they could confirm the upload sizes they are seeing? :)
My last 17219 unit downloaded at 31.24MiB; the upload was 83.58MiB and took just over two minutes to upload successfully. This unit was run on my laptop without pausing.
Image
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Recent WUs are too large to upload

Post by Neil-B »

... thanks both ... so yup - the OP is right and this project is just shipping large uploads as standard ... fingers crossed Joe_H's engagement with the researcher pays dividends (and the flag still works as hoped?)
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Recent WUs are too large to upload

Post by bruce »

Neil-B wrote:I think what bruce was getting at wasn't the upload failing but questioning if during the course of processing these larger upload wus there were any errors that the client/core managed ... When an error occurs mid wu and the core can correct it then the error reports can significantly inflate the size of the final package ... Pausing a wu many times can sometimes have the same effect.
Right.

The upload package includes compressed versions of the logs as well as shapshots of the atoms. The researcher is responsible for the number of atoms and the number of snapshots.

The logs can grow quite significantly if there are a lot of errors reported or a lot of Pause/Unpause steps and that's something you might be able to manage.
Markus_Laker
Posts: 20
Joined: Sun Dec 01, 2019 11:36 am

Re: Recent WUs are too large to upload

Post by Markus_Laker »

Thanks to everyone. I appreciate your engagement and help.

The most common reason I pause the folding process is if I have to reboot the PC in order to discard an over-sized WU. My workflow is:

Code: Select all

cd /var/lib/fahclient/work
sudo mv 02 hide-02    # or whichever number it happens to be
sudo shutdown -r now
If there's a way to interrupt an upload without rebooting, I haven't found it. In particular,

Code: Select all

sudo /etc/init.d/FAHClient stop
seems not to do it.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Recent WUs are too large to upload

Post by bruce »

Markus_Laker wrote:Thanks to everyone. I appreciate your engagement and help.

If there's a way to interrupt an upload without rebooting, I haven't found it. In particular,

Code: Select all

sudo /etc/init.d/FAHClient stop
seems not to do it.
Which version of Linux are you running?

Please post the top ~100 lines of FAH's log.txt
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Recent WUs are too large to upload

Post by bruce »

We need a better way to manage big WUs for people who have an internet connections that uploads slower than most others. We still have a percentage of Donors on DSL or slower.

(I suffered with a slow connection for a long time.)
Joe_H
Site Admin
Posts: 7856
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Recent WUs are too large to upload

Post by Joe_H »

bruce wrote:We need a better way to manage big WUs for people who have an internet connections that uploads slower than most others. We still have a percentage of Donors on DSL or slower.

(I suffered with a slow connection for a long time.)
True, and it is not just DSL. There are residential cable internet connections still that haven low limits on upload speed, 25 mbps down but under 5 mbps up. There are also people on metered wireless services.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Markus_Laker
Posts: 20
Joined: Sun Dec 01, 2019 11:36 am

Re: Recent WUs are too large to upload

Post by Markus_Laker »

I wish! Right now, I have 14Mbps down and 444kbps up. But I have to share that with my wife, who does several hours a day of video conferences for her business, and I also need it for my work PC's connection to the office. And this is the fastest I can get from any ISP, at any price.
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Recent WUs are too large to upload

Post by Neil-B »

Daft question but have you checked out Starlink ... Still early days but they have beta up and running I think in some areas ... ours here in UK is slated for autumn this year .. and it might not work due to field of view reqs in your area but I mention it just in case you hadn't considered them - it was your "at any price" that made me think of them !!
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Recent WUs are too large to upload

Post by bruce »

Most people send tiny bursts of responses at 0.44Mbps and it may be adequate for you to converse with web servers that can talk to you at 14Mbps.

For anybody who has better bandwidth UP you can get this project if you configure max-packet=big. Without that setting, you won't get that project ... which is only a problem for the project owner.
Markus_Laker
Posts: 20
Joined: Sun Dec 01, 2019 11:36 am

Re: Recent WUs are too large to upload

Post by Markus_Laker »

Apologies for necroposting. Joe_H, would you mind having a word with the researcher behind project 17220, please?

Code: Select all

16:38:15:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:17220 run:6863 clone:0 gen:1 core:0xa7 unit:0x00000000000000010000434400001acf
16:38:15:WU03:FS01:Uploading 31.77MiB to 206.223.170.146
Many thanks!
Post Reply