Problems downloading statistics files?

Moderators: Site Moderators, FAHC Science Team

Post Reply
agrikk
Posts: 7
Joined: Wed Dec 18, 2013 8:57 pm

Problems downloading statistics files?

Post by agrikk »

In the last few days I noticed that my regular download of files from apps.foldingathome.org have been failing or taking an incredibly long time to complete.

https://apps.foldingathome.org/daily_us ... ry.txt.bz2
https://apps.foldingathome.org/daily_te ... ry.txt.bz2

normally take a few seconds to download, but now it's taking upwards of twenty minutes with multiple retries and connection closures. I'm on the end of a ten gigabyte pipe and I'm getting 44kB/s download speeds.

Is there something going on that I should know about? Is this a temporary phenomenon?
PantherX
Site Moderator
Posts: 7020
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: Problems downloading statistics files?

Post by PantherX »

Welcome to the F@H Forum agrikk,

I have asked the people responsible about this so let's see what happens. I am aware of another report about the Individual Donor Stats not loading (viewtopic.php?f=16&t=35987).
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
UofM.MartinK
Posts: 59
Joined: Tue Apr 07, 2020 8:53 pm

Re: Problems downloading statistics files?

Post by UofM.MartinK »

If many people need these files regularly, and the front-end is not easy to scale up for whatever reason, would be publishing the stats as torrent a potential solution?

This could even be community-driven, if there are no legal implications against it...
PantherX
Site Moderator
Posts: 7020
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: Problems downloading statistics files?

Post by PantherX »

The update is that everything seems fine from the back-end. If it is a routing or a network issue, there's almost nothing that F@H can do about it.
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
agrikk
Posts: 7
Joined: Wed Dec 18, 2013 8:57 pm

Re: Problems downloading statistics files?

Post by agrikk »

PantherX wrote:Welcome to the F@H Forum agrikk,

I have asked the people responsible about this so let's see what happens. I am aware of another report about the Individual Donor Stats not loading (viewtopic.php?f=16&t=35987).
Thank you! :)

I've been pulling these stats files since 2013 and since covid, I understand that availability (both in data files and actual work units) has been taxed.
UofM.MartinK
Posts: 59
Joined: Tue Apr 07, 2020 8:53 pm

Re: Problems downloading statistics files?

Post by UofM.MartinK »

PantherX wrote:The update is that everything seems fine from the back-end. If it is a routing or a network issue, there's almost nothing that F@H can do about it.
Other than (also) distributing it as torrent? :)

Each file with date tag - why not? Is it worth an experiment?

Update: add a fancy way of signing to guarantee authenticity... perhaps even as part of the filename? :)
PantherX
Site Moderator
Posts: 7020
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: Problems downloading statistics files?

Post by PantherX »

UofM.MartinK wrote:...Other than (also) distributing it as torrent? :)

Each file with date tag - why not? Is it worth an experiment?

Update: add a fancy way of signing to guarantee authenticity... perhaps even as part of the filename? :)
I didn't mention the feature during my conversation. However, you're more than welcomed to submit your ideas here (assuming that it hasn't already been suggested before): https://github.com/FoldingAtHome/fah-issues/issues
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
UofM.MartinK
Posts: 59
Joined: Tue Apr 07, 2020 8:53 pm

Re: Problems downloading statistics files?

Post by UofM.MartinK »

Doesn't seem to be in high demand, based on the reaction here, and the files are smaller than I thought :)

If bandwidth or availability at various locations would be a problem regularly, once could also distribute them with a CDN like cloudflare.

But that doesn't seem to be a case, do no need to fix something that ain't broken :)
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Problems downloading statistics files?

Post by Neil-B »

phew ... the torrent elephant in the room seems to have wandered off of its own accord ... relaxes, smiles, heart rate lowers ... :D
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
cine.chris
Posts: 78
Joined: Sun Apr 26, 2020 1:29 pm

Re: Problems downloading statistics files?

Post by cine.chris »

I'm trying to obtain a daily archive of FAH user/team bzipped files for 2020 Jan-Nov.
Hoping that someone on this thread might be able to help me locate a source.
Image Image
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Problems downloading statistics files?

Post by bruce »

You may be out-of-luck.

As far as FAH is concerned, the database is constantly changing and always up-to-date. There's no need for a history. (Well, there are backups, of course, but they're not distributed because they're obsolete.) The data are updated minute-to-minute as new reports of completed WUs are come in. No part of the database is considered as static. (Well, except for the current statistics for Donors who have died or left FAH, but there's no attempt to identify them.)

The third party stats sites do retain one (or more) previous flat-file and calculate the differences between the latest and that previous version. From that, they are able to calculate trends and generate the kind of history you're looking for. They may be able to provide the historic information you're looking for.
cine.chris
Posts: 78
Joined: Sun Apr 26, 2020 1:29 pm

Re: Problems downloading statistics files?

Post by cine.chris »

I connected with agrikk.
He has data back to 2014, which I think will be more than adequate for the patterns that I want to look at.
@Bruce
What part of So Cal?
I'm a So Cal native, born in Burbank, raised in the 'valley', North Hollywood.
Lockheed moved me to Georgia in '93 with the F22 program.
Image Image
Post Reply