Stats not updating?

Moderators: Site Moderators, PandeGroup

Re: Stats not updating?

Postby drougnor » Mon Sep 11, 2017 11:38 pm

We got two hourly updates this time, the 2pm Eastern and 3pm Eastern, before it stopped again.

What I've been able to piece together of how this system works is that the flat file gets generated with the current timestamp in PST and then gets populated with the current hour's worth of data (Name, Points total, WUs Total, Team number for the Users and Team Number, Team Name, Points total and WU total for the Teams). It appears that the crash in this particular case is happening after the blank files get generated with the timestamp, but before the data gets inserted. The system sees a file that it then zips up and publishes the empty file and zipped archive.

However, as the process manages to complete at least a few times before the next error, this is telling me that the database itself IS indeed whole and hale, esp since we can view the data through the interactive web portal on the official stats page.

Not that any of this is really relevant to most of you, but it was fun for me to piece together over a bit of time of examining error messages and figuring out what I was doing wrong and what that meant for the backend system and how it was apparently constructed.

d
drougnor
 
Posts: 133
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Postby ChristianVirtual » Mon Sep 11, 2017 11:55 pm

It is also relevant to me. How else I can send push notification on stats changes when the stats file is empty ... (maybe I should at least push a note that stats is down and hope one in PG has an Apple Watch ;-)
ImageImage
Please contribute your logs to http://ppd.fahmm.net
User avatar
ChristianVirtual
 
Posts: 1507
Joined: Tue May 28, 2013 12:14 pm
Location: 日本 東京

Re: Stats not updating?

Postby bruce » Tue Sep 12, 2017 3:00 am

It's strange, but we're not seeing a single type of failure over the last few weeks.

In one case, http://fah-web.stanford.edu which contains /cgi-bin and other sub-files was off-line. Last time, the message was that the server was too busy. Now I can get to certain scripts that are inside of fah-web.stanford.edu/cgi-bin. I wonder how many DIFFERENT problems we're actually looking at.
bruce
 
Posts: 21416
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Stats not updating?

Postby SombraGuerrero » Tue Sep 12, 2017 3:28 am

That's the problem with HTTP500s. Even when you can drill down to some level of error detail, they're still so generic, they can mean anything. It seems to me that there's almost a cascading failure in the file system. Some sort of dirty process or cluster of files that's corrupting permissions, or page faulting or locking down shared resources in an access violation type of way. If it were a hardware failure, it wouldn't so consistently right itself with a simple reboot. That is, unless the storage devices are failing and there are literal read/write or file I/O problems happening. Do we know if the server has a something like a RAID array set up? Something to at least provide file system redundancy if no other type of redundancy?
SombraGuerrero
 
Posts: 98
Joined: Mon Mar 16, 2009 3:06 am

Re: Stats not updating?

Postby QuintLeo » Tue Sep 12, 2017 3:33 am

Hopefully they'll get the move to new hardware done SOON, since the old hardware has been showing WAY too much downtime the last couple weeks to be trusted.
QuintLeo
 
Posts: 39
Joined: Sat Jun 17, 2017 5:43 am

Re: Stats not updating?

Postby rwh202 » Tue Sep 12, 2017 7:43 am

And 'Trust' is what a lot of this is about. If donors can't trust that the project is being run and managed in a competent way, then what's the incentive of putting in their own effort?

Yes, I realise that the stats server isn't the same as the 'science', but it provides the most visible and understandable evidence of output to the majority of donors. Also, it it considered trivial to get right, so if something 'simple' can't be done right, what faith is there in the rest of the system?

To suggest that money or resource is the problem is not believable - A few days effort and a couple $100 a month on Azure / AWS and I'm sure they could be up and running (plus, if they spoke nicely, I bet it wouldn't cost a penny).
rwh202
 
Posts: 294
Joined: Mon Nov 15, 2010 8:51 pm
Location: South Coast, UK

Re: Stats not updating?

Postby drougnor » Tue Sep 12, 2017 12:26 pm

A possible potential as well may be a result of the open ended nature of letting people pick names as long and mostly complicated as they want. I'm not sure what kind of error trapping is being done on name creation, but with the influx of names that have a long hash in them that are being used for the Folding-Coin mining operations, there is a slim chance that someone accidentally created a name that is crashing one of the processes.

I'm not saying it's absolutely LIKELY, but it is something I hope the Folding programmers have looked at and eliminated as a potential cause.

d
drougnor
 
Posts: 133
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Postby SombraGuerrero » Tue Sep 12, 2017 12:33 pm

That is feasible. I recently had one of our vendors at work have to change the data type of a field from like varchar to blob2 because users were simply trying to stuff too much in there.
SombraGuerrero
 
Posts: 98
Joined: Mon Mar 16, 2009 3:06 am

Re: Stats not updating?

Postby Mike.Barr » Tue Sep 12, 2017 1:31 pm

There was an error accessing/using the database.
The Folding@home team is working to fix this issue.

At least 24 hours since this message posted...Team updates not being posted.

Mike
Mike.Barr
 
Posts: 3
Joined: Thu Aug 18, 2016 1:41 pm

Re: Stats not updating?

Postby drougnor » Tue Sep 12, 2017 2:37 pm

Official page is back up. Waiting to see if my system picks up the flat file now.

**edit** So, as I said earlier, the official stats page is presenting, but now the .txt file is no longer being published. The .bz2, while still empty, is there, however.
drougnor
 
Posts: 133
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Postby EmperiuM » Tue Sep 12, 2017 3:54 pm

There is data on the .bz file, but mine, for instance, is the same that I had on the Date of last work unit 2017-09-08 09:06:40.
For me, it didn't count my points since that day :(
EmperiuM
 
Posts: 4
Joined: Tue Jan 06, 2015 2:22 pm

Re: Stats not updating?

Postby nsummy » Tue Sep 12, 2017 4:03 pm

On my stats page it says: Date of last work unit 2017-09-11 11:00:16

:(
nsummy
 
Posts: 4
Joined: Fri Aug 11, 2017 1:46 pm

Re: Stats not updating?

Postby drougnor » Tue Sep 12, 2017 4:51 pm

I can verify with A-B comparisons on the last 4 data sets/flat files the system produced have been the identical data.

d
drougnor
 
Posts: 133
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Postby msultan » Tue Sep 12, 2017 5:11 pm

We are looking into the stats issue. Please give me a few hours to see what might be up. Our initial attempt to debug this didnt work.
User avatar
msultan
Pande Group Member
 
Posts: 133
Joined: Mon Jun 24, 2013 10:27 pm

Re: Stats not updating?

Postby drougnor » Tue Sep 12, 2017 5:20 pm

By all means take all the time you need. I'm much happier with a complete and correct solution that takes some time than a 'we think it's working, oooops, guess not' rushed patch job.
drougnor
 
Posts: 133
Joined: Tue Dec 29, 2009 2:21 am

PreviousNext

Return to Issues with a specific server

Who is online

Users browsing this forum: SteveWillis and 1 guest

cron