Stats not updating?

Moderators: Site Moderators, PandeGroup

Re: Stats not updating?

Postby msultan » Tue Sep 12, 2017 5:57 pm

We are currently running a manual stat update, it should take a few hours since A LOT of WUs are currently un-accredited. I am sorry this wasnt caught earlier on.
User avatar
msultan
Pande Group Member
 
Posts: 133
Joined: Mon Jun 24, 2013 10:27 pm

Re: Stats not updating?

Postby msultan » Tue Sep 12, 2017 7:04 pm

The stats should now start to get updated. i am still crediting the missing WUs but they should hopefully be done by the end of today. I am doing them in batches to figure out exactly why the system is breaking. Please let us know of further issues.
User avatar
msultan
Pande Group Member
 
Posts: 133
Joined: Mon Jun 24, 2013 10:27 pm

Re: Stats not updating?

Postby drougnor » Tue Sep 12, 2017 7:06 pm

Values did increment in the 2pm Eastern Update, so we'll see things getting back to normal on the 3rd party sites over the next few updates.

Thank you for the hard work you all do!

d
drougnor
 
Posts: 132
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Postby ChristianVirtual » Tue Sep 12, 2017 7:08 pm

I got a file and push was send as scheduled ... thanks for the work msultan
ImageImage
Please contribute your logs to http://ppd.fahmm.net
User avatar
ChristianVirtual
 
Posts: 1474
Joined: Tue May 28, 2013 12:14 pm
Location: 日本 東京

Re: Stats not updating?

Postby bruce » Tue Sep 12, 2017 7:27 pm

drougnor wrote:... over the next few updates.

Thank you for the hard work you all do!


Let me emphasize those words.

1) You'll be seeing incomplete updates until this is completely resolved. [I don't want a long list of folks saying "I completed WU xx (xx,xx,xx) but it wasn't credited."]

2) Many, many thanks to msultan and his team of FAH members who are supporting him.
bruce
 
Posts: 21270
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Stats not updating?

Postby msultan » Tue Sep 12, 2017 8:52 pm

Thanks a lot for the support everyone. This is the least we can do given that all of you are generously donating compute time for our calculations. I think I have the root cause figured out but it will likely require taking stats offline for more than a few hours. Please let me know if there are more issues. Again, I am really sorry about the problems that this has been causing recently.
User avatar
msultan
Pande Group Member
 
Posts: 133
Joined: Mon Jun 24, 2013 10:27 pm

Re: Extreme Overclocking Stats not updating

Postby QuintLeo » Tue Sep 12, 2017 11:08 pm

And the Standford stats server is borked AGAIN - seems like they "fix them" then they last perhaps a few hours and go down/fail AGAIN.

IMO Stanford should farm out their stats to an outside third party that actually CARES about keeping the stats infrastructure working, the last month has been totally UNACCEPTABLE on downtime to the point it's driving some folks OUT OF FOLDING (and has ME very close to that point).
QuintLeo
 
Posts: 35
Joined: Sat Jun 17, 2017 5:43 am

Re: Stats not updating?

Postby QuintLeo » Tue Sep 12, 2017 11:13 pm

The problem is on Stanford's end - the server(s) that generate the "flat file" they REQUIRE third party stat sites to use has been borked more of the time this past week than not, and quite a bit of the time for the last month.
The issue has been going on so long it's starting to tick some folks off enough to QUIT FOLDING ENTIRELY - and it's got *ME* very close to that point, given that Standford has made it obvious through their INACTION that they don't care about the issue.
QuintLeo
 
Posts: 35
Joined: Sat Jun 17, 2017 5:43 am

Re: Stats not updating?

Postby drougnor » Tue Sep 12, 2017 11:14 pm

I'm hoping that the current outage is you having taken them down for the fix you mentioned. If not, they've been down again since the 5pm Eastern update.

d
drougnor
 
Posts: 132
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Postby SombraGuerrero » Tue Sep 12, 2017 11:14 pm

I will be interested in a high level root cause analysis. As a QA analyst and coder, curiosity abounds.
SombraGuerrero
 
Posts: 97
Joined: Mon Mar 16, 2009 3:06 am

Re: Stats not updating?

Postby drougnor » Tue Sep 12, 2017 11:17 pm

QuintLeo wrote:The problem is on Stanford's end - the server(s) that generate the "flat file" they REQUIRE third party stat sites to use has been borked more of the time this past week than not, and quite a bit of the time for the last month.
The issue has been going on so long it's starting to tick some folks off enough to QUIT FOLDING ENTIRELY - and it's got *ME* very close to that point, given that Standford has made it obvious through their INACTION that they don't care about the issue.


Except that you are responding directly after they HAVE taken action, potentially diagnosed the deeper issue and are potentially IN THE MIDDLE Of fixing it. Please don't spill your sour grapes all over the rest of us who are actually rooting for this to be fixed and who are doing our best to give useful information to assist IN that fixing.

Thanks.

d
drougnor
 
Posts: 132
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Postby SteveWillis » Wed Sep 13, 2017 12:16 am

It appears to be down again.
Image
My thanks to my very indulgent wife
http://folding.extremeoverclocking.com/user_summary.php?s=&u=712804

3 AMD Linux rigs 3, 4, and 5 GPUs 7 X GTX 1080, 5 X GTX 1080 TI
SteveWillis
 
Posts: 210
Joined: Fri Apr 15, 2016 12:42 am

Re: Stats not updating?

Postby drougnor » Wed Sep 13, 2017 12:37 am

We got an update at 5pm EDT, missed one at 6pm EDT, picked up another at 7pm EDT and my system isn't downloading for the current hour, so it still see's the 7pm timestamp. They may still actively be working on things on the server, trying to flush the bug. We're likely to see more hinkyness before they get it sorted completely.

d
drougnor
 
Posts: 132
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Postby bruce » Wed Sep 13, 2017 1:00 am

Yes, they are actively working on the problem.

The best assumption that seems to fit all the know facts, is that the server has been having hardware problems that appear as unexpected crashes. The hardware is old and does need to be replaced but it's essential that the stats for all WUs completed during these crashes are successfully placed in an updated copy of the stats db -- and that takes some extraordinary efforts.

There's already a plan to move the databasae and it's associated processes to new hardware -- and that will necessitate some planned downtime -- together with careful management of the data associated with the updates that should be occurring during the downtime so they can be re-added on the relocated database. Once all that has been accomplished, the automatic updates should resume. In the meantime, there's a lot of manual operations that need to be processed, checked, and rechecked.

I plead for patience on your part. I know the Pande Group takes this problem very seriously and they're doing everything they can to reach a permanent solution -- without a permanent loss of any data.
bruce
 
Posts: 21270
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Stats not updating?

Postby drougnor » Wed Sep 13, 2017 1:38 am

Each error I've seen on my side for the last few hours have been DIFFERENT errors, so that's definitely a sign of progress being made instead of the server just sitting down and playing a few games of sol.exe on a non windows box. I'm definitely happy to see the progress.

However, it's getting late on the east coast so I'll likely have to call it a night soon. I'll happily check on things and provide a quick update on how things appear on my side of the system in the AM.

Thanks again, guys and gals!

d
drougnor
 
Posts: 132
Joined: Tue Dec 29, 2009 2:21 am

PreviousNext

Return to Issues with a specific server

Who is online

Users browsing this forum: Ricorocks and 1 guest

cron