Not getting credit

Moderators: Site Moderators, FAHC Science Team

msultan
Pande Group Member
Posts: 134
Joined: Mon Jun 24, 2013 10:27 pm

Re: Stats not updating?

Post by msultan »

We are currently running a manual stat update, it should take a few hours since A LOT of WUs are currently un-accredited. I am sorry this wasnt caught earlier on.
msultan
Pande Group Member
Posts: 134
Joined: Mon Jun 24, 2013 10:27 pm

Re: Stats not updating?

Post by msultan »

The stats should now start to get updated. i am still crediting the missing WUs but they should hopefully be done by the end of today. I am doing them in batches to figure out exactly why the system is breaking. Please let us know of further issues.
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Post by drougnor »

Values did increment in the 2pm Eastern Update, so we'll see things getting back to normal on the 3rd party sites over the next few updates.

Thank you for the hard work you all do!

d
ChristianVirtual
Posts: 1596
Joined: Tue May 28, 2013 12:14 pm
Location: Tokyo

Re: Stats not updating?

Post by ChristianVirtual »

I got a file and push was send as scheduled ... thanks for the work msultan
ImageImage
Please contribute your logs to http://ppd.fahmm.net
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Stats not updating?

Post by bruce »

drougnor wrote:... over the next few updates.

Thank you for the hard work you all do!
Let me emphasize those words.

1) You'll be seeing incomplete updates until this is completely resolved. [I don't want a long list of folks saying "I completed WU xx (xx,xx,xx) but it wasn't credited."]

2) Many, many thanks to msultan and his team of FAH members who are supporting him.
msultan
Pande Group Member
Posts: 134
Joined: Mon Jun 24, 2013 10:27 pm

Re: Stats not updating?

Post by msultan »

Thanks a lot for the support everyone. This is the least we can do given that all of you are generously donating compute time for our calculations. I think I have the root cause figured out but it will likely require taking stats offline for more than a few hours. Please let me know if there are more issues. Again, I am really sorry about the problems that this has been causing recently.
QuintLeo
Posts: 52
Joined: Sat Jun 17, 2017 5:43 am

Re: Extreme Overclocking Stats not updating

Post by QuintLeo »

And the Standford stats server is borked AGAIN - seems like they "fix them" then they last perhaps a few hours and go down/fail AGAIN.

IMO Stanford should farm out their stats to an outside third party that actually CARES about keeping the stats infrastructure working, the last month has been totally UNACCEPTABLE on downtime to the point it's driving some folks OUT OF FOLDING (and has ME very close to that point).
QuintLeo
Posts: 52
Joined: Sat Jun 17, 2017 5:43 am

Re: Stats not updating?

Post by QuintLeo »

The problem is on Stanford's end - the server(s) that generate the "flat file" they REQUIRE third party stat sites to use has been borked more of the time this past week than not, and quite a bit of the time for the last month.
The issue has been going on so long it's starting to tick some folks off enough to QUIT FOLDING ENTIRELY - and it's got *ME* very close to that point, given that Standford has made it obvious through their INACTION that they don't care about the issue.
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Post by drougnor »

I'm hoping that the current outage is you having taken them down for the fix you mentioned. If not, they've been down again since the 5pm Eastern update.

d
SombraGuerrero
Posts: 118
Joined: Mon Mar 16, 2009 3:06 am

Re: Stats not updating?

Post by SombraGuerrero »

I will be interested in a high level root cause analysis. As a QA analyst and coder, curiosity abounds.
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Post by drougnor »

QuintLeo wrote:The problem is on Stanford's end - the server(s) that generate the "flat file" they REQUIRE third party stat sites to use has been borked more of the time this past week than not, and quite a bit of the time for the last month.
The issue has been going on so long it's starting to tick some folks off enough to QUIT FOLDING ENTIRELY - and it's got *ME* very close to that point, given that Standford has made it obvious through their INACTION that they don't care about the issue.
Except that you are responding directly after they HAVE taken action, potentially diagnosed the deeper issue and are potentially IN THE MIDDLE Of fixing it. Please don't spill your sour grapes all over the rest of us who are actually rooting for this to be fixed and who are doing our best to give useful information to assist IN that fixing.

Thanks.

d
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: Stats not updating?

Post by SteveWillis »

It appears to be down again.
Image

1080 and 1080TI GPUs on Linux Mint
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Post by drougnor »

We got an update at 5pm EDT, missed one at 6pm EDT, picked up another at 7pm EDT and my system isn't downloading for the current hour, so it still see's the 7pm timestamp. They may still actively be working on things on the server, trying to flush the bug. We're likely to see more hinkyness before they get it sorted completely.

d
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Stats not updating?

Post by bruce »

Yes, they are actively working on the problem.

The best assumption that seems to fit all the know facts, is that the server has been having hardware problems that appear as unexpected crashes. The hardware is old and does need to be replaced but it's essential that the stats for all WUs completed during these crashes are successfully placed in an updated copy of the stats db -- and that takes some extraordinary efforts.

There's already a plan to move the databasae and it's associated processes to new hardware -- and that will necessitate some planned downtime -- together with careful management of the data associated with the updates that should be occurring during the downtime so they can be re-added on the relocated database. Once all that has been accomplished, the automatic updates should resume. In the meantime, there's a lot of manual operations that need to be processed, checked, and rechecked.

I plead for patience on your part. I know the Pande Group takes this problem very seriously and they're doing everything they can to reach a permanent solution -- without a permanent loss of any data.
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Post by drougnor »

Each error I've seen on my side for the last few hours have been DIFFERENT errors, so that's definitely a sign of progress being made instead of the server just sitting down and playing a few games of sol.exe on a non windows box. I'm definitely happy to see the progress.

However, it's getting late on the east coast so I'll likely have to call it a night soon. I'll happily check on things and provide a quick update on how things appear on my side of the system in the AM.

Thanks again, guys and gals!

d
Post Reply