Dramatically Lower Points

Moderators: Site Moderators, FAHC Science Team

Dramatically Lower Points

Postby oliverjdent » Mon Feb 22, 2021 8:56 pm

I have been averaging around one million points per day until 2/19/21. Even with the same number of CPUs and GPUs working, the number of points went down to 500,000 points bottoming out at 270,362 points on 2/20/21.

https://folding.extremeoverclocking.com/user_summary.php?s=&u=186507

I have been folding since June 2006. This is the first time the number of points my machines have generated has dropped off so dramatically without either a hardware failure or a FAH client failure.

If anyone can explain why my points have dropped by so much I would appreciate the assistance.

If there is something I need to do to bring my points back up to the levels I have expected please let me know.

Thank you.

Oliver.
oliverjdent
 
Posts: 9
Joined: Thu Jan 02, 2014 7:36 pm

Re: Dramatically Lower Points

Postby JimboPalmer » Tue Feb 23, 2021 12:15 am

I am (rarely) getting a WU that refuses to upload, I think at least one server has no disk space. It tries multiple servers and ports and eventually is uploaded.

I am hearing that others, with different workloads and resources, have even more upload failures in their logs. Some are still not uploaded once they pass the deadlines.

I would think that would be in your log.
Tsar of all the Rushers
I tried to remain childlike, all I achieved was childish.
A friend to those who want no friends
JimboPalmer
 
Posts: 2177
Joined: Mon Feb 16, 2009 5:12 am
Location: Greenwood MS USA

Re: Dramatically Lower Points

Postby Neil-B » Tue Feb 23, 2021 12:18 am

Eoc wasnt showing a big reduction in wu throughput .. might be useful to check if there has been a shift in project your kit has been folding .. also check to see if gpus are using cuda not opencl if nvidia ones
1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro, Quadro M1000M 2GB, FAH 7.6.21
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro, GTX 750Ti 2GB, FAH 7.6.21
Neil-B
 
Posts: 1721
Joined: Sun Mar 22, 2020 6:52 pm
Location: UK

Re: Dramatically Lower Points

Postby v00d00 » Tue Feb 23, 2021 3:40 am

Admittedly their have been a few workunits knocking about that give less PPD than average. I've noticed it and others on beta have and comments have been made. I dont think those workunits were ever adjusted (asuming they needed to be). Can't remeber which Projects they were from, but I had a diabetes projects from the High Priority workunits that finished in about 40 mins on a RTX 2080 and they gave less PPD than doing most other workunits. Not that I care per se, they get done all the same regardless of points. But obviously other people do.
Image
User avatar
v00d00
 
Posts: 385
Joined: Sun Dec 02, 2007 5:53 am
Location: UK

Re: Dramatically Lower Points

Postby oliverjdent » Tue Feb 23, 2021 2:51 pm

@JimboPalmer, thank you for the suggestion. I will check my logs today for upload failures.

@Neil-B, I have seen multiple "Completed" entries in the FAH Control viewer that didn't seem to upload and clear. This might be the cause.

@v00d00, thank you for the feedback.

This morning, I show 930,927 points on 41 work units. I typically complete 20 to 30 WU per day. So, it looks like things are returning to normal.

Thank you all for your assistance.

Oliver.
oliverjdent
 
Posts: 9
Joined: Thu Jan 02, 2014 7:36 pm

Re: Dramatically Lower Points

Postby Neil-B » Tue Feb 23, 2021 3:29 pm

Do your logs show multiple attempts to upload the completed wus with short response errors where they don't? ... there are issues at the moment with at least one av/firewall stopping uploads working properly
Neil-B
 
Posts: 1721
Joined: Sun Mar 22, 2020 6:52 pm
Location: UK

Re: Dramatically Lower Points

Postby Neil-B » Tue Feb 23, 2021 3:31 pm

Sometimes av/firewall issues self resolve as the vendors update their products security definitions
Neil-B
 
Posts: 1721
Joined: Sun Mar 22, 2020 6:52 pm
Location: UK

Re: Dramatically Lower Points

Postby oliverjdent » Wed Feb 24, 2021 12:44 am

I went through my logs on all three of my machines. I didn't find any issues with uploading finished WUs.

I did find this error several times in one of my machine's GPU output (GeForce GTX 660). It is now running without errors.
01:22:08:WU01:FS01:0x22:An exception occurred at step 39908: Particle coordinate is nan
01:22:08:WU01:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
01:22:08:WU01:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
01:22:09:WARNING:WU01:FS01:FahCore returned: CORE_RESTART (98 = 0x62)

On my main machine I found these two errors in the last 24 hours:
******************************* Date: 2021-02-22 *******************************
10:18:02:ERROR:WU02:FS01:Exception: Server did not assign work unit
10:18:03:ERROR:WU02:FS01:Exception: Server did not assign work unit
10:19:03:ERROR:WU02:FS01:Exception: Server did not assign work unit
10:18:02:ERROR:WU02:FS01:Exception: Server did not assign work unit
10:18:03:ERROR:WU02:FS01:Exception: Server did not assign work unit
10:19:03:ERROR:WU02:FS01:Exception: Server did not assign work unit

16:49:55:WARNING:WU02:FS00:Failed to get assignment from 'assign1.foldingathome.org:80': No WUs available for this configuration
16:49:55:WARNING:WU02:FS00:Failed to get assignment from 'assign1.foldingathome.org:80': No WUs available for this configuration

Now, both the CPU and GPU are working without issue.
oliverjdent
 
Posts: 9
Joined: Thu Jan 02, 2014 7:36 pm

Re: Dramatically Lower Points

Postby Neil-B » Wed Feb 24, 2021 11:58 am

Firstly for the main machine the two errors may well just be down to availability of WUs for that kits configuration - it happens from time to time - there will be a rash of posts on the forums along the lines of "run out of wus" or "are we low on wus" - basically the researchers sometimes are too busy handling all the science processed and haven't got more projects in the queue - it is a relatively rare occurrence but when it happens can leave slots/machine idle - sometimes just every now and then but at other times totally idle for days - the researchers do try to avoid this obviously.

The GTX 660 error is usually related to OC'ing issues (or running at a clock where the GPU is unstable - sometimes even normal clocks can suffer this) ... if it only happens occasionally and wus still complete it may be a case of just monitor it ... if wus start failing because of it then action needs to be taken ... even factory overclocks can suffer from this especially with cuda enable a8 core wus ... sometimes age/dust/dried tim all play into it ... a search of the forums for "particle is nan" will give you more of a picture than I can and some of the other edge cases that might cause this error.
Neil-B
 
Posts: 1721
Joined: Sun Mar 22, 2020 6:52 pm
Location: UK

Re: Dramatically Lower Points

Postby v00d00 » Thu Feb 25, 2021 12:44 am

Generally if you get one NaN error, report it, it may be that workunit which is faulty. If you get loads of them its a hardware issue, drop the overclock or maybe underclock a little, until it goes away.

Beyond that do the standard maintenance tasks that are required. Vacuum the card out from time to time if it gets dusty or you see high temps on the card. If underclocking the card doesnt fix the problem, it may be a PSU issue, test with a different PSU. if the card is oldish you could strip it and apply some new thermal paste to the gpu core and reassemble, also if any exposed chips on the card are getting hot, you could try adding a heatsink to them.
User avatar
v00d00
 
Posts: 385
Joined: Sun Dec 02, 2007 5:53 am
Location: UK

Re: Dramatically Lower Points

Postby oliverjdent » Fri Feb 26, 2021 2:43 pm

@Neil-B and v00d00,

Thank you for the responses and the good information. I will check my logs more regularly.

I hope both of you have a good weekend.

Oliver.
oliverjdent
 
Posts: 9
Joined: Thu Jan 02, 2014 7:36 pm

Re: Dramatically Lower Points

Postby Neil-B » Fri Feb 26, 2021 3:48 pm

... and yourself ... fold on :)
Neil-B
 
Posts: 1721
Joined: Sun Mar 22, 2020 6:52 pm
Location: UK


Return to Discussions of General-FAH topics

Who is online

Users browsing this forum: Google [Bot] and 3 guests

cron