Project 13424 (Moonshot) very low PPD

Moderators: Site Moderators, FAHC Science Team

Project 13424 (Moonshot) very low PPD

Postby Shirty » Mon Aug 24, 2020 1:54 pm

Just an observation, the PPD on these units seems to be wildly lower then I would expect, around half of normal output. Currently running the following units:

GTX 1080Ti - P13424(R147,C32,G0) - 730,000 ppd - 3:05 tpf
RTX 2060 - P13424(R30,C27,G1) - 730,000 ppd - 3.05 tpf
RTX 2070 - P13424(R120,C5,G1) - 800,000 ppd - 2.57 tpf
RTX 2070 Super - P13424(R90,C12,G1) - 950,000 ppd - 2.38 tpf
RTX 2070 Super - P13424(R21,C18,G1) - 1,100,000 ppd - 2:22 tpf
RTX 2070 Super - P13424(R50,C49,G1) - 1,180,000 ppd - 2.16 tpf

Any idea why these are all running so slowly, utilising circa 60% of GPU power? It has literally shaved 6 million ppd off my daily output since I started picking these up this morning.

Once all 11 of my cards pick these up (as they invariably will) I will see my total output halved from around 21-22 million to 10-11 million ppd.
Image
Shirty
 
Posts: 49
Joined: Thu Jul 11, 2019 11:19 pm

Re: Project 13424 (Moonshot) very low PPD

Postby HaloJones » Mon Aug 24, 2020 5:44 pm

Strange.My 1070s are down on 13422 but only by around 20% not 50%.

EDIT: Meant to write 13424
Last edited by HaloJones on Tue Aug 25, 2020 9:03 am, edited 1 time in total.
1x Titan X, 5x 1070, 1x 970, 1 x Ryzen 3600

Image
HaloJones
 
Posts: 869
Joined: Thu Jul 24, 2008 11:16 am

Re: Project 13424 (Moonshot) very low PPD

Postby Shirty » Mon Aug 24, 2020 6:13 pm

I've got no real issue with 13422, which fortuitously I seem to have gone back to on most of my cards since I posted the OP. I just thought it might benefit John to see my findings in case it's a wider issue.
Shirty
 
Posts: 49
Joined: Thu Jul 11, 2019 11:19 pm

Re: Project 13424 (Moonshot) very low PPD

Postby Joe_H » Mon Aug 24, 2020 6:39 pm

My understanding is this project has been pulled for now, problems were encountered with many WUs.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Joe_H
Site Admin
 
Posts: 6608
Joined: Tue Apr 21, 2009 5:41 pm
Location: W. MA

Re: Project 13424 (Moonshot) very low PPD

Postby Shirty » Mon Aug 24, 2020 7:31 pm

Cheers, hopefully it'll be back in New and improved form before long!
Shirty
 
Posts: 49
Joined: Thu Jul 11, 2019 11:19 pm

Re: Project 13424 (Moonshot) very low PPD

Postby JohnChodera » Mon Aug 24, 2020 8:05 pm

We had an issue in the preparation and quality control with the latest batch, so we've halted 13424-5 and rolled back to Sprint 2. We think we've fixed the issue and will re-launch tomorrow.

Thanks so much for your patience! These projects have produced incredibly useful data, and despite each week being nearly identical to the last, there always seems to be a new unanticipated challenge that sneaks in somehow!

~ John Chodera // MSKCC
User avatar
JohnChodera
Pande Group Member
 
Posts: 406
Joined: Fri Feb 22, 2013 10:59 pm

Re: Project 13424 (Moonshot) very low PPD

Postby JohnChodera » Tue Aug 25, 2020 4:14 pm

Just an update on this: We've identified the issue here: A nearly co-linear bond between old and new atoms connected to a central heavy atom appears to be causing the OpenMM CCMA algorithm some pathological problems. Removing this single constraint speeds up the simulation by multiple times. This issue was likely the source of the significant RUN-to-RUN variation in earlier 134xx Moonshot projects.

We're tracking this issue here, for those that want to follow along:
https://github.com/openmm/openmm/issues/2814

While working on a long-term solution, we will re-prepare the system without constraints and start the sprint running shortly.

Thanks again for your patience!

~ John Chodera // MSKCC
User avatar
JohnChodera
Pande Group Member
 
Posts: 406
Joined: Fri Feb 22, 2013 10:59 pm

Re: Project 13424 (Moonshot) very low PPD

Postby foldingfanmucde » Sat Aug 29, 2020 6:30 pm

Hi,
Observations on my side today, 29th August are:
Only project 13424 WUs seem to be currently being assigned to all my GPUs (NVIDIA and AMD) and PPD down by around 25% compared with e.g. WUs from project 13422 / 13423.
Maybe those symptoms are useful.
BR
1. AMD Ryzen 9 3900X / 64GB RAM / 2x NVIDIA RTX 2080 Ti / Win10 64 Pro / F@H 7.6.13 / ~6-7M PPD /
2. Intel Core i7 4790K / 32GB RAM / 1x AMD RX 5700 XT / Win10 64 Pro / F@H 7.6.13 / ~1-1.5M PPD /
foldingfanmucde
 
Posts: 9
Joined: Sat Aug 01, 2020 11:26 am
Location: Munich, Germany

Re: Project 13424 (Moonshot) very low PPD

Postby Neil-B » Sat Aug 29, 2020 7:07 pm

For the most part 13422/13423 PPD were very high (on my kit between 50%/100% depending on WU) to allow for the occasional low one ... now that the issue with low ones has been resolved I'd have expected PPDs on current sprint to be near normal ... I am seeing PPDs maybe 5% to 10% higher (if that) than I might have expected for my kit on this core so actually seem quite reasonable to me.
1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent, Quadro K420 1GB, FAH 7.6.13
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro, Quadro M1000M 2GB, FAH 7.6.13
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro, GTX 750Ti 2GB, FAH 7.6.13
Neil-B
 
Posts: 1409
Joined: Sun Mar 22, 2020 6:52 pm
Location: UK

Re: Project 13424 (Moonshot) very low PPD

Postby koolpep » Sat Aug 29, 2020 8:33 pm

Confirming @foldingfanmuc.de - PPD down 25-30% with 13424 units compared to 13422 units. Actually base credit and bonus credit is down - processing time similar per WU.
koolpep
 
Posts: 1
Joined: Sat Aug 29, 2020 8:29 pm

Re: Project 13424 (Moonshot) very low PPD

Postby HaloJones » Sat Aug 29, 2020 8:51 pm

13424 is less than 13422 but 13422 was significantly above previous WU for my cards.
HaloJones
 
Posts: 869
Joined: Thu Jul 24, 2008 11:16 am

Re: Project 13424 (Moonshot) very low PPD

Postby PantherX » Sat Aug 29, 2020 8:53 pm

Welcome to the F@H Forum koolpep,

To paraphrase what Neil-B said, the Project 13424/13425 will now be using normal base points instead of the "higher base points" which previous Projects in 134XX series were using. The rational was in the current Project 13424/13425, there isn't much variation from RUN-to-RUN thus, there was no need to compensate for that. The previous Projects in 134XX series had a large variation between RUN-to-RUN so to ensure that the average PPD would be similar, a higher base points was temporarily used to compensate for the very low PPD in some RUNs.
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
User avatar
PantherX
Site Moderator
 
Posts: 6765
Joined: Wed Dec 23, 2009 10:33 am
Location: Land Of The Long White Cloud

Re: Project 13424 (Moonshot) very low PPD

Postby Neil-B » Sat Aug 29, 2020 8:56 pm

Neil-B
 
Posts: 1409
Joined: Sun Mar 22, 2020 6:52 pm
Location: UK

Re: Project 13424 (Moonshot) very low PPD

Postby JohnChodera » Sat Aug 29, 2020 10:18 pm

Just to echo @PantherX above: We've finally fixed the issues that were causing lots of heterogeneity, so we're now using a base credit more consistent with other projects. With previous 134xx projects, you may have been the lucky ones that never hit any slow RUNs---if so, you got extra points! If not, you probably had a more normal amount of PPD. Now that we were able to eliminate that RUN-to-RUN variation, we're back to more or less normal!

~ John Chodera // MSKCC
User avatar
JohnChodera
Pande Group Member
 
Posts: 406
Joined: Fri Feb 22, 2013 10:59 pm

Re: Project 13424 (Moonshot) very low PPD

Postby cine.chris » Sun Aug 30, 2020 3:40 am

All my systems are down today at 82-83% of what I've been sustaining for several weeks. My goal was 300M pts for August which I'll make, but within hours now. User/Donor abandonment is already high, this could accelerate that rate. I'm still seeing normal levels of power consumption & GPU utilization, both a red flag to me that these WU are consuming resources (unlike some others I whined about) at high levels. Points don't cost anyone anything... except disgruntled & frustrated donors that abandon a project as that's the only metric & compensation they see.
cine.chris
 
Posts: 40
Joined: Sun Apr 26, 2020 2:29 pm

Next

Return to Issues with a specific WU

Who is online

Users browsing this forum: No registered users and 2 guests

cron