Project 13414

Moderators: Site Moderators, FAHC Science Team

Project 13414

Postby Curt3g » Mon Jun 22, 2020 6:28 am

Following up on a post from this morning that I may not have put in the right forum (leave it to a new guy :D ).

viewtopic.php?f=74&t=35624&start=15#p338142 (includes log extracts)

The final processing time on Project: 13414 (Run 824, Clone 39, Gen 1) was a little over 13 hours (RTX 2060).

Not experienced enough to know if this is an issue or not.

Cheers,
Curt
Curt3g
 
Posts: 14
Joined: Sun Mar 29, 2020 5:27 pm

Re: Project 13414

Postby bruce » Mon Jun 22, 2020 6:59 am

The preferred deadline is 1.0 days and the final deadline is 2.0 days for that project. 13 hours isn't unreasonable.
bruce
 
Posts: 19830
Joined: Thu Nov 29, 2007 11:13 pm
Location: So. Cal.

Re: Project 13414

Postby Curt3g » Mon Jun 22, 2020 3:36 pm

Project: 13412 (Run 2439, Clone 10, Gen 0) was processed a little bit later last night, with a run time a hair over 4 hrs.

Cheers,
Curt3g
 
Posts: 14
Joined: Sun Mar 29, 2020 5:27 pm

Re: Project 13414

Postby JohnChodera » Mon Jun 22, 2020 3:57 pm

Wow, we were very much aiming for ~4h, so 13h sounds like something weird is going on. We're seeing some heterogeneity in our calculation, and are looking into what is causing this.

Thanks so much!

~ John Chodera // MSKCC
User avatar
JohnChodera
Pande Group Member
 
Posts: 399
Joined: Fri Feb 22, 2013 10:59 pm

Re: Project 13414

Postby Curt3g » Mon Jun 22, 2020 6:15 pm

Thanks John. Picked up another 13414 this morning (Run 255, Clone 83, Gen 0). The estimated time between frames is 2 min 18 sec, so looks to be in the 4 hour range also.

Cheers,
Curt3g
 
Posts: 14
Joined: Sun Mar 29, 2020 5:27 pm

Re: Project 13414

Postby Ichbin3 » Mon Jun 22, 2020 7:13 pm

I got a 13414 recently which was unexpected slow too
13414 (841,39,1)
TPF was 1:31 (other 13414 where 1:08 normaly) on a 2080Ti
Last edited by Ichbin3 on Tue Jun 23, 2020 5:38 am, edited 1 time in total.
Asus Z87-A, i5-4570, RTX 2080Ti@180W, RTX 2080Ti@200W, Mint 19.3
Ichbin3
 
Posts: 74
Joined: Thu May 28, 2020 9:06 am
Location: Germany

Re: Project 13414

Postby kc2lrc » Tue Jun 23, 2020 4:49 am

I'm seeing some interesting performance numbers myself:

- 13415 on GTX 970, Debian 10: 1m 21s TPF, 106K points/day - nvidia-smi says GPU usage is around 46%
- 13415 on GTX 960, Windows 10: 1m 38s TPF, 79K points/day, HWMon shows GPU usage of 47%
- 13414 on GTX 1080, Debian 10: 2m 12s TPF, 1,076K points/day, nvidia-smi says GPU usage is 98%

Haven't seen it hit any of my 750Ti's yet.

Cheers -
Sam
Image
kc2lrc
 
Posts: 21
Joined: Mon Jan 05, 2009 12:10 am

Re: Project 13414

Postby Ichbin3 » Tue Jun 23, 2020 4:29 pm

Just another slow one at my 2080Ti
13414 (1143,7,0)
TPF 1:46 vs 1:08 a normal one
Ichbin3
 
Posts: 74
Joined: Thu May 28, 2020 9:06 am
Location: Germany

Re: Project 13414

Postby cine.chris » Wed Jun 24, 2020 12:03 am

I`ve seen a couple of very low PPD for WU 13414 today, which is why I`m here. Latest is on a 1660Ti, GPU utilization was at 75%.
cine.chris
 
Posts: 40
Joined: Sun Apr 26, 2020 2:29 pm

Re: Project 13414

Postby Joe_H » Wed Jun 24, 2020 12:55 am

cine.chris wrote:I`ve seen a couple of very low PPD for WU 13414 today, which is why I`m here. Latest is on a 1660Ti, GPU utilization was at 75%.


What were the PRCG numbers for the underperforming WUs? They are trying to track down which Runs seem to have this issue and see if they can figure out why.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Joe_H
Site Admin
 
Posts: 6533
Joined: Tue Apr 21, 2009 5:41 pm
Location: W. MA

Re: Project 13414

Postby kc2lrc » Wed Jun 24, 2020 2:48 am

My slugs were:

13415 (R 835, C 51, G 1) - 1m 21s on GTX 970 (Debian 10, Core 2 Duo E8200), reported 210ns/day, GPU usage around 47%
13415 (R 850, C 50, G 0) - 1m 23s on GTX 970, reported 207ns/day, wasn't around to see GPU usage
13415 (R 792, C 51, G 1) - 1m 40s on GTX 960 (Windows 10, i7-4790K), reported 128ns/day, GPU usage around 46%
13413 (R 767, C 3, G 0) - 2m 5s on GTX 960, reported 130ns/day, wasn't around to see GPU usage

No slugs are churning right now, but I'll keep my eyes open when I'm around.

Cheers -
Sam
kc2lrc
 
Posts: 21
Joined: Mon Jan 05, 2009 12:10 am

Re: Project 13414

Postby kc2lrc » Wed Jun 24, 2020 2:52 am

Noticed one more currently running:

13415 (1610, 8, 0) - 1m 44s on GTX 750Ti (Debian 10, Core 2 E8200), nvidia-smi reports GPU usage around 58%

Cheers -
Sam
kc2lrc
 
Posts: 21
Joined: Mon Jan 05, 2009 12:10 am

Re: Project 13414

Postby Curt3g » Wed Jun 24, 2020 4:50 am

And a couple more today (RTX 2060), first looks as expected, second looks slow. And didn't mention in previous posts, running 24/7.

From earlier today: 13414 (Run 1816, Clone 4, Gen 1), TPF at 2 min 10 sec
Currently running: 13414 (Run 1553, Clone 12, Gen 0), TPF at 7 min, 18 sec

Regards,
Curt
Curt3g
 
Posts: 14
Joined: Sun Mar 29, 2020 5:27 pm

Re: Project 13414

Postby uyaem » Wed Jun 24, 2020 9:54 am

Adding to this another Run that seems to be slow (although with sample size = 1, I can't be sure if it's the entire Run):
Project 13414 (Run 879, Clone 26, Gen 0)

I've observed the same behaviour as others 75% GPU load (nVidia 1660 GTX).
Image
CPU: Ryzen 9 3900X (1x21 CPUs) ~ GPU: nVidia GeForce GTX 1660 Super (Asus)
uyaem
 
Posts: 222
Joined: Sat Mar 21, 2020 8:35 pm
Location: Esslingen, Germany

Re: Project 13414

Postby ajm » Wed Jun 24, 2020 10:07 am

Two slow ones more currently running:
13414 (1209, 20, 0) -> 1080 ti, TPF 3 mins 06 secs
13414 (1586, 20, 0) -> 2070 S, TPF 2 mins 40 secs

EDIT: one more, also currently running:
13414 (1618, 12, 0) -> 1050 ti, TPF 8 mins 51 secs

Three others currently running, but with a normal TPF:
13414 (2227, 14, 1) -> 1050 ti, TPF 6 mins 13 secs
13414 (1275, 23, 0) -> 5700 XT, TPF 2 mins 05 secs
13414 (2737, 22, 0) -> 2080 S, TPF 1 mins 22 secs
ajm
 
Posts: 552
Joined: Sat Mar 21, 2020 6:22 am
Location: Lucerne, Switzerland

Next

Return to Issues with a specific WU

Who is online

Users browsing this forum: Google [Bot] and 3 guests

cron