Hot Loads !!!

If you think it might be a driver problem, see viewforum.php?f=79

Moderators: slegrand, Site Moderators, PandeGroup

Re: Hot Loads !!!

Postby WiSK » Thu Feb 23, 2012 10:59 pm

bruce wrote:If I seem dismissive, it's because Stanford University really has no control over how you choose to overclock your hardware or what fan profile you decide to use. I'm afraid that it's the only good choice available to you right now. Threatening to quit does not lead to a good outcome for anyone.


I'm not threatening to quit, these work units have lead to me switching off my GPU client. I could just have done that and left Stanford none the wiser, but I naïvely assumed that feedback would be appreciated. My experience of this forum is quite negative. However dumb a user is, telling him he's doing it wrong just drives him away. Think of the other users who search and end up here looking for a solution, but then read a site admin being dismissive of the problem. They won't bother to post, and that's your loss.

I'm also not sure why you are mentioning overclocking.
Image
WiSK
 
Posts: 21
Joined: Tue Dec 06, 2011 7:04 am

Re: Hot Loads !!!

Postby MtM » Thu Feb 23, 2012 11:14 pm

Maybe it's safe to assume anyone from a team with 'overclock' or 'extreme' raises the suspicion of running hardware overclocked. I would assume that, and I would first ask this if someone has an issue. You would be surprised how often it turns out that an overclock deemed stable by someone reporting an issue is still the cause, as evident by the issue going away when running under stock clocks.

However, I'm quite sure he only mentioned it as example, and not with the intent to suggest your issue is related to overclocking.

Your feedback is appreciated, and no one wants to dismiss anything you've said, in contrary. Changes come slow around here, someone once compared f@h to an ocean liner, it can not turn on a dime. But all this input is read, and considered, and maybe adjustments are already underway be it directly or indirectly by making people consider an issue as more important then before reading your posts.

I'm sorry your experience has been negative. But maybe the issue has not been with the forum, or the people who answered your questions, maybe the issue also is the questions you're asking or the concerns you're raising. They are not new, they come back every x amount of time. Not everyone sadly does a forum search before asking or making comments, and if those people post or ask things which they should already know the answer to, the answer might not be given as elaborate as one would be able to get through a search.

It takes a lot of time to 'eloquently' give answers, and when the answers have been given before, do you expect everyone to keep giving them in that matter when they have already spend so much of their time giving them before? Everyone here is human, and everyone here wants to help the project or they would not be here. Maybe if you think of that, you'll look for another reason for answers you've been given, other then 'dismissive' or otherwise negative.

Edit: just to make sure, I'm only speaking for myself here so maybe others don't share my opinion.
MtM
 
Posts: 3233
Joined: Fri Jun 27, 2008 2:20 pm
Location: The Netherlands

Re: Hot Loads !!!

Postby MiTuS-2 » Fri Feb 24, 2012 9:36 am

I started receiving these WUs 4 days ago, and the morning i noticed for the first time ever in my log:
Code: Select all
[07:57:08] Working on Protein
[07:57:08] Client config found, loading data.
[07:57:11] CoreStatus = 63 (99)

so after 5 errors in a row my client was shut down.
my GTX 460 it's not overclocked, so it was quite weird for me.
anyway i solved the problem changing my power options.
Apparently the problem was due to the turning off of the display for the energy saving. If my monitor was turned off while receiveing the new WU, the error would come.
now everything's fine
cheers
User avatar
MiTuS-2
 
Posts: 12
Joined: Mon Feb 08, 2010 9:59 pm
Location: Italy

Re: Hot Loads !!!

Postby bruce » Fri Feb 24, 2012 12:19 pm

That's very interesting.

Many months ago I discovered what might be the same sort of thing but I didn't connect it with anything. If my power savings settings had turned off my monitor, sometimes my system would hang and I couldn't wake it up. I switched to the black screensaver and it hasn't hung since. I attributed it to a Microsoft bug but did no real research on it. At that time I didn't notice any changes in the log.

In thinking about it now, it's more likely an NVidia driver bug. Whether or not that's true, it's time to do some more research into the problem.
bruce
Site Admin
 
Posts: 14971
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Hot Loads !!!

Postby MtM » Fri Feb 24, 2012 2:19 pm

I'm going to try to replicate the error in about 10 minutes ( slot will complete a work unit then ), so let me know if I'm doing this wrong.

I've set my display to turn off using power plan settings. My slot will finish a wu in 14 minutes, will download the next wu at 99%. So if this is correct, I should get a failed unit, the one which is downloaded with the screen turned off?

It might be that my screen switched of to late ( was borderline close switching to the laptop ) but my unit started processing normally. I'll leave the powersetting as is so I can verify next time. ( os win7 x64 fahclient v7 7.1.48 fahcore_11 nvidia ( can check version later if needed ).
Last edited by MtM on Fri Feb 24, 2012 2:46 pm, edited 1 time in total.
MtM
 
Posts: 3233
Joined: Fri Jun 27, 2008 2:20 pm
Location: The Netherlands

Re: Hot Loads !!!

Postby *hondo* » Fri Feb 24, 2012 2:26 pm

Well I'm not too happy about it but I've now dropped the GPU usage down to 75% to minimise the heat, I can't honestly figure out why I'm getting 803X because I'm not using the advanced flag.
*hondo*
 
Posts: 127
Joined: Sat Mar 08, 2008 9:50 am
Location: England UK

Re: Hot Loads !!!

Postby antropofob » Fri Feb 24, 2012 3:48 pm

bruce wrote:That's very interesting.
In thinking about it now, it's more likely an NVidia driver bug. Whether or not that's true, it's time to do some more research into the problem.


I went into Sherlock Holmes mode, see here: viewtopic.php?f=67&t=20891#p209051
antropofob
 
Posts: 18
Joined: Mon Aug 22, 2011 8:03 am

Re: Hot Loads !!!

Postby BrainEater » Thu Mar 01, 2012 12:35 am

I'm back from holidays....I've disconnected my UPS , and gotten all 3 580's blazing.

These unit's seem to have gone down a bit in power draw.....I was happy to see my kill-a-watt register 1 Kw ....Hahahaha !!!!

7im : I did'nt know about that link.I will pay more attention.

:lol:
BrainEater
 
Posts: 17
Joined: Sat Mar 28, 2009 3:16 pm

Re: Hot Loads !!!

Postby Calibrator » Thu Mar 01, 2012 4:12 am

Ouch! My dual GTX570's are smoking hot the last couple of weeks! The GPU temps have increased to 75+°C with fan speeds at 80+%! Wow! I'm really pushing out the units but holy crap! I'm closely monitoring it...
Calibrator
 
Posts: 2
Joined: Tue Aug 26, 2008 12:50 am

Re: Hot Loads !!!

Postby GreyWhiskers » Thu Mar 01, 2012 9:34 am

I had reported earlier the challenges I was having with my recent (Oct 2011) i7 2860QM laptop with GTX560M GPU when running the 803x work units. The temp kept spiking into the mid 90's (Celsius) and I was concerned about things like fan performance and overall lifetime issues running the GPU that hard. I had had some instances where the GPU quit, but the Nvidia driver would recover.

Anyway, there had been some discussion about the application of the FAH_GPU_IDLE environment variable. I have been running with FAH_GPU_IDLE=30 for the last week or so - and think that helped the problem a lot:
-- I haven't seen these Afterburner graphics posted before - and thought it was interesting. The GPU usage is normally flat at 99% while folding, but you an see the idle at 30 sawtooths the usage, with a small trough at the bottom. you can also see from the temp plot at the top that GPU temp is varying between about 68 and 72 deg -- which is quite acceptable.
-- at the top of the screen snips is the widget from the CyberPower UPS the laptop is plugged into, reading out wall-plug wattage for both the GPU and the SMP8 WUs. When the GPU usage goes down, the pwr consumption heads down, and the temp goes down. I also hear the laptop's GPU fan cycling its speed too - not a steady high as it was with the temp over 90.

I was getting between 8K and 13K ppd for the 803x work units before applying the env variables, so I think I have room to play with the variable to see how I can set it to keep the temp in the 70s.

Environment variable set: FAH_GPU_IDLE = 30 under Win 7 Home Prem and FAH v7.1.48
GPU core clock underclocked to 742 from default 775

Running PRCG: 8032 (3, 50, 29) on GPU and 8001 (68, 39, 136) on SMP8
GPU performance: 8:46 TPF, 6313 ppd

Image

Image
User avatar
GreyWhiskers
 
Posts: 720
Joined: Mon Oct 25, 2010 5:57 am
Location: Saratoga, California USA

Re: Hot Loads !!!

Postby GreyWhiskers » Fri Mar 02, 2012 8:07 am

Quick follow-up. I was running the production Core 15 version 2.20 (Tue Aug 2 12:06:37 PDT 2011) for the above results.
User avatar
GreyWhiskers
 
Posts: 720
Joined: Mon Oct 25, 2010 5:57 am
Location: Saratoga, California USA

Re: Hot Loads !!!

Postby BrainEater » Wed Mar 07, 2012 12:45 am

I'll update too.

Hotter : yes
more power : yes
Massive PPD increase : YES

From the numbers I'm seeing , you guys have at least 5-10% headroom......Giver ! *

:twisted:

*post a warning for those with sketchy thermal solutions !!
BrainEater
 
Posts: 17
Joined: Sat Mar 28, 2009 3:16 pm

Re: Hot Loads !!!

Postby antropofob » Fri Mar 09, 2012 2:16 pm

@GreyWhiskers:
I am running V7 client and folding on my GTX560, so I was wondering is it possible to use FAH_GPU_IDLE while running V7?
And if yes, do I have to set it as environment variable in Windows or is it possible to just add it as a slot option in the client itself?
Thanks
antropofob
 
Posts: 18
Joined: Mon Aug 22, 2011 8:03 am

Re: Hot Loads !!!

Postby GreyWhiskers » Fri Mar 09, 2012 7:22 pm

antropofob wrote:@GreyWhiskers:
I am running V7 client and folding on my GTX560, so I was wondering is it possible to use FAH_GPU_IDLE while running V7?
And if yes, do I have to set it as environment variable in Windows or is it possible to just add it as a slot option in the client itself?
Thanks


@antropofob. Good questions.

a. Yes, it works in v7. I'm running v7.1.50 now. The env variables affect the FAH Core, which is independent (sort of) of the Client/Console.

b. No FAHControl slot options (yet). This has been proposed as an enhancement to v7, but I don't see it in the v7 ticket log.

c. I'm running Win7 home prem. to get to the env variables, Control panel - SYSTEM. Select ADVANCED SYSTEM SETTINGS from list in upper left pane. Select ENVIRONMENT VARIABLES, then select NEW in upper pane. Enter FAH_GPU_IDLE as the name, and a value. Select OK for everything.

Once you do that, go back to the FAHConsole. Right click on the GPU slot and select pause. After a few seconds the slot will show Paused, and the Work Queue item will show Ready. Then, just right click the slot again and select FOLD. It will start up again.

Since my problem was thermal control, I find MSI afterburner very useful to show you what's going on and what's the resulting temp. Afterburner plot of the GPU usage will show you how it is periodically idling the GPU. I had to play with the value of FAH_GPU_IDLE to find one that allowed enough cooling on the down cycles to allow for the max temp to be appropriately limited. It does work, albeit reducing your ppd count (as Heinlein would say, TANSTAAFL).

Good luck.
User avatar
GreyWhiskers
 
Posts: 720
Joined: Mon Oct 25, 2010 5:57 am
Location: Saratoga, California USA

Re: Hot Loads !!!

Postby antropofob » Fri Mar 09, 2012 7:53 pm

GreyWhiskers wrote:a. Yes, it works in v7. I'm running v7.1.50 now. The env variables affect the FAH Core, which is independent (sort of) of the Client/Console.


Good luck.


Thanks, will come handy when summer heat wave comes...
antropofob
 
Posts: 18
Joined: Mon Aug 22, 2011 8:03 am

PreviousNext

Return to V6 GPU3 beta (including Fermi) OpenMM

Who is online

Users browsing this forum: No registered users and 0 guests