GPU Slowdowns & Systems Hibernation-type State :(

[braindancer]

Same here. I think this works, as funny as it is.

[braindancer]

This happened to me again, despite the priority setting. One of the clients froze up, that in turn froze up 3 other GPU clients and slowed down the CPU client by a factor of 3.

[Leonardo]

I wish I had advice for you.

[PantherX]

This happened to me again, despite the priority setting. One of the clients froze up, that in turn froze up 3 other GPU clients and slowed down the CPU client by a factor of 3.

You can try out the new WHQL Drivers. Maybe it might improve your situation.

[braindancer]

258.96 is the version I am using right now.

[Leonardo]

Also, in the Nvidia Control Panel, go to 3D Settings\Manage 3D settings\Power management mode and select "Prefer maximum performance."

[braindancer]

Thanks, will try this once I get home. I'm traveling now, and won't risk doing this over VNC. Last time I touched the driver via remote session, it froze up the machine and I lost 3 days of folding until I came back home.

[Leonardo]

Thanks, will try this once I get home. I'm traveling now, and won't risk doing this over VNC. Last time I touched the driver via remote session, it froze up the machine and I lost 3 days of folding until I came back home.

Awful things can happen if you attempt delicate changes over a remote connection with low bandwidth. I learned that the hard way.

Anyway, let us know how it goes when you return and try the changes.

[braindancer]

For the record, the problem was not resolved by changing to "maximum performance". Here's a screenshot of how PPD look like when this happens:


This happens approximately once a day, which basically means that the farm needs to be closely monitored at all times. I am at a loss

[Leonardo]

I had this happen again the other day with a couple GPU2 clients. I restarted the affected computer and the slowdown went away. I have no explanation. Before, as I wrote above, multiple clients on several machines showed the slowdown behavior. At that time, changing the priority to higher worked. Later, when all was running well, I changed priorities back to low. Except for the problem the other day, this has not been a problem in the last few weeks.

I have no explanation other than it's some kind of combination of factors - drivers, client, work unit....who knows?

[braindancer]

The saddest part is that judging by participation in this thread, the problem is extremely rare...

[bruce]

I have no explanation other than it's some kind of combination of factors - drivers, client, work unit....who knows?

Are you monitoring temperature?

[PantherX]

Experiment with downclock/overclocking them and see what happens. I suggest that you use Afterburner 1.6.0 as it works for me without any problems. What happens if you change the offending GPUs to GPU3 BETA Client so that they can process the FahCore_15 WUs? Same results or not?

[Leonardo]

PantherX, in my experience, the slowdown is not related components temperatures. It just happens, seemingly out of the blue.