Power requirements - GPUs cutting out

A forum for discussing FAH-related hardware choices and info on actual products (not speculation).

Moderator: Site Moderators

Forum rules
Please read the forum rules before posting.
Post Reply
Ripper36
Posts: 60
Joined: Sun Sep 18, 2011 8:55 am

Power requirements - GPUs cutting out

Post by Ripper36 »

I have an annoying problem with my folding rig i72600k @3.70 GHz, with two Gigabyte GTX570OC graphics cards. Using a Coolermaster SilentPro Gold 1000W power supply.

At present I am only folding on the two gpus. Mainly getting Core17 units.

I get recurring crashes which leave no trace in logs or windows events manager - makes me think a power overload is stopping the whole system.

A log of the GPU sensors shows no apparent change and both are operating at about 69 degrees.

If I only fold with one card there are no issues.

Any suggestions about power requirements for the system welcomed. Otherwise it must be a random memory failure on the second card that the various test routines fail to show up.

Thanks
Image
Zagen30
Posts: 823
Joined: Tue Mar 25, 2008 12:45 am
Hardware configuration: Core i7 3770K @3.5 GHz (not folding), 8 GB DDR3 @2133 MHz, 2xGTX 780 @1215 MHz, Windows 7 Pro 64-bit running 7.3.6 w/ 1xSMP, 2xGPU

4P E5-4650 @3.1 GHz, 64 GB DDR3 @1333MHz, Ubuntu Desktop 13.10 64-bit

Re: Power requirements - GPUs cutting out

Post by Zagen30 »

I find it hard to believe that your PSU wouldn't be providing enough power. The system as a whole can't be drawing much more than 600W (even that's probably a sizable overestimate), and Cooler Master is hardly some cheap no-name brand. If you wanted to be sure it wasn't a power overload, you could use a Kill-a-Watt to measure the draw at the wall.

When you say that if you only fold with one card there are no issues, are you always pausing the same card? If you are, you could try pausing the other one and running the one that you normally pause. That would help determine if it is a problem with the second card.
Image
Joe_H
Site Admin
Posts: 7856
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Power requirements - GPUs cutting out

Post by Joe_H »

One other thing to check is if there is sufficient air flow to cool the components on the card away from the temperature sensors, especially on the bottom of the PCB that would be close to the second card.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
jrweiss
Posts: 707
Joined: Tue Dec 04, 2007 6:56 am
Hardware configuration: Ryzen 7 5700G, 22.40.46 VGA driver; 32GB G-Skill Trident DDR4-3200; Samsung 860EVO 1TB Boot SSD; VelociRaptor 1TB; MSI GTX 1050ti, 551.23 studio driver; BeQuiet FM 550 PSU; Lian Li PC-9F; Win11Pro-64, F@H 8.3.5.

[Suspended] Ryzen 7 3700X, MSI X570MPG, 32GB G-Skill Trident Z DDR4-3600; Corsair MP600 M.2 PCIe Gen4 Boot, Samsung 840EVO-250 SSDs; VelociRaptor 1TB, Raptor 150; MSI GTX 1050ti, 526.98 driver; Kingwin Stryker 500 PSU; Lian Li PC-K7B. Win10Pro-64, F@H 8.3.5.
Location: @Home
Contact:

Re: Power requirements - GPUs cutting out

Post by jrweiss »

Is the 12V voltage from the PSU stable? Use CPUid's HWMonitor or similar to ensure you're not getting voltage drops when the second card is loaded; many current GPUs are sensitive to voltage fluctuations. If both GPUs are running from a single cable, try separate cables, since total draw could exceed 36 amps.

Also ensure you have enough airflow from your case intake fans to handle the exhaust from the 6 GPU fans plus the PSU fan plus the CPU exhaust. It's possible the PSU is overheating if it's starved for air, even if the GPUs are OK.
Ryzen 7 5700G, 22.40.46 VGA driver; MSI GTX 1050ti, 551.23 studio driver
Ryzen 7 3700X; MSI GTX 1050ti, 551.23 studio driver [Suspended]
Ripper36
Posts: 60
Joined: Sun Sep 18, 2011 8:55 am

Re: Power requirements - GPUs cutting out

Post by Ripper36 »

Thanks for all these great suggestions.

There's plenty of airflow with a big Coolermaster case, multiple fans - though the air circulation is possibly not so good around PSU so I'll check that.

The GPU-Z sensors show the voltage for the cards completely stable, but CPUID - HW monitor shows 12v voltage varies sometimes from 12.025 to 12.074

- I will investigate further.
Image
jrweiss
Posts: 707
Joined: Tue Dec 04, 2007 6:56 am
Hardware configuration: Ryzen 7 5700G, 22.40.46 VGA driver; 32GB G-Skill Trident DDR4-3200; Samsung 860EVO 1TB Boot SSD; VelociRaptor 1TB; MSI GTX 1050ti, 551.23 studio driver; BeQuiet FM 550 PSU; Lian Li PC-9F; Win11Pro-64, F@H 8.3.5.

[Suspended] Ryzen 7 3700X, MSI X570MPG, 32GB G-Skill Trident Z DDR4-3600; Corsair MP600 M.2 PCIe Gen4 Boot, Samsung 840EVO-250 SSDs; VelociRaptor 1TB, Raptor 150; MSI GTX 1050ti, 526.98 driver; Kingwin Stryker 500 PSU; Lian Li PC-K7B. Win10Pro-64, F@H 8.3.5.
Location: @Home
Contact:

Re: Power requirements - GPUs cutting out

Post by jrweiss »

A 0.05V fluctuation is nothing to worry about. However, HWMon probably takes that at the motherboard supply, which has a dedicated cable. That shows the PSU itself is stable.

The actual GPU voltage is not the only concern; the 12V supply to the card must be stable. That's why I asked whether both GPUs are powered by the same cable -- 36 amps is a lot for a single skinny cable...
Ryzen 7 5700G, 22.40.46 VGA driver; MSI GTX 1050ti, 551.23 studio driver
Ryzen 7 3700X; MSI GTX 1050ti, 551.23 studio driver [Suspended]
Ripper36
Posts: 60
Joined: Sun Sep 18, 2011 8:55 am

Re: Power requirements - GPUs cutting out

Post by Ripper36 »

No, each card has its own cable back to the PSU.
Image
jrweiss
Posts: 707
Joined: Tue Dec 04, 2007 6:56 am
Hardware configuration: Ryzen 7 5700G, 22.40.46 VGA driver; 32GB G-Skill Trident DDR4-3200; Samsung 860EVO 1TB Boot SSD; VelociRaptor 1TB; MSI GTX 1050ti, 551.23 studio driver; BeQuiet FM 550 PSU; Lian Li PC-9F; Win11Pro-64, F@H 8.3.5.

[Suspended] Ryzen 7 3700X, MSI X570MPG, 32GB G-Skill Trident Z DDR4-3600; Corsair MP600 M.2 PCIe Gen4 Boot, Samsung 840EVO-250 SSDs; VelociRaptor 1TB, Raptor 150; MSI GTX 1050ti, 526.98 driver; Kingwin Stryker 500 PSU; Lian Li PC-K7B. Win10Pro-64, F@H 8.3.5.
Location: @Home
Contact:

Re: Power requirements - GPUs cutting out

Post by jrweiss »

Swap the cables and see if the affected card changes; or simply replace the cable on the affected card if it's modular. It COULD be as simple as a bad cable...
Ryzen 7 5700G, 22.40.46 VGA driver; MSI GTX 1050ti, 551.23 studio driver
Ryzen 7 3700X; MSI GTX 1050ti, 551.23 studio driver [Suspended]
PS3EdOlkkola
Posts: 184
Joined: Tue Aug 26, 2014 9:48 pm
Hardware configuration: 10 SMP folding slots on Intel Phi "Knights Landing" system, configured as 24 CPUs/slot
9 AMD GPU folding slots
31 Nvidia GPU folding slots
50 total folding slots
Average PPD/slot = 459,500
Location: Dallas, TX

Re: Power requirements - GPUs cutting out

Post by PS3EdOlkkola »

The Gigabyte GTX570OC has one 6 pin PCIe connector and one 8 pin PCIe connector (version 1.0), and two 6pin PCIe connectors for version 2.0. The Coolermaster SilentPro Gold 1000W is a semi-modular design with PCIe 6+2 pins x 4 and PCIe 6 pins x 4 connectors. The modular connectors for PCIe support 2 x 6 pin connectors, and the rest of the PCIe connectors are directly wired to the power supply. You said that each card has its own cable back to the power supply, so I'd assume that you're using an 8 pin connector to connect each GPU with the main cable (for version 1, 6 pin for version 2) to the PS and the modular 6 pin connector to each GPU. I would suggest removing the modular 6 pin connectors and use the 6 pin PCIe connectors on the main cable to connect to the GPUs. The PS is a single-rail design and can push 82 amps. Each GTX570OC when not over-clocked pulls 35 amps at full load (benched with FurMark) with two cards, or 85% of the PS capacity, so your PS should easily be able to handle the load of both. If you're overclocking at 15% over stock clocks, power draw jumps to 41 amps, so you're right at the PS capacity. I'd suggest switching the cables as described, and if that does not solve the problem, try substantially underclocking your GPUs by 400MHz to start and slowly increase the clock rate 50 MHz at a time until you find the point of instability. Back off 50 MHz from your unstable setting and all should work.
Image
Hardware config viewtopic.php?f=66&t=17997&p=277235#p277235
Napoleon
Posts: 887
Joined: Wed May 26, 2010 2:31 pm
Hardware configuration: Atom330 (overclocked):
Windows 7 Ultimate 64bit
Intel Atom330 dualcore (4 HyperThreads)
NVidia GT430, core_15 work
2x2GB Kingston KVR1333D3N9K2/4G 1333MHz memory kit
Asus AT3IONT-I Deluxe motherboard
Location: Finland

Re: Power requirements - GPUs cutting out

Post by Napoleon »

PS3EdOlkkola wrote:Each GTX570OC when not over-clocked pulls 35 amps at full load (benched with FurMark) with two cards
12V*(35A + 35A) = 420W + 420W = 840W... seriously? FurMark is vicious, but 840W total? I thought 75W + 2x150W (mobo + 2xPCIE 8pin cables) was the spec'd max for a single card. Well... dual GPU cards might draw a bit more than 375W at full load, but last I checked, GTX570 isn't one of those.

http://www.geforce.com/hardware/desktop ... ifications
http://www.geforce.com/hardware/desktop ... ifications

732MHz vs 607MHz, 219W vs 365W.
jrweiss
Posts: 707
Joined: Tue Dec 04, 2007 6:56 am
Hardware configuration: Ryzen 7 5700G, 22.40.46 VGA driver; 32GB G-Skill Trident DDR4-3200; Samsung 860EVO 1TB Boot SSD; VelociRaptor 1TB; MSI GTX 1050ti, 551.23 studio driver; BeQuiet FM 550 PSU; Lian Li PC-9F; Win11Pro-64, F@H 8.3.5.

[Suspended] Ryzen 7 3700X, MSI X570MPG, 32GB G-Skill Trident Z DDR4-3600; Corsair MP600 M.2 PCIe Gen4 Boot, Samsung 840EVO-250 SSDs; VelociRaptor 1TB, Raptor 150; MSI GTX 1050ti, 526.98 driver; Kingwin Stryker 500 PSU; Lian Li PC-K7B. Win10Pro-64, F@H 8.3.5.
Location: @Home
Contact:

Re: Power requirements - GPUs cutting out

Post by jrweiss »

Yes, TDP of the GTX570 is 219W. A Tom's Hardware review indicated power draw was in line with the TDP. An OC version might draw a bit more, but still in the range of 20 amps vs the 35 amps claimed above.

Still, a bad cable could be a problem with a 20 amp draw...

Have you isolated the problem to a specific slot or card? Have you swapped the cards to opposite slots, still using the same slot to drive the monitor? Have you tried running the monitor from the integrated gfx rather than one of the discrete GPUs? Have you tried Folding only on the card in the second slot?

Also, have you tried backing off to stock clocks on the CPU?

There are a lot of things to try before deciding on a "random memory failure" on one of the cards...
Ryzen 7 5700G, 22.40.46 VGA driver; MSI GTX 1050ti, 551.23 studio driver
Ryzen 7 3700X; MSI GTX 1050ti, 551.23 studio driver [Suspended]
Ripper36
Posts: 60
Joined: Sun Sep 18, 2011 8:55 am

Re: Power requirements - GPUs cutting out

Post by Ripper36 »

Thanks for all this valuable information. I will work through it over the next week!
Image
jrweiss
Posts: 707
Joined: Tue Dec 04, 2007 6:56 am
Hardware configuration: Ryzen 7 5700G, 22.40.46 VGA driver; 32GB G-Skill Trident DDR4-3200; Samsung 860EVO 1TB Boot SSD; VelociRaptor 1TB; MSI GTX 1050ti, 551.23 studio driver; BeQuiet FM 550 PSU; Lian Li PC-9F; Win11Pro-64, F@H 8.3.5.

[Suspended] Ryzen 7 3700X, MSI X570MPG, 32GB G-Skill Trident Z DDR4-3600; Corsair MP600 M.2 PCIe Gen4 Boot, Samsung 840EVO-250 SSDs; VelociRaptor 1TB, Raptor 150; MSI GTX 1050ti, 526.98 driver; Kingwin Stryker 500 PSU; Lian Li PC-K7B. Win10Pro-64, F@H 8.3.5.
Location: @Home
Contact:

Re: Power requirements - GPUs cutting out

Post by jrweiss »

One other thought... I've been reading in various reviews and forums that these GPUs may have throttling protections that may cut in at 70 or 80C, depending on BIOS, driver, and the phase of the moon (or detection of a particular load pattern). If both GPUs level at 69C, you may be running into one. After the other troubleshooting, you might want to research your particular GPU/driver combination and settings.
Ryzen 7 5700G, 22.40.46 VGA driver; MSI GTX 1050ti, 551.23 studio driver
Ryzen 7 3700X; MSI GTX 1050ti, 551.23 studio driver [Suspended]
Post Reply