GTX 1080 Bricked while folding? :(

It seems that a lot of GPU problems revolve around specific versions of drivers. Though NVidia has their own support structure, you can often learn from information reported by others who fold.

Moderators: Site Moderators, FAHC Science Team

atomey
Posts: 4
Joined: Sun Apr 19, 2020 4:11 pm

GTX 1080 Bricked while folding? :(

Post by atomey »

My main desktop, among other computers, has been folding for CPU and GPU workloads. Early this morning I woke up and heard clicking sounds coming from my computer. After some investigation, I realized my audio interface was looping on and off because of a reboot loop caused by a DPC watch dog violation Blue Screen. The culprit is the Nvidia display driver.

Since around 3 AM this was happening for me. I had my Folding@Home client configured for light workload and there didn't appear to be any cooling issues.

Now I cannot use the 1080 GTX with the Nvidia drivers without the system eventually crashing from system interrupts, which mostly choke the CPU. If I disable the GTX in the device manager and use the integrated graphics, I have no issues.

The GPU still powers on and gives a video signal and even seems to work in 2D sometimes. Once you try to use an Nvidia driver with it, however, the fans go full blast and I can hear some coil whine noises. The GPU still looks brand new and has no visible defects I can see. I highly doubt it was a cooling problem and I'm using a brand new power supply, so I doubt it is a power issue. I reseated the GPU also but no luck.

This is an EVGA 1080 GTX ACX 3.0 GPU if it matters, running Windows 10 Pro 64 bit with an Intel i7 8086k.

The GPU is about 4 years old almost now and I wanted to replace it soon anyways but if I get a 2080 TI I'm concerned Folding@Home will brick my new GPU also... If I can salvage this thing or find the culprit, it would be nice.
HaloJones
Posts: 920
Joined: Thu Jul 24, 2008 10:16 am

Re: GTX 1080 Bricked while folding? :(

Post by HaloJones »

Cards will eventually fail whether it be because they're running FAH or Doom. If the Nvidia driver causes it to die, then you're outta luck. I feel your pain.

Personally, I choose to never use factory cooling solutions and have all my cards water-cooled either with custom or with AIO cooling. That puts my temperatures down in the 40s or at worst very low 50s.

Factory cooling typically puts gpus up at 80C or even higher and that for me is way too hot for extended use which is what is normally the case with FAH.
single 1070

Image
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: GTX 1080 Bricked while folding? :(

Post by Neil-B »

I feel for you with the card issue … but I am slightly confused … you indicate you client was set for light loads … iirc the GPU is paused and doesn't fold when set for light?
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Jan
Posts: 80
Joined: Tue Mar 31, 2020 6:46 pm

Re: GTX 1080 Bricked while folding? :(

Post by Jan »

Neil-B wrote:I feel for you with the card issue … but I am slightly confused … you indicate you client was set for light loads … iirc the GPU is paused and doesn't fold when set for light?
Correct. The GPU probably died while not working for FAH. Still, everything HaloJones said applies.
atomey
Posts: 4
Joined: Sun Apr 19, 2020 4:11 pm

Re: GTX 1080 Bricked while folding? :(

Post by atomey »

HaloJones wrote:Cards will eventually fail whether it be because they're running FAH or Doom. If the Nvidia driver causes it to die, then you're outta luck. I feel your pain.

Personally, I choose to never use factory cooling solutions and have all my cards water-cooled either with custom or with AIO cooling. That puts my temperatures down in the 40s or at worst very low 50s.

Factory cooling typically puts gpus up at 80C or even higher and that for me is way too hot for extended use which is what is normally the case with FAH.
Understood, my 8086k is already water cooled with a Corsair kit with pretty low temps (though I've never had a CPU failure).

Have you had any GPU failures with watercooling? Any suggestions for a 2080 TI? I got a replacement EVGA XC Ultra 2080 TI, but it is air-cooled. I would imagine you can get a separate block to replace the built-in heatsink but perhaps I need to get a different card.

Jan wrote:
Neil-B wrote:I feel for you with the card issue … but I am slightly confused … you indicate you client was set for light loads … iirc the GPU is paused and doesn't fold when set for light?
Correct. The GPU probably died while not working for FAH. Still, everything HaloJones said applies.
Ok, that makes me relieved. I've seen other anecdotes of EVGA 1080s failing after 3-4 years so perhaps that's the lifespan.
Nathan_P
Posts: 1180
Joined: Wed Apr 01, 2009 9:22 pm
Hardware configuration: Asus Z8NA D6C, 2 x5670@3.2 Ghz, , 12gb Ram, GTX 980ti, AX650 PSU, win 10 (daily use)

Asus Z87 WS, Xeon E3-1230L v3, 8gb ram, KFA GTX 1080, EVGA 750ti , AX760 PSU, Mint 18.2 OS

Not currently folding
Asus Z9PE- D8 WS, 2 E5-2665@2.3 Ghz, 16Gb 1.35v Ram, Ubuntu (Fold only)
Asus Z9PA, 2 Ivy 12 core, 16gb Ram, H folding appliance (fold only)
Location: Jersey, Channel islands

Re: GTX 1080 Bricked while folding? :(

Post by Nathan_P »

I've bricked several cards over the years, a GTS 250, A GTX 275/250 combo card, a 1070Ti, 8800GTX, 2 x 1080 to name but a few. I've also bricked a couple of cards that never saw a WU in their life. They fail, sometimes quickly, sometimes they go on for years. Not sure on EVGA's warranty so look into that before spending on a new card. If you do need a new one, get one with a good warranty or at least from a decent retailer that will cover the warranty for you
Image
HaloJones
Posts: 920
Joined: Thu Jul 24, 2008 10:16 am

Re: GTX 1080 Bricked while folding? :(

Post by HaloJones »

https://www.ekwb.com/shop/ek-fc-rtx-208 ... ckel-plexi will fit your GPU but of course you also need tubes, a pump, a reservoir and a radiator. Whereabouts are you?

I have custom cooling on four 1070s and a TitanX but one of the 1070s is cooled with a NZXT G12 and a 280mm Corsair AIO but I wouldn't recommend that for a 2080ti.

Either do full custom which is going to cost you maybe $300 at least or you could set a lower performance profile when folding by using something like Afterburner to set a power limit of maybe 80%?
single 1070

Image
HaloJones
Posts: 920
Joined: Thu Jul 24, 2008 10:16 am

Re: GTX 1080 Bricked while folding? :(

Post by HaloJones »

Oh, and yes, I've lost a lot of cards over the years to folding. A TitanX, at least one 980ti, numerous 970s; most weren't water-cooled although some where. It really all depends on the card's cooling system. I had a 1070ti that I ran with air because it had fantastic cooling and with the fan set to 80% which I still couldn't hear it ran in the low 60s centigrade.
single 1070

Image
kiore
Posts: 931
Joined: Fri Jan 16, 2009 5:45 pm
Location: USA

Re: GTX 1080 Bricked while folding? :(

Post by kiore »

I have lost fans on cards, a high range GTX1080ti hybrid card fan just seized after 8 months of albeit heavy use, entire card replaced by manufacturer with one that lasted considerably longer and still going strong.
For would be RTX 2080ti buyers I regret my twin fan air cooled version, this a very hot card and I find myself in the strange position of power limiting it as a matter of course for folding use to 85% and yes I have extra fans and a roomy case, and yes I tried taking the sides off, it actually works slightly better with sides on and 4 x 140mm case fans going.
Image
i7 7800x RTX 3070 OS= win10. AMD 3700x RTX 2080ti OS= win10 .

Team page: http://www.rationalskepticism.org
atomey
Posts: 4
Joined: Sun Apr 19, 2020 4:11 pm

Re: GTX 1080 Bricked while folding? :(

Post by atomey »

HaloJones wrote:https://www.ekwb.com/shop/ek-fc-rtx-208 ... ckel-plexi will fit your GPU but of course you also need tubes, a pump, a reservoir and a radiator. Whereabouts are you?

I have custom cooling on four 1070s and a TitanX but one of the 1070s is cooled with a NZXT G12 and a 280mm Corsair AIO but I wouldn't recommend that for a 2080ti.

Either do full custom which is going to cost you maybe $300 at least or you could set a lower performance profile when folding by using something like Afterburner to set a power limit of maybe 80%?
That block seems pretty cheap, at least compared to the official EVGA one here: https://www.evga.com/products/product.a ... HC-1189-B1

I'm out in Los Angeles but $300 seems reasonable to protect a $1200 investment or at least extend lifespan. If i get a whole kit with the tubes, pump, reservoir, radiator, I would imagine its interchangeable with other blocks, no?
kiore wrote:I have lost fans on cards, a high range GTX1080ti hybrid card fan just seized after 8 months of albeit heavy use, entire card replaced by manufacturer with one that lasted considerably longer and still going strong.
For would be RTX 2080ti buyers I regret my twin fan air cooled version, this a very hot card and I find myself in the strange position of power limiting it as a matter of course for folding use to 85% and yes I have extra fans and a roomy case, and yes I tried taking the sides off, it actually works slightly better with sides on and 4 x 140mm case fans going.
What are your ambient air temps? That's always a factor too, even with the best air cooling.

The 2080 TI i just got seems to be around 46 C idle and 82C with load
HaloJones
Posts: 920
Joined: Thu Jul 24, 2008 10:16 am

Re: GTX 1080 Bricked while folding? :(

Post by HaloJones »

atomey wrote: That block seems pretty cheap, at least compared to the official EVGA one here: https://www.evga.com/products/product.a ... HC-1189-B1

I'm out in Los Angeles but $300 seems reasonable to protect a $1200 investment or at least extend lifespan. If i get a whole kit with the tubes, pump, reservoir, radiator, I would imagine its interchangeable with other blocks, no?
The block I linked is made by the largest manufacturer of water-cooling parts. EK are very reputable. I don't know where Tustin is in California but the micro center there has

https://www.microcenter.com/product/391 ... -reservoir - $23
https://www.microcenter.com/product/391 ... 3-8-tubing - $29
https://www.microcenter.com/product/513 ... g-radiator - $69
You will need 3/8" ID tubing to work with that pump and 3/8" OD Barbs with clamps for the radiator and the gpu block so four barbs and clamps.

No idea where you can buy that EK gpu block in the US. I'm in London, UK so not very local to you.

The parts listed were to keep the price down for you. I don't necessarily recommend them but they will work.
single 1070

Image
atomey
Posts: 4
Joined: Sun Apr 19, 2020 4:11 pm

Re: GTX 1080 Bricked while folding? :(

Post by atomey »

HaloJones wrote:
atomey wrote: That block seems pretty cheap, at least compared to the official EVGA one here: https://www.evga.com/products/product.a ... HC-1189-B1

I'm out in Los Angeles but $300 seems reasonable to protect a $1200 investment or at least extend lifespan. If i get a whole kit with the tubes, pump, reservoir, radiator, I would imagine its interchangeable with other blocks, no?
The block I linked is made by the largest manufacturer of water-cooling parts. EK are very reputable. I don't know where Tustin is in California but the micro center there has

https://www.microcenter.com/product/391 ... -reservoir - $23
https://www.microcenter.com/product/391 ... 3-8-tubing - $29
https://www.microcenter.com/product/513 ... g-radiator - $69
You will need 3/8" ID tubing to work with that pump and 3/8" OD Barbs with clamps for the radiator and the gpu block so four barbs and clamps.

No idea where you can buy that EK gpu block in the US. I'm in London, UK so not very local to you.

The parts listed were to keep the price down for you. I don't necessarily recommend them but they will work.
Unfortunately no MIcrocenters near LA but I'll order online regardless.

I think I get the idea. I did the configurator on the EK site and ended up with $600+ in parts but I was going aggressive on the build. I think I'll start with something more modest and will definitely use EK. My case already has a 360 radiator unfortunately and only room for another 240 at most.

Anyways thanks for the info!
HaloJones
Posts: 920
Joined: Thu Jul 24, 2008 10:16 am

Re: GTX 1080 Bricked while folding? :(

Post by HaloJones »

The 82C you're hitting is what NVidia think is a good max temp. You can use Afterburner to increase the fan speed to a level that doesn't hurt your ears but improves the cooling.
single 1070

Image
kiore
Posts: 931
Joined: Fri Jan 16, 2009 5:45 pm
Location: USA

Re: GTX 1080 Bricked while folding? :(

Post by kiore »

I understand that NVIDIA may think 82c a reasonable temp, but I prefer to keep temps below 75c to allow for spikes higher that that I have run burning hot cards at 100% fans before but prefer not to do this is my sleeping area.
Image
i7 7800x RTX 3070 OS= win10. AMD 3700x RTX 2080ti OS= win10 .

Team page: http://www.rationalskepticism.org
Darth_Peter_dualxeon
Posts: 51
Joined: Fri Mar 20, 2020 3:13 am
Hardware configuration: EVGA SR-2 motherboard
2x Xeon x5670 CPU
64 GB ECC DDR3
Nvidia RTX 2070

Re: GTX 1080 Bricked while folding? :(

Post by Darth_Peter_dualxeon »

Well, if I were in your place, I'd test that card with a clean freshly installed system and driver and some stress test.
(easy to do if you have a spare disk to fool around.)
it's still not known if the driver or other softwares went stupid or the card is dead.

By the way, if the card is really dead, I'd recommend the gpu type I have right now, an MSI Armor 8G RTX 2070 instead, it's in comparable price range as second-hand GTX 1080. (and if you are lucky, it can be bought second-hand too)
https://www.msi.com/Graphics-card/GeFor ... cification
That has been folding on full load since a month without problem, never goes above 65 Celsius,only 175 watts, and it has a huge cooler and case has good airflow, but it barely fit into the case due to the taller heatsink size)

Or I'd recommend anything with < 200W TDP and 2-3 fans (not the blower fan version, my Tesla C2050 was running at 99C at full load at 250W when used, although that's the industrial version that did survive this)
Post Reply