WARNING Do not upgrade to 375/376.xx drivers (for xx<48)

It seems that a lot of GPU problems revolve around specific versions of drivers. Though NVidia has their own support structure, you can often learn from information reported by others who fold.

Moderators: Site Moderators, FAHC Science Team

foldy
Posts: 2061
Joined: Sat Dec 01, 2012 3:43 pm
Hardware configuration: Folding@Home Client 7.6.13 (1 GPU slots)
Windows 7 64bit
Intel Core i5 2500k@4Ghz
Nvidia gtx 1080ti driver 441

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by foldy »

snapshot wrote:If someone can point me to instructions on how to hack the relevant files then I might give it a try.
I did not try this myself but if you want to experiment:
http://null-bin.blogspot.de/2015/08/how ... river.html
But don't blame me if it does not work or Windows refuses to boot anymore.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by bruce »

It's a shame that NV has not fixed their drivers yet. Apparently the 1050 and the 1050 Ti are the only two GPUs which cannot revert to older drivers without some kind of a hack. If you do try foldy's hack, let us know how it went.
red454
Posts: 35
Joined: Sun Apr 22, 2012 1:35 pm
Hardware configuration: ASUS Rampage V Extreme | Core i7-6950X | Corsair 500R | Corsair Vengeance 32GB | Corsair H100 | be quiet Dark Power Pro 1000 | Samsung 1TB SSD | MSI 2070 Super - RTX 3070ti - RTX 4080/4090 | Win10 Pro
Location: Indy
Contact:

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by red454 »

I have to wonder how many times this has happened with earlier drivers and how long it took for Nvidia to fix it?
Image
Kougar
Posts: 61
Joined: Fri Apr 11, 2008 2:39 am
Hardware configuration: Core i7 920 @ 4.3GHz 1.42v (HT on)
Gigabyte GA-X58-UD5 (F10)
3 x 2GB OCZ Platinum 16400MHz 8-8-8-24 1T
EVGA GTX 260 w/ D-Tek Fuzion 2 GFX
ASUS Xonar DX | Cooler Master UCP 1kW
Intel X25-M 80GB SSD | Windows 7 x64
Swiftech Apogee GTZ + MCP655 Pump & Thermochill PA120.3 Radiator
Location: Texas

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by Kougar »

So this is why the 750 Ti I have throws all sorts of crazy Event Log messages?

\Device\Video4
Graphics SM Global Exception on (GPC 0, TPC 4): Physical Multiple Warp Errors
\Device\Video4
Graphics SM Warp Exception on (GPC 0, TPC 4): Out Of Range Address
\Device\Video4
Graphics Exception: ESR 0x505e48=0x11000e 0x505e50=0x4 0x505e44=0xd3eff2 0x505e4c=0x7f

I recognize what warps are and how NVIDIA uses them to move data through the pipeline, pretty insane if NVIDIA still has not fixed and its been going on for over six weeks...
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by bruce »

Kougar: Nobody else has reported that yet -- and it might be useful.

I presume you're running 375.xx on your 75-Ti. Are there any messages in FAH's log or are they only in the Event log?

How about adding whatever details are missing and reporting it on https://forums.geforce.com/

(There'll be a long topic with the driver version in the title.)
Kougar
Posts: 61
Joined: Fri Apr 11, 2008 2:39 am
Hardware configuration: Core i7 920 @ 4.3GHz 1.42v (HT on)
Gigabyte GA-X58-UD5 (F10)
3 x 2GB OCZ Platinum 16400MHz 8-8-8-24 1T
EVGA GTX 260 w/ D-Tek Fuzion 2 GFX
ASUS Xonar DX | Cooler Master UCP 1kW
Intel X25-M 80GB SSD | Windows 7 x64
Swiftech Apogee GTZ + MCP655 Pump & Thermochill PA120.3 Radiator
Location: Texas

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by Kougar »

Huh odd. Yes this first started occurring after I installed the 376.19 drivers. My Titan Black blew itself up so I moved the 750 Ti into this system. In the original system the 750 Ti was using the 372 drivers.

Dunno if you want the reply in its own thread or not. Here's a partial of the log, I found the system had stopped GPU folding because of the number of failed units, so currently I just removed the GPU folding slot entirely. The event log has errors for every single failed WU I believe, always one of the listed three causes.

Code: Select all

03:35:47:WU00:FS01:0x21:*********************** Log Started 2016-12-09T03:35:47Z ***********************
03:35:47:WU00:FS01:0x21:Project: 11710 (Run 0, Clone 2, Gen 66)
03:35:47:WU00:FS01:0x21:Unit: 0x000000668ca304e75810bc06d111539f
03:35:47:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
03:35:47:WU00:FS01:0x21:Machine: 1
03:35:47:WU00:FS01:0x21:Reading tar file core.xml
03:35:47:WU00:FS01:0x21:Reading tar file integrator.xml
03:35:47:WU00:FS01:0x21:Reading tar file state.xml
03:35:47:WU00:FS01:0x21:Reading tar file system.xml
03:35:48:WU00:FS01:0x21:Digital signatures verified
03:35:48:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
03:35:48:WU00:FS01:0x21:Version 0.0.17
03:35:54:WU00:FS01:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
03:35:54:WU00:FS01:0x21:Saving result file logfile_01.txt
03:35:54:WU00:FS01:0x21:Saving result file log.txt
03:35:54:WU00:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
03:35:57:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:35:57:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:11710 run:0 clone:2 gen:66 core:0x21 unit:0x000000668ca304e75810bc06d111539f
03:35:57:WU00:FS01:Uploading 7.00KiB to 140.163.4.231
03:35:57:WU00:FS01:Upload complete
03:35:57:WU00:FS01:Server responded WORK_ACK (400)
03:35:57:WU00:FS01:Cleaning up
03:35:58:WU00:FS01:Assigned to work server 171.67.108.105
03:35:58:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GM107 [GeForce GTX 750 Ti] from 171.67.108.105
03:35:58:WU00:FS01:Downloading 19.29MiB
03:36:00:WU00:FS01:Download complete
03:36:00:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9175 run:9 clone:9 gen:134 core:0x21 unit:0x000000c0ab436c6957b24c286b3f1719
03:36:00:WU00:FS01:Starting
03:36:00:WU00:FS01:Core PID:6680
03:36:00:WU00:FS01:FahCore 0x21 started
03:36:01:WU00:FS01:0x21:*********************** Log Started 2016-12-09T03:36:01Z ***********************
03:36:01:WU00:FS01:0x21:Project: 9175 (Run 9, Clone 9, Gen 134)
03:36:01:WU00:FS01:0x21:Unit: 0x000000c0ab436c6957b24c286b3f1719
03:36:01:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
03:36:01:WU00:FS01:0x21:Machine: 1
03:36:01:WU00:FS01:0x21:Reading tar file core.xml
03:36:01:WU00:FS01:0x21:Reading tar file integrator.xml
03:36:01:WU00:FS01:0x21:Reading tar file state.xml
03:36:01:WU00:FS01:0x21:Reading tar file system.xml
03:36:02:WU00:FS01:0x21:Digital signatures verified
03:36:02:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
03:36:02:WU00:FS01:0x21:Version 0.0.17
03:36:08:WU00:FS01:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
03:36:08:WU00:FS01:0x21:Saving result file logfile_01.txt
03:36:08:WU00:FS01:0x21:Saving result file log.txt
03:36:08:WU00:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
03:36:10:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:36:10:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9175 run:9 clone:9 gen:134 core:0x21 unit:0x000000c0ab436c6957b24c286b3f1719
03:36:10:WU00:FS01:Uploading 7.00KiB to 171.67.108.105
03:36:10:WU00:FS01:Upload complete
03:36:10:WU00:FS01:Server responded WORK_ACK (400)
03:36:10:WU00:FS01:Cleaning up
03:36:11:WU02:FS01:Assigned to work server 171.67.108.105
03:36:11:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GM107 [GeForce GTX 750 Ti] from 171.67.108.105
03:36:11:WU02:FS01:Downloading 19.47MiB
03:36:13:WU02:FS01:Download complete
03:36:13:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:9177 run:9 clone:6 gen:172 core:0x21 unit:0x00000108ab436c6957b24c295de5e08f
03:36:13:WU02:FS01:Starting
03:36:13:WU02:FS01:Core PID:14584
03:36:13:WU02:FS01:FahCore 0x21 started
03:36:14:WU02:FS01:0x21:*********************** Log Started 2016-12-09T03:36:14Z ***********************
03:36:14:WU02:FS01:0x21:Project: 9177 (Run 9, Clone 6, Gen 172)
03:36:14:WU02:FS01:0x21:Unit: 0x00000108ab436c6957b24c295de5e08f
03:36:14:WU02:FS01:0x21:CPU: 0x00000000000000000000000000000000
03:36:14:WU02:FS01:0x21:Machine: 1
03:36:14:WU02:FS01:0x21:Reading tar file core.xml
03:36:14:WU02:FS01:0x21:Reading tar file integrator.xml
03:36:14:WU02:FS01:0x21:Reading tar file state.xml
03:36:14:WU02:FS01:0x21:Reading tar file system.xml
03:36:14:WU02:FS01:0x21:Digital signatures verified
03:36:14:WU02:FS01:0x21:Folding@home GPU Core21 Folding@home Core
03:36:14:WU02:FS01:0x21:Version 0.0.17
03:36:21:WU02:FS01:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
03:36:21:WU02:FS01:0x21:Saving result file logfile_01.txt
03:36:21:WU02:FS01:0x21:Saving result file log.txt
03:36:21:WU02:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
03:36:26:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:36:26:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:9177 run:9 clone:6 gen:172 core:0x21 unit:0x00000108ab436c6957b24c295de5e08f
03:36:26:WU02:FS01:Uploading 7.00KiB to 171.67.108.105
03:36:27:WU02:FS01:Upload complete
03:36:27:WU02:FS01:Server responded WORK_ACK (400)
03:36:27:WU02:FS01:Cleaning up
03:36:27:WU00:FS01:Assigned to work server 171.67.108.104
03:36:27:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GM107 [GeForce GTX 750 Ti] from 171.67.108.104
03:36:27:WU00:FS01:Downloading 80.23MiB
03:36:33:WU00:FS01:Download 79.07%
03:36:34:WU00:FS01:Download complete
03:36:34:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9213 run:44 clone:42 gen:3 core:0x21 unit:0x0000000cab436c685796c11d4b8578a5
03:36:34:WU00:FS01:Starting
03:36:34:WU00:FS01:Core PID:4488
03:36:34:WU00:FS01:FahCore 0x21 started
03:36:35:WU00:FS01:0x21:*********************** Log Started 2016-12-09T03:36:35Z ***********************
03:36:35:WU00:FS01:0x21:Project: 9213 (Run 44, Clone 42, Gen 3)
03:36:35:WU00:FS01:0x21:Unit: 0x0000000cab436c685796c11d4b8578a5
03:36:35:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
03:36:35:WU00:FS01:0x21:Machine: 1
03:36:35:WU00:FS01:0x21:Reading tar file core.xml
03:36:35:WU00:FS01:0x21:Reading tar file integrator.xml
03:36:35:WU00:FS01:0x21:Reading tar file state.xml
03:36:35:WU00:FS01:0x21:Reading tar file system.xml
03:36:37:WU00:FS01:0x21:Digital signatures verified
03:36:37:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
03:36:37:WU00:FS01:0x21:Version 0.0.17
03:36:41:FS01:Paused
03:36:41:FS01:Shutting core down
03:36:41:WU00:FS01:0x21:WARNING:Console control signal 1 on PID 4488
03:36:41:WU00:FS01:0x21:Exiting, please wait. . .
03:37:04:WU00:FS01:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
03:37:04:WU00:FS01:0x21:Saving result file logfile_01.txt
03:37:04:WU00:FS01:0x21:Saving result file log.txt
03:37:04:WU00:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
03:37:07:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by bruce »

Yes, the message exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5) has only been found with the 375.xx/376.xx drivers. You can restart GPU folding if you reinstall an older driver version.
Kougar
Posts: 61
Joined: Fri Apr 11, 2008 2:39 am
Hardware configuration: Core i7 920 @ 4.3GHz 1.42v (HT on)
Gigabyte GA-X58-UD5 (F10)
3 x 2GB OCZ Platinum 16400MHz 8-8-8-24 1T
EVGA GTX 260 w/ D-Tek Fuzion 2 GFX
ASUS Xonar DX | Cooler Master UCP 1kW
Intel X25-M 80GB SSD | Windows 7 x64
Swiftech Apogee GTZ + MCP655 Pump & Thermochill PA120.3 Radiator
Location: Texas

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by Kougar »

Thanks for the help Bruce. I made a regression post in the 376.19 thread.

Quick aside, do any flags give preference for the AVX WUs or are those luck of the draw? Trying to keep my PPD above the EVGA monthly minimum with the Titan out of commission
Last edited by Kougar on Sat Dec 10, 2016 10:38 pm, edited 1 time in total.
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by drougnor »

Personally, I'd love to see a rep of the OpenMM software programmers take on this: http://forums.evga.com/Got-word-from-NV ... 93454.aspx

The explanation makes perfect sense and if it IS a bug that's been surfaced by the driver code, it's in ALL of our best interest for OpenMM to dig into it to fix it, because that's a possible vulnerability that can be leveraged by not so nice coders to make our hardware do not so nice things . . .
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by bruce »

Kougar wrote:... do any flags give preference for the AVX WUs or are those luck of the draw? Trying to keep my PPD above the EVGA monthly minimum with the Titan out of commission
Off Topic, but see viewtopic.php?f=105&t=29273&p=291129#p291129
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by bruce »

drougnor wrote:Personally, I'd love to see a rep of the OpenMM software programmers take on this: http://forums.evga.com/Got-word-from-NV ... 93454.aspx

The explanation makes perfect sense and if it IS a bug that's been surfaced by the driver code, it's in ALL of our best interest for OpenMM to dig into it to fix it, because that's a possible vulnerability that can be leveraged by not so nice coders to make our hardware do not so nice things . . .
True, but it's strange that it appeared only in the 375-6 series of drivers.

Yes, a bug has been found in OpenMM, but fixing it doesn't resolve the issue.:?: In other words (contrary to external appearances) SOMETHING is happening.

At this point they're continuing to coordinate and when there is any concrete information about a fix, we'll hear about it whether it's in OpenMM or in driver 375-6 or both.
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by drougnor »

bruce wrote:
True, but it's strange that it appeared only in the 375-6 series of drivers.
Nah, not strange at all. The way I parse the message from NVidia is "We did some optimization of our code based on expected results and pushed it out in that version that started breaking things. Our 'optimization' brought to light unexpected behavior in the underlying support code of the OpenMM infrastructure."

Basically, NVidia is owning up to making a change in how the CUDA code interfaces with the OpenMM software, but it is causing entirely unexpected results to be returned and are under the belief that there's a synchronization error in the underlying code.

I'm glad to know that there is cooperation between the two teams, but now my curious nerd side has been poked with a sharp stick and wants to know the nitty gritty details, once it all gets hammered out. ;-)
SombraGuerrero
Posts: 118
Joined: Mon Mar 16, 2009 3:06 am

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by SombraGuerrero »

I hope the OpenMM team will see it Nvidia's way. I would hate for Nvidia to have to branch or suppress some of their future Game Ready optimizations just to keep a bunch of non-gaming do-gooders like us happy! On the other hand, if they can control it at the app profile level at compile time, perhaps that isn't even an issue. Of course, this ultimately will mean that core_21 will need to be rebuilt if OpenMM implements a fix.
snapshot
Posts: 132
Joined: Thu Apr 09, 2009 7:25 pm
Location: Wiltshire, UK

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by snapshot »

foldy wrote:
snapshot wrote:If someone can point me to instructions on how to hack the relevant files then I might give it a try.
I did not try this myself but if you want to experiment:
http://null-bin.blogspot.de/2015/08/how ... river.html
But don't blame me if it does not work or Windows refuses to boot anymore.
Thanks for that. It's a bit laptop-specific but I think there are enough clues in there to at least have a look. I'll be using a test PC and taking a fresh Acronis TIH image first....
2N5R
Posts: 14
Joined: Wed Dec 07, 2016 1:31 pm

Re: WARNING ! Do not upgrade to 375/376.xx drivers.

Post by 2N5R »

Anybody checked out the latest driver 376.33?
Post Reply