BAD_WORK_UNIT (114 = 0x72)

[PantherX]

...I ran a full benchmark on it which stresses it pretty hard and it can that without issue...

Please note that 3D Mark isn't as stressful as folding so it wouldn't be a fair representation.

Use DDU (https://www.guru3d.com/files-details/di ... nload.html) to wipe out the existing Driver installation and then install the latest from Nvidia. I think that it might be related to driver issues.

[Hoovoloo]

It never actually starts folding so not sure the difference in stress makes any difference. I have used DDU and done a clean install several times now and have re-seated the card and I uninstalled FAH and re-installed it after the driver refresh and the problem still persists

I did try running hashcat as it has a benchmark mode (not because i wanted to crack passwords) and that too seems to have an issue, failing with similar errors. They seem to be aware of the possibility of these issues and have a patch that disable the kernel timeouts to avoid it but not sure why they are being hit in the first place. especially as it was running fin until the end of the week. Is there anything other than the drivers that may be impacting on it?

[PantherX]

If hashcat isn't running, then it is likely an OpenCL issue. Do you know what changed, did Windows install an update, did you update an application, etc?

[lafrad]

I've had a bunch of BAD_WORK_UNIT results from project 13405 and 13405.

It happens on both Intel & AMD platforms, on 3 different RTX cards....

All on linux with latest NVIDIA drivers.

[PantherX]

Please note that Projects 13404 and 13405 are highly experimental and while they have a higher error rate, they are still producing scientifically valuable data. That's the only exception as you should very rarely encounter bad WUs.

[lafrad]

13404 has given me 3 out of 8 "BAD_WORK_UNITS",
13405 has given me 1 out of 14 "BAD_WORK_UNITS".

I don't mind if its costing me points, but those error rates sound like it's a pretty inefficient way to get results...

[PantherX]

...I don't mind if its costing me points, but those error rates sound like it's a pretty inefficient way to get results...

In science, "failed" experiments are also very valuable. In this case, scientists are getting results which is still very useful. Think of it like this, trial and error is still a scientific method so even the errors are valuable in this case

[Hoovoloo]

OK: Issue at least partially understood. As the hashcat errors report as a resource issue and involve timeouts I decided to try and run it with the CPU folding paused and it then runs fine so the issue seems to be that it is not now capable of running the CPU and GPU folding concurrently. Maybe a resource management issue as it was managing it fine and it is not exactly a low spec machine

[PantherX]

Considering that you have an AMD Ryzen 9 3900X 12-Core Processor, I am curious as to why that would happen. Can you please post the first 100 lines of the log file without any filters or redaction?

[Hoovoloo]

This is the top of log file. Hard to count the rows but i think that has at least the first 100 . the GPU is paused at this point as it has been causing so many issues

Code: Select all

*********************** Log Started 2020-05-16T10:35:42Z ***********************
10:35:42:Trying to access database...
10:35:42:Successfully acquired database lock
10:35:42:Read GPUs.txt
10:35:42:Enabled folding slot 00: READY cpu:22
10:35:42:Enabled folding slot 01: PAUSED gpu:0:TU104 [GeForce RTX 2070 Super] 8218 (by user)
10:35:43:****************************** FAHClient ******************************
10:35:43:        Version: 7.6.13
10:35:43:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
10:35:43:      Copyright: 2020 foldingathome.org
10:35:43:       Homepage: https://foldingathome.org/
10:35:43:           Date: Apr 27 2020
10:35:43:           Time: 21:21:01
10:35:43:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
10:35:43:         Branch: master
10:35:43:       Compiler: Visual C++ 2008
10:35:43:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
10:35:43:       Platform: win32 10
10:35:43:           Bits: 32
10:35:43:           Mode: Release
10:35:43:         Config: C:\Users\strud\AppData\Roaming\FAHClient\config.xml
10:35:43:******************************** CBang ********************************
10:35:43:           Date: Apr 24 2020
10:35:43:           Time: 17:07:55
10:35:43:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
10:35:43:         Branch: master
10:35:43:       Compiler: Visual C++ 2008
10:35:43:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
10:35:43:       Platform: win32 10
10:35:43:           Bits: 32
10:35:43:           Mode: Release
10:35:43:******************************* System ********************************
10:35:43:            CPU: AMD Ryzen 9 3900X 12-Core Processor
10:35:43:         CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
10:35:43:           CPUs: 24
10:35:43:         Memory: 15.92GiB
10:35:43:    Free Memory: 12.67GiB
10:35:43:        Threads: WINDOWS_THREADS
10:35:43:     OS Version: 6.2
10:35:43:    Has Battery: false
10:35:43:     On Battery: false
10:35:43:     UTC Offset: 1
10:35:43:            PID: 15104
10:35:43:            CWD: C:\Users\strud\AppData\Roaming\FAHClient
10:35:43:  Win32 Service: false
10:35:43:             OS: Windows 10 Home
10:35:43:        OS Arch: AMD64
10:35:43:           GPUs: 1
10:35:43:          GPU 0: Bus:9 Slot:0 Func:0 NVIDIA:8 TU104 [GeForce RTX 2070 Super]
10:35:43:                 8218
10:35:43:  CUDA Device 0: Platform:0 Device:0 Bus:9 Slot:0 Compute:7.5 Driver:10.2
10:35:43:OpenCL Device 0: Platform:0 Device:0 Bus:9 Slot:0 Compute:1.2 Driver:442.92
10:35:43:******************************* libFAH ********************************
10:35:43:           Date: Apr 15 2020
10:35:43:           Time: 14:53:14
10:35:43:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
10:35:43:         Branch: master
10:35:43:       Compiler: Visual C++ 2008
10:35:43:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
10:35:43:       Platform: win32 10
10:35:43:           Bits: 32
10:35:43:           Mode: Release
10:35:43:***********************************************************************
10:35:43:<config>
10:35:43:  <!-- User Information -->
10:35:43:  <passkey v='*****'/>
10:35:43:  <team v='68281'/>
10:35:43:  <user v='Hoovoloo'/>
10:35:43:
10:35:43:  <!-- Folding Slots -->
10:35:43:  <slot id='0' type='CPU'/>
10:35:43:  <slot id='1' type='GPU'>
10:35:43:    <paused v='true'/>
10:35:43:  </slot>


I have found the error that is being generated in the event viewer which is

Code: Select all

Log Name:      System
Source:        nvlddmkm
Date:          16/05/2020 15:39:05
Event ID:      13
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DESKTOP-CDEFV44
Description:
The description for Event ID 13 from source nvlddmkm cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

\Device\Video3
Graphics Exception: ESR 0x51d7b0=0xb0005 0x51d7b4=0x20 0x51d7a8=0x4c1eb72 0x51d7ac=0x174

The message resource is present but the message was not found in the message table

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="nvlddmkm" />
    <EventID Qualifiers="49322">13</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2020-05-16T14:39:05.735063400Z" />
    <EventRecordID>6800</EventRecordID>
    <Channel>System</Channel>
    <Computer>DESKTOP-CDEFV44</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\Video3</Data>
    <Data>Graphics Exception: ESR 0x51d7b0=0xb0005 0x51d7b4=0x20 0x51d7a8=0x4c1eb72 0x51d7ac=0x174</Data>
    <Binary>0000000002003000000000000D00AAC0000000000000000000000000000000000000000000000000</Binary>
  </EventData>
</Event>

Which is a bit odd given that i think this relates to WDDM? and I have TDRLevel set to 0 in the registry

[Hoovoloo]

Code: Select all

The first instance I can see of that nvlddmkm error is on 8/5 at 03:48 which ties in with when the issue started
in the 12 hrs preceding that there are a number of errors

Code: Select all

Log Name:      System
Source:        Microsoft-Windows-DistributedCOM
Date:          07/05/2020 16:52:07
Event ID:      10016
Task Category: None
Level:         Warning
Keywords:      Classic
User:          DESKTOP-CDEFV44\strud
Computer:      DESKTOP-CDEFV44
Description:
The machine-default permission settings do not grant Local Activation permission for the COM Server application with CLSID
{C2F03A33-21F5-47FA-B4BB-156362A2F239}
 and APPID
{316CDED5-E4AE-4B15-9113-7055D84DCC97}
 to the user DESKTOP-CDEFV44\strud SID (S-1-5-21-2999111672-3752298693-3911862395-1001) from address LocalHost (Using LRPC) running in the application container Microsoft.Windows.ShellExperienceHost_10.0.18362.449_neutral_neutral_cw5n1h2txyewy SID (S-1-15-2-155514346-2573954481-755741238-1654018636-1233331829-3075935687-2861478708). This security permission can be modified using the Component Services administrative tool.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-DistributedCOM" Guid="{1B562E86-B7AA-4131-BADC-B6F3A001407E}" EventSourceName="DCOM" />
    <EventID Qualifiers="0">10016</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2020-05-07T15:52:07.910799900Z" />
    <EventRecordID>4221</EventRecordID>
    <Correlation ActivityID="{01827c04-42de-4b87-98cd-ae0da9f9255d}" />
    <Execution ProcessID="1268" ThreadID="17552" />
    <Channel>System</Channel>
    <Computer>DESKTOP-CDEFV44</Computer>
    <Security UserID="S-1-5-21-2999111672-3752298693-3911862395-1001" />
  </System>
  <EventData>
    <Data Name="param1">machine-default</Data>
    <Data Name="param2">Local</Data>
    <Data Name="param3">Activation</Data>
    <Data Name="param4">{C2F03A33-21F5-47FA-B4BB-156362A2F239}</Data>
    <Data Name="param5">{316CDED5-E4AE-4B15-9113-7055D84DCC97}</Data>
    <Data Name="param6">DESKTOP-CDEFV44</Data>
    <Data Name="param7">strud</Data>
    <Data Name="param8">S-1-5-21-2999111672-3752298693-3911862395-1001</Data>
    <Data Name="param9">LocalHost (Using LRPC)</Data>
    <Data Name="param10">Microsoft.Windows.ShellExperienceHost_10.0.18362.449_neutral_neutral_cw5n1h2txyewy</Data>
    <Data Name="param11">S-1-15-2-155514346-2573954481-755741238-1654018636-1233331829-3075935687-2861478708</Data>
  </EventData>
</Event>

Code: Select all

Log Name:      Application
Source:        Microsoft-Windows-AppModel-State
Date:          07/05/2020 22:58:03
Event ID:      12
Task Category: None
Level:         Error
Keywords:     
User:          DESKTOP-CDEFV44\strud
Computer:      DESKTOP-CDEFV44
Description:
Failure to write an application setting for package microsoft.windowscommunicationsapps_8wekyb3d8bbwe. Error Code: -2147023878
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-AppModel-State" Guid="{bff15e13-81bf-45ee-8b16-7cfead00da86}" />
    <EventID>12</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2020-05-07T21:58:03.139097500Z" />
    <EventRecordID>2230</EventRecordID>
    <Correlation />
    <Execution ProcessID="3628" ThreadID="17636" />
    <Channel>Application</Channel>
    <Computer>DESKTOP-CDEFV44</Computer>
    <Security UserID="S-1-5-21-2999111672-3752298693-3911862395-1001" />
  </System>
  <EventData>
    <Data Name="ErrorString">microsoft.windowscommunicationsapps_8wekyb3d8bbwe</Data>
    <Data Name="Error">-2147023878</Data>
  </EventData>
</Event>Log Name:      Application
Source:        Microsoft-Windows-AppModel-State
Date:          07/05/2020 22:58:03
Event ID:      12
Task Category: None
Level:         Error
Keywords:     
User:          DESKTOP-CDEFV44\strud
Computer:      DESKTOP-CDEFV44
Description:
Failure to write an application setting for package microsoft.windowscommunicationsapps_8wekyb3d8bbwe. Error Code: -2147023878
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-AppModel-State" Guid="{bff15e13-81bf-45ee-8b16-7cfead00da86}" />
    <EventID>12</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2020-05-07T21:58:03.139097500Z" />
    <EventRecordID>2230</EventRecordID>
    <Correlation />
    <Execution ProcessID="3628" ThreadID="17636" />
    <Channel>Application</Channel>
    <Computer>DESKTOP-CDEFV44</Computer>
    <Security UserID="S-1-5-21-2999111672-3752298693-3911862395-1001" />
  </System>
  <EventData>
    <Data Name="ErrorString">microsoft.windowscommunicationsapps_8wekyb3d8bbwe</Data>
    <Data Name="Error">-2147023878</Data>
  </EventData>
</Event>

Code: Select all

Log Name:      Microsoft-Windows-AppModel-Runtime/Admin
Source:        Microsoft-Windows-AppModel-Runtime
Date:          08/05/2020 00:03:03
Event ID:      69
Task Category: None
Level:         Error
Keywords:      Process
User:          SYSTEM
Computer:      DESKTOP-CDEFV44
Description:
Failed with 0x490 modifying AppModel Runtime status for package Microsoft.GamingServices_2.40.29002.0_x64__8wekyb3d8bbwe for user DESKTOP-CDEFV44\strud (current status = 0x0, desired status = 0x20).
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-AppModel-Runtime" Guid="{f1ef270a-0d32-4352-ba52-dbab41e1d859}" />
    <EventID>69</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x2000000000000001</Keywords>
    <TimeCreated SystemTime="2020-05-07T23:03:03.876492000Z" />
    <EventRecordID>2050</EventRecordID>
    <Correlation ActivityID="{aa05fb96-23aa-0014-4055-07aaaa23d601}" />
    <Execution ProcessID="13140" ThreadID="9484" />
    <Channel>Microsoft-Windows-AppModel-Runtime/Admin</Channel>
    <Computer>DESKTOP-CDEFV44</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="ErrorCode">1168</Data>
    <Data Name="PackageFullName">Microsoft.GamingServices_2.40.29002.0_x64__8wekyb3d8bbwe</Data>
    <Data Name="User">S-1-5-21-2999111672-3752298693-3911862395-1001</Data>
    <Data Name="DesiredStatus">32</Data>
    <Data Name="CurrentStatus">0</Data>
  </EventData>
</Event>

Code: Select all

Log Name:      Microsoft-Windows-AppModel-Runtime/Admin
Source:        Microsoft-Windows-AppModel-Runtime
Date:          08/05/2020 00:03:07
Event ID:      69
Task Category: None
Level:         Error
Keywords:      Process
User:          SYSTEM
Computer:      DESKTOP-CDEFV44
Description:
Failed with 0x490 modifying AppModel Runtime status for package Microsoft.StorePurchaseApp_12005.1001.1.0_x64__8wekyb3d8bbwe for user DESKTOP-CDEFV44\strud (current status = 0x0, desired status = 0x20).
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-AppModel-Runtime" Guid="{f1ef270a-0d32-4352-ba52-dbab41e1d859}" />
    <EventID>69</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x2000000000000001</Keywords>
    <TimeCreated SystemTime="2020-05-07T23:03:07.999923400Z" />
    <EventRecordID>2054</EventRecordID>
    <Correlation ActivityID="{aa05fb96-23aa-0015-35c4-06aaaa23d601}" />
    <Execution ProcessID="13140" ThreadID="18052" />
    <Channel>Microsoft-Windows-AppModel-Runtime/Admin</Channel>
    <Computer>DESKTOP-CDEFV44</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="ErrorCode">1168</Data>
    <Data Name="PackageFullName">Microsoft.StorePurchaseApp_12005.1001.1.0_x64__8wekyb3d8bbwe</Data>
    <Data Name="User">S-1-5-21-2999111672-3752298693-3911862395-1001</Data>
    <Data Name="DesiredStatus">32</Data>
    <Data Name="CurrentStatus">0</Data>
  </EventData>
</Event>

Code: Select all

Log Name:      System
Source:        Microsoft-Windows-WindowsUpdateClient
Date:          08/05/2020 00:03:13
Event ID:      20
Task Category: Windows Update Agent
Level:         Error
Keywords:      Failure,Installation
User:          SYSTEM
Computer:      DESKTOP-CDEFV44
Description:
Installation Failure: Windows failed to install the following update with error 0x80073D02: 9MWPM2CQNLHN-Microsoft.GamingServices.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-WindowsUpdateClient" Guid="{945a8954-c147-4acd-923f-40c45405a658}" />
    <EventID>20</EventID>
    <Version>1</Version>
    <Level>2</Level>
    <Task>1</Task>
    <Opcode>13</Opcode>
    <Keywords>0x8000000000000028</Keywords>
    <TimeCreated SystemTime="2020-05-07T23:03:13.787873800Z" />
    <EventRecordID>4234</EventRecordID>
    <Correlation />
    <Execution ProcessID="4468" ThreadID="9224" />
    <Channel>System</Channel>
    <Computer>DESKTOP-CDEFV44</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="errorCode">0x80073d02</Data>
    <Data Name="updateTitle">9MWPM2CQNLHN-Microsoft.GamingServices</Data>
    <Data Name="updateGuid">{e9631576-8b01-426f-952e-bb2e82da56d0}</Data>
    <Data Name="updateRevisionNumber">1</Data>
    <Data Name="serviceGuid">{855e8a7c-ecb4-4ca3-b045-1dfa50104289}</Data>
  </EventData>
</Event>

Code: Select all

Log Name:      Microsoft-Windows-AppModel-Runtime/Admin
Source:        Microsoft-Windows-AppModel-Runtime
Date:          08/05/2020 00:03:17
Event ID:      69
Task Category: None
Level:         Error
Keywords:      Process
User:          SYSTEM
Computer:      DESKTOP-CDEFV44
Description:
Failed with 0x490 modifying AppModel Runtime status for package Microsoft.WindowsStore_12005.1001.1.0_x64__8wekyb3d8bbwe for user DESKTOP-CDEFV44\strud (current status = 0x0, desired status = 0x20).
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-AppModel-Runtime" Guid="{f1ef270a-0d32-4352-ba52-dbab41e1d859}" />
    <EventID>69</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x2000000000000001</Keywords>
    <TimeCreated SystemTime="2020-05-07T23:03:17.123610700Z" />
    <EventRecordID>2057</EventRecordID>
    <Correlation ActivityID="{aa05fb96-23aa-000f-7b31-06aaaa23d601}" />
    <Execution ProcessID="13140" ThreadID="13372" />
    <Channel>Microsoft-Windows-AppModel-Runtime/Admin</Channel>
    <Computer>DESKTOP-CDEFV44</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="ErrorCode">1168</Data>
    <Data Name="PackageFullName">Microsoft.WindowsStore_12005.1001.1.0_x64__8wekyb3d8bbwe</Data>
    <Data Name="User">S-1-5-21-2999111672-3752298693-3911862395-1001</Data>
    <Data Name="DesiredStatus">32</Data>
    <Data Name="CurrentStatus">0</Data>
  </EventData>
</Event>

Not sure any of them are related?

[uyaem]

There is a newer driver version available, maybe you've got more luck with it.
This is from my log:

Code: Select all

16:16:22:  CUDA Device 0: Platform:0 Device:0 Bus:45 Slot:0 Compute:6.1 Driver:11.0
16:16:22:OpenCL Device 0: Platform:0 Device:0 Bus:45 Slot:0 Compute:1.2 Driver:445.75

[Hoovoloo]

Thanks. The driver I am using is the Studio version of the Games driver that you have. I have tried both of them and get the Studio driver was recommended by Nvidia as a possible solution

[PantherX]

I have been using the Studio version on my GTX 1080 Ti and didn't have any issue:
04:02:34: GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:8 GP102 [GeForce GTX 1080 Ti] 11380
04:02:34: CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:10.2
04:02:34:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:442.19

[Hoovoloo]

Yes it is very odd. As far as I can see it had run fine up until that point. I looked at the previous 12 hours of errors on the basis that it was unlikely to have gone longer than that without finding a valid WU to try running. None of the other errors I have listed above seem related. Will see if I can identify any related changes to the setup in the same time window