unknown enum error after finish

It seems that a lot of GPU problems revolve around specific versions of drivers. Though AMD has their own support structure, you can often learn from information reported by others who fold.

Moderators: Site Moderators, FAHC Science Team

mwroggenbuck
Posts: 127
Joined: Tue Mar 24, 2020 12:47 pm

Re: unknown enum error after finish

Post by mwroggenbuck »

Additional information:

During that same time, I had 41 WU that were successful from various projects. 10 of those would have been repeats of failed WU.

My 1 in 5 estimate was pretty darn close. :lol:
mwroggenbuck
Posts: 127
Joined: Tue Mar 24, 2020 12:47 pm

Re: unknown enum error after finish

Post by mwroggenbuck »

One more failure today. Project 13409 again. Other projects worked just fine. I would like to blame 13409, but why does it pass on the second run? This one did (as did all the rest). I'm starting to lose sleep over this... :evil:
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: unknown enum error after finish

Post by Neil-B »

Really daft question and total longshot might be totally irrelevant but do you use bitdefender?
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
mwroggenbuck
Posts: 127
Joined: Tue Mar 24, 2020 12:47 pm

Re: unknown enum error after finish

Post by mwroggenbuck »

Yes, I use bitdefender (the free version). Why do you ask? I have used it for long before this problem started.

I examined my log files some more. All but one time, project 13409 would fail on the first try. Sometimes it would fail part way through with bad WU or some other error. I am not seeing any of these other errors in other projects. My computer really does not like project 13409.

I can remove bitdefender if you have a good reason, but I don't think that is it. I have already exempted my FAH data directory from Bitdefender scans.
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: unknown enum error after finish

Post by Neil-B »

The has been a recent minor rash (2 or 3) of reports about what may be false positives with this another folder reported same error and traced it to a bitdefender cloud virus report ... search bitdefender and look for oracle1 thread which report was added to ... sorry on phone so linking is a pain
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: unknown enum error after finish

Post by Neil-B »

There may be other issues you are having for other reasons but the completing but not properly uploading is similar enough that I mention it ... iirc it was the server bitdefender didn't like rather than any file

Sorry I was wrong the report is similar ... the one with the BD link is the third most recent topic in the specific server issues forum

Servers change, projects can have different characteristics, heuristics can evolve and white lists can be slow to adjust for new false positives ... what has worked can sometimes start to have issues ... not saying this is what is causing your issues but it may be worth considering/looking into

Hope whatever it is gets sorted as I understand how annoying such must be
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
mwroggenbuck
Posts: 127
Joined: Tue Mar 24, 2020 12:47 pm

Re: unknown enum error after finish

Post by mwroggenbuck »

No need to apologize. I will entertain any idea at this time! And the AntiVirus is a good place to start.

I actually did have problems with McAffe and FAH. I could not exempt an entire directory from a scan, and my Windows Event log was showing files being locked. I had a series of threads in this forum a month or so ago about that. The failure at the end of the run was the same as I am talking about here. However, the problem went away when I removed McAffe.

I will not rule out BitDefender, but I think it is very low probability. I have not created a statistics experiment in a few decades, but it is pretty clear to me that project 13409 has a very abnormal failure rate compared to other projects on my machine.

Oh well, my issues are minor. I just hope we all help find a solution to the virus 19 problem. That would really be something else....
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: unknown enum error after finish

Post by Neil-B »

To true ... if it is some weird BD false positive on web connectivity then hopefully they will shortlist and at least those errors will go away :)

Stay Healthy :)
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
mwroggenbuck
Posts: 127
Joined: Tue Mar 24, 2020 12:47 pm

Re: unknown enum error after finish

Post by mwroggenbuck »

Hello all,

I thought I would update this a little. I had a 13049 project that failed 3 times before I killed it. After that I never received another 13409 (I don't know if the server recognized that I was having problems with that WU). Things worked great until this morning. A project (not 13409) failed in the same way--gets to 100%, says finished, then gives unknown error before upload. HOWEVER, this time is was not the ntdll.dll that failed. It was amdocl64.dll, which is an opencl dll from AMD. So something was different this time.

Still, if I only fail once a week, I can tolerate that. I just thought I would update the information here.

This is the windows error log. Maybe it will be of use.

Code: Select all

Faulting application name: FahCore_22.exe, version: 0.0.0.0, time stamp: 0x5e9fcb57
Faulting module name: amdocl64.dll, version: 10.0.3075.12, time stamp: 0x5ecc0fe0
Exception code: 0xc0000409
Fault offset: 0x0000000001a4b649
Faulting process id: 0x2364
Faulting application start time: 0x01d642201d36397f
Faulting application path: D:\C_Alt\data\FAHClient\cores\cores.foldingathome.org\v7\win\64bit\Core_22.fah\FahCore_22.exe
Faulting module path: C:\windows\System32\DriverStore\FileRepository\u0355166.inf_amd64_b850e0f0c3bce936\B355483\amdocl64.dll
Report Id: f2f5e7f6-5aa2-4d5c-909c-a60fad9f1d42
Faulting package full name: 
Faulting package-relative application ID: 
Joe_H
Site Admin
Posts: 7856
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: unknown enum error after finish

Post by Joe_H »

What is the version number shown in the log for Core_22? The project number could matter, there are some that are beta testing a new version of Core_22, it is possible a setting was incorrect and one of these was assigned to you.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
mwroggenbuck
Posts: 127
Joined: Tue Mar 24, 2020 12:47 pm

Re: unknown enum error after finish

Post by mwroggenbuck »

version of core is 0.0.5

project was 11760
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: unknown enum error after finish

Post by bruce »

Thanks for the report. There are several provisional version between 0.0.5 and whatever version reaches final release and fixing the error and the infinite loop were both on the list of things that needed to be fixed.
mwroggenbuck
Posts: 127
Joined: Tue Mar 24, 2020 12:47 pm

Re: unknown enum error after finish

Post by mwroggenbuck »

Even with the new 22 core, this is still happening (twice in the last 24 hours).

I am not using the default location for the FAH program or data. Could that be a problem? I also only have a GPU slot (no cpu slot). Could that be an issue?

I am just brainstorming here...
HaloJones
Posts: 920
Joined: Thu Jul 24, 2008 10:16 am

Re: unknown enum error after finish

Post by HaloJones »

If a user is not doing the default installation it's always going to be difficult to diagnose. Why are you not using the default locations? My only suggestion is to do a default install, consider disabling all anti-virus/firewall apart from the default Microsoft options. With a gpu, de-install all gpu drivers with DU, then install the latest drivers. Do this before re-installing FAH without changing what it wants to do.
single 1070

Image
mwroggenbuck
Posts: 127
Joined: Tue Mar 24, 2020 12:47 pm

Re: unknown enum error after finish

Post by mwroggenbuck »

Hello all,

I may have found the problem and a work around. I will need more time to be sure, but my last 4 runs were fine.

In my experience, when a program dies with an intermittent unknown error near the exit of the process, it is usually a treading issue. Or things are not destructed properly.

Based on that theory, I did the following:

1) I bumped up the priority of the actual 22 core. Setting this value in the advanced control panel would only change the priority of the wrapper. I set the actual core. As the priority is still below normal, I doubt this is doing much, but I wanted to try it.
2) I also isolated the affinity of the actual 22 core to one cpu. I saw there were 21 threads associated with that process. Locking them to one cpu will prevent these threads from stomping on each other.

This has had the following effects:

1) I have not seen the unknown error (although I really need some more run time).
2) The utilization of the GPU is a little less (sometimes falling to 0 utilization), but mostly good. I can accept less utilization if I can be sure of a result that gets back to the server. Not having the GPU fully loaded might also be a reason my system has not generated the error so far.

I am not recommending that any user actually make these changes. I need more time, and this problem seems to be isolated to me for some reason. I just wanted to mention it here to see if this information makes the people who maintain the cores think of something.

If this continues to work, I will write myself a little program that wakes up every 1/2 hour or so, iterates through the running programs, and changes the priority and affinity of FAH core programs.

Of course, with my luck (since I am writing message and making it public), everything will fail now, and I will be back to square one. :roll:

I will keep the forum informed.

==============================

Update. I wrote a program that checks the log file every three minutes. Once the process gets to 95 percent, I then change the process properties. This lets me get most of the job done using all CPUs, as opposed to changing sometime on the first half hour.
Last edited by mwroggenbuck on Fri Jun 26, 2020 1:39 pm, edited 1 time in total.
Locked