Pause/Resume gpu core 17 takes 2.5min with high CPU usage

It seems that a lot of GPU problems revolve around specific versions of drivers. Though AMD has their own support structure, you can often learn from information reported by others who fold.

Moderators: Site Moderators, PandeGroup

Pause/Resume gpu core 17 takes 2.5min with high CPU usage

Postby foldy » Fri Dec 27, 2013 9:48 pm

When i pause the client and resume again, then i realize a high cpu usage with gpu core 17 workunits for about 2.5 min (
only one cpu core is used). This was not the case some weeks ago, where on resume high cpu usage was only for some seconds.
foldy
 
Posts: 943
Joined: Sat Dec 01, 2012 3:43 pm

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby 7im » Fri Dec 27, 2013 10:34 pm

It may be the updated core 17. Or it may be which frame number you resumed from. Some core 17 WUs write check points every 2 frames, some every 5.
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
User avatar
7im
 
Posts: 15237
Joined: Thu Nov 29, 2007 4:30 pm
Location: Arizona

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby bruce » Sat Dec 28, 2013 3:22 am

On Linux, the OpenCL drivers reportedly take a full CPU constantly. On Windows, it's only for a few minutes every checkpoint. This seems to be the way AMD designed their new drivers.

Is it possible that you're not comparing the CPU utilization of Core_17 with your memory of previous drivers and/or previous FahCores?
bruce
 
Posts: 21413
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby 7im » Sat Dec 28, 2013 7:14 am

To answer your next question, no, it's not broken. ;)
User avatar
7im
 
Posts: 15237
Joined: Thu Nov 29, 2007 4:30 pm
Location: Arizona

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby foldy » Sat Dec 28, 2013 8:41 am

OK, i checked previous core_17 version and previous driver and it is the same behavior.
So this is how it works and i had a false memory... :oops:
foldy
 
Posts: 943
Joined: Sat Dec 01, 2012 3:43 pm

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby prjindigo » Sat Jan 04, 2014 5:35 pm

Aye, hash and maths check to verify it will run the same way in a predictable manner.
prjindigo
 
Posts: 41
Joined: Wed Mar 30, 2011 7:49 am

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby Eddy » Tue Feb 04, 2014 11:30 am

OpenCL needs a "GPU kernel" to be compiled before it can run, and it takes that CPU time.
I think it's can be optimized because it has to be compiled just 1 time per GPU, and I have seen openCL softwares that saves and re-use the same kernel without recompiling.
Eddy
 
Posts: 5
Joined: Tue Feb 04, 2014 10:31 am

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby Jim Saunders » Wed Feb 05, 2014 2:54 am

It does that for me too if that helps.

Jim
Good science and heat for my basement you say?
Jim Saunders
 
Posts: 52
Joined: Fri Jan 03, 2014 4:53 am

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby 7im » Wed Feb 05, 2014 5:16 am

Eddy wrote:OpenCL needs a "GPU kernel" to be compiled before it can run, and it takes that CPU time.
I think it's can be optimized because it has to be compiled just 1 time per GPU, and I have seen openCL softwares that saves and re-use the same kernel without recompiling.


Have you confirmed this compiling in Fah GPU? Or is this just a guess based on other programs?
User avatar
7im
 
Posts: 15237
Joined: Thu Nov 29, 2007 4:30 pm
Location: Arizona

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby bruce » Wed Feb 05, 2014 6:21 am

Core_17 on Windows does run with high CPU utilization when start a WU. Then it runs with low CPU utilization until it reached the next checkpoint (commonly two 1% frames, but it depends on the project). Then I see high CPU utilization for a period of time again. On linux, it uses one CPU core continuously with no intervening periods of low CPU utilization

Without detailed knowledge of the actual code internal to FahCore_17.exe, allocating the Windows utilization between compiling a kernel, searching the data for inconsistencies, converting/preparing data from one form to another (such a generating data for the viewer) os any number of other possibilities is virtually impossible. I might assume one thing and you might assume something else and neither of us would have the facts required to convince the other of the accuracy of our guess.

I have asked Proteneer (the Developer) if anything can be done to reduce this CPU processing without compromising the results. Or maybe there's some CPU-based code that can be moved to the GPU efficiently. I'm sure they'll take another look at it, but based on external symptoms we can see, there's no assurance that we will notice any changes.

Over the course of a long WU, it seems that the CPU portion of the analysis takes about 5%. of the time (During about 20 hours on my system, 1 CPU hour, which is a lot more than the 2.5 min that you're seeing just at the beginning.)
bruce
 
Posts: 21413
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby Eddy » Wed Feb 05, 2014 11:17 am

7im wrote:Have you confirmed this compiling in Fah GPU? Or is this just a guess based on other programs?
It's just a guess, I've seen programs "compiling" a kernel the first time for looong minutes, then saving it in a cache file (named against gpu core and drivers or something) and never doing that again at least until configuration or parameter changes.

bruce wrote:On Linux, the OpenCL drivers reportedly take a full CPU constantly.
I think (by my linux experience) that it depends on how or which program do you use to check CPU usage. Linux divides CPU percentage in system time (kernel and drivers - gpu drivers too), user time (user programs) I/O wait (time spent waiting for I/O), sometimes also Soft and Hard interrupts time. Take a look at "man vmstat" and ran "vmstat 1" to see which one is it calculating. IO wait is not real cpu calculation, nor it spend power, it's really just waiting, I guess that it's your case.

bruce wrote:I have asked Proteneer (the Developer) if anything can be done to reduce this CPU processing without compromising the results. Or maybe there's some CPU-based code that can be moved to the GPU efficiently. I'm sure they'll take another look at it, but based on external symptoms we can see, there's no assurance that we will notice any changes.
Please ask specially for the OpenCL kernel caching thing, because if a 5% CPU time is acceptable above much time, It's not acceptable in a configuration where the gpu slot is set to start when idle, and paused when the computer is no longer idle.. it can became 100%,200% etc.. of cpu wasted time.
I'll try to figure out and remember which programs did this, but if I'm not wrong, one of them was blender cycles render ... I'm not sure though .. maybe I saw it on some OpenCL benchmarck or something...
Eddy
 
Posts: 5
Joined: Tue Feb 04, 2014 10:31 am

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby foldy » Sun Nov 02, 2014 5:27 pm

In this post by user Napoleon he found a way to pause and resume the gpu core without the 2.5min high cpu usage.
viewtopic.php?f=14&t=26619

Using a process tool you can suspend the FahCore_17.exe process, which means FahControl still sees the gpu core running,
but it isn't, the GPU usage goes to 0%. I then can start a 3D game without conflict.

When i resume the FahCore_17.exe process using a process tool then immidiatly gpu usage goes to 99% again, no extra cpu usage is seen.
So it looks like the OpenCL kernel cache survived in the gpu.

Maybe this is a hint how a fast gpu core pause and resume could be implemented officially.
But still the question: Is it safe to suspend and resume the FahCore_17.exe process :?:
foldy
 
Posts: 943
Joined: Sat Dec 01, 2012 3:43 pm

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby FaaR » Sun Nov 23, 2014 1:18 am

foldy wrote:But still the question: Is it safe to suspend and resume the FahCore_17.exe process :?:

It "should be*", with the caveat that since FaH doesn't know you've paused its GPU core it won't write any checkpoints during the time it is paused, meaning if you shut your PC or FaH down, you'll lose the work done since the last checkpoint save.

*Caveat: I have no idea if FaH periodically challenges its various cores running to see if they're healthy; if you suspend its process, it obviously can't respond to any challenge... FaH might, possibly (speculation) decide the core has crashed and proceed to shut it down/restart it... *shrug* Who knows. :)
FaaR
 
Posts: 72
Joined: Tue Aug 19, 2008 1:32 am

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby Napoleon » Sun Nov 23, 2014 11:12 am

FaaR wrote:*Caveat: I have no idea if FaH periodically challenges its various cores running to see if they're healthy; if you suspend its process, it obviously can't respond to any challenge... FaH might, possibly (speculation) decide the core has crashed and proceed to shut it down/restart it... *shrug* Who knows. :)

Maybe not, see https://fah.stanford.edu/projects/FAHClient/ticket/1059
But that's still just maybe...
User avatar
Napoleon
 
Posts: 1200
Joined: Wed May 26, 2010 2:31 pm
Location: Finland

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Postby bruce » Mon Nov 24, 2014 5:04 pm

If the PC sleeps, contents of video memory are lost while main ram is checkpointed and restored. Images on the screen can be refreshed but fah may hang or crash. The FahCore should be smart enough to restart from the previous ckpt but it's not yet that smart. The same problem happens if use MS rdt.
bruce
 
Posts: 21413
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Next

Return to Problems with AMD/ATI drivers

Who is online

Users browsing this forum: No registered users and 1 guest

cron