Pause/Resume gpu core 17 takes 2.5min with high CPU usage

It seems that a lot of GPU problems revolve around specific versions of drivers. Though AMD has their own support structure, you can often learn from information reported by others who fold.

Moderators: Site Moderators, FAHC Science Team

foldy
Posts: 2061
Joined: Sat Dec 01, 2012 3:43 pm
Hardware configuration: Folding@Home Client 7.6.13 (1 GPU slots)
Windows 7 64bit
Intel Core i5 2500k@4Ghz
Nvidia gtx 1080ti driver 441

Pause/Resume gpu core 17 takes 2.5min with high CPU usage

Post by foldy »

When i pause the client and resume again, then i realize a high cpu usage with gpu core 17 workunits for about 2.5 min (
only one cpu core is used). This was not the case some weeks ago, where on resume high cpu usage was only for some seconds.
7im
Posts: 10189
Joined: Thu Nov 29, 2007 4:30 pm
Hardware configuration: Intel i7-4770K @ 4.5 GHz, 16 GB DDR3-2133 Corsair Vengence (black/red), EVGA GTX 760 @ 1200 MHz, on an Asus Maximus VI Hero MB (black/red), in a blacked out Antec P280 Tower, with a Xigmatek Night Hawk (black) HSF, Seasonic 760w Platinum (black case, sleeves, wires), 4 SilenX 120mm Case fans with silicon fan gaskets and silicon mounts (all black), a 512GB Samsung SSD (black), and a 2TB Black Western Digital HD (silver/black).
Location: Arizona
Contact:

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by 7im »

It may be the updated core 17. Or it may be which frame number you resumed from. Some core 17 WUs write check points every 2 frames, some every 5.
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by bruce »

On Linux, the OpenCL drivers reportedly take a full CPU constantly. On Windows, it's only for a few minutes every checkpoint. This seems to be the way AMD designed their new drivers.

Is it possible that you're not comparing the CPU utilization of Core_17 with your memory of previous drivers and/or previous FahCores?
7im
Posts: 10189
Joined: Thu Nov 29, 2007 4:30 pm
Hardware configuration: Intel i7-4770K @ 4.5 GHz, 16 GB DDR3-2133 Corsair Vengence (black/red), EVGA GTX 760 @ 1200 MHz, on an Asus Maximus VI Hero MB (black/red), in a blacked out Antec P280 Tower, with a Xigmatek Night Hawk (black) HSF, Seasonic 760w Platinum (black case, sleeves, wires), 4 SilenX 120mm Case fans with silicon fan gaskets and silicon mounts (all black), a 512GB Samsung SSD (black), and a 2TB Black Western Digital HD (silver/black).
Location: Arizona
Contact:

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by 7im »

To answer your next question, no, it's not broken. ;)
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
foldy
Posts: 2061
Joined: Sat Dec 01, 2012 3:43 pm
Hardware configuration: Folding@Home Client 7.6.13 (1 GPU slots)
Windows 7 64bit
Intel Core i5 2500k@4Ghz
Nvidia gtx 1080ti driver 441

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by foldy »

OK, i checked previous core_17 version and previous driver and it is the same behavior.
So this is how it works and i had a false memory... :oops:
prjindigo
Posts: 31
Joined: Wed Mar 30, 2011 7:49 am

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by prjindigo »

Aye, hash and maths check to verify it will run the same way in a predictable manner.
Eddy
Posts: 5
Joined: Tue Feb 04, 2014 10:31 am

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by Eddy »

OpenCL needs a "GPU kernel" to be compiled before it can run, and it takes that CPU time.
I think it's can be optimized because it has to be compiled just 1 time per GPU, and I have seen openCL softwares that saves and re-use the same kernel without recompiling.
Jim Saunders
Posts: 45
Joined: Fri Jan 03, 2014 4:53 am
Hardware configuration: A: i5 + 2 GTX 660
B: i5 + 2 GTX 670
C: i7 + GTX670

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by Jim Saunders »

It does that for me too if that helps.

Jim
Good science and heat for my basement you say?
7im
Posts: 10189
Joined: Thu Nov 29, 2007 4:30 pm
Hardware configuration: Intel i7-4770K @ 4.5 GHz, 16 GB DDR3-2133 Corsair Vengence (black/red), EVGA GTX 760 @ 1200 MHz, on an Asus Maximus VI Hero MB (black/red), in a blacked out Antec P280 Tower, with a Xigmatek Night Hawk (black) HSF, Seasonic 760w Platinum (black case, sleeves, wires), 4 SilenX 120mm Case fans with silicon fan gaskets and silicon mounts (all black), a 512GB Samsung SSD (black), and a 2TB Black Western Digital HD (silver/black).
Location: Arizona
Contact:

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by 7im »

Eddy wrote:OpenCL needs a "GPU kernel" to be compiled before it can run, and it takes that CPU time.
I think it's can be optimized because it has to be compiled just 1 time per GPU, and I have seen openCL softwares that saves and re-use the same kernel without recompiling.
Have you confirmed this compiling in Fah GPU? Or is this just a guess based on other programs?
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by bruce »

Core_17 on Windows does run with high CPU utilization when start a WU. Then it runs with low CPU utilization until it reached the next checkpoint (commonly two 1% frames, but it depends on the project). Then I see high CPU utilization for a period of time again. On linux, it uses one CPU core continuously with no intervening periods of low CPU utilization

Without detailed knowledge of the actual code internal to FahCore_17.exe, allocating the Windows utilization between compiling a kernel, searching the data for inconsistencies, converting/preparing data from one form to another (such a generating data for the viewer) os any number of other possibilities is virtually impossible. I might assume one thing and you might assume something else and neither of us would have the facts required to convince the other of the accuracy of our guess.

I have asked Proteneer (the Developer) if anything can be done to reduce this CPU processing without compromising the results. Or maybe there's some CPU-based code that can be moved to the GPU efficiently. I'm sure they'll take another look at it, but based on external symptoms we can see, there's no assurance that we will notice any changes.

Over the course of a long WU, it seems that the CPU portion of the analysis takes about 5%. of the time (During about 20 hours on my system, 1 CPU hour, which is a lot more than the 2.5 min that you're seeing just at the beginning.)
Eddy
Posts: 5
Joined: Tue Feb 04, 2014 10:31 am

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by Eddy »

7im wrote:Have you confirmed this compiling in Fah GPU? Or is this just a guess based on other programs?
It's just a guess, I've seen programs "compiling" a kernel the first time for looong minutes, then saving it in a cache file (named against gpu core and drivers or something) and never doing that again at least until configuration or parameter changes.
bruce wrote:On Linux, the OpenCL drivers reportedly take a full CPU constantly.
I think (by my linux experience) that it depends on how or which program do you use to check CPU usage. Linux divides CPU percentage in system time (kernel and drivers - gpu drivers too), user time (user programs) I/O wait (time spent waiting for I/O), sometimes also Soft and Hard interrupts time. Take a look at "man vmstat" and ran "vmstat 1" to see which one is it calculating. IO wait is not real cpu calculation, nor it spend power, it's really just waiting, I guess that it's your case.
bruce wrote:I have asked Proteneer (the Developer) if anything can be done to reduce this CPU processing without compromising the results. Or maybe there's some CPU-based code that can be moved to the GPU efficiently. I'm sure they'll take another look at it, but based on external symptoms we can see, there's no assurance that we will notice any changes.
Please ask specially for the OpenCL kernel caching thing, because if a 5% CPU time is acceptable above much time, It's not acceptable in a configuration where the gpu slot is set to start when idle, and paused when the computer is no longer idle.. it can became 100%,200% etc.. of cpu wasted time.
I'll try to figure out and remember which programs did this, but if I'm not wrong, one of them was blender cycles render ... I'm not sure though .. maybe I saw it on some OpenCL benchmarck or something...
foldy
Posts: 2061
Joined: Sat Dec 01, 2012 3:43 pm
Hardware configuration: Folding@Home Client 7.6.13 (1 GPU slots)
Windows 7 64bit
Intel Core i5 2500k@4Ghz
Nvidia gtx 1080ti driver 441

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by foldy »

In this post by user Napoleon he found a way to pause and resume the gpu core without the 2.5min high cpu usage.
viewtopic.php?f=14&t=26619

Using a process tool you can suspend the FahCore_17.exe process, which means FahControl still sees the gpu core running,
but it isn't, the GPU usage goes to 0%. I then can start a 3D game without conflict.

When i resume the FahCore_17.exe process using a process tool then immidiatly gpu usage goes to 99% again, no extra cpu usage is seen.
So it looks like the OpenCL kernel cache survived in the gpu.

Maybe this is a hint how a fast gpu core pause and resume could be implemented officially.
But still the question: Is it safe to suspend and resume the FahCore_17.exe process :?:
FaaR
Posts: 66
Joined: Tue Aug 19, 2008 1:32 am

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by FaaR »

foldy wrote:But still the question: Is it safe to suspend and resume the FahCore_17.exe process :?:
It "should be*", with the caveat that since FaH doesn't know you've paused its GPU core it won't write any checkpoints during the time it is paused, meaning if you shut your PC or FaH down, you'll lose the work done since the last checkpoint save.

*Caveat: I have no idea if FaH periodically challenges its various cores running to see if they're healthy; if you suspend its process, it obviously can't respond to any challenge... FaH might, possibly (speculation) decide the core has crashed and proceed to shut it down/restart it... *shrug* Who knows. :)
Napoleon
Posts: 887
Joined: Wed May 26, 2010 2:31 pm
Hardware configuration: Atom330 (overclocked):
Windows 7 Ultimate 64bit
Intel Atom330 dualcore (4 HyperThreads)
NVidia GT430, core_15 work
2x2GB Kingston KVR1333D3N9K2/4G 1333MHz memory kit
Asus AT3IONT-I Deluxe motherboard
Location: Finland

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by Napoleon »

FaaR wrote:*Caveat: I have no idea if FaH periodically challenges its various cores running to see if they're healthy; if you suspend its process, it obviously can't respond to any challenge... FaH might, possibly (speculation) decide the core has crashed and proceed to shut it down/restart it... *shrug* Who knows. :)
Maybe not, see https://fah.stanford.edu/projects/FAHClient/ticket/1059
But that's still just maybe...
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Pause/Resume gpu core 17 takes 2.5min with high CPU usag

Post by bruce »

If the PC sleeps, contents of video memory are lost while main ram is checkpointed and restored. Images on the screen can be refreshed but fah may hang or crash. The FahCore should be smart enough to restart from the previous ckpt but it's not yet that smart. The same problem happens if use MS rdt.
Post Reply