FAH config doesn't take effect until FAHClient is restarted

Moderators: Site Moderators, FAHC Science Team

Post Reply
lknl
Posts: 26
Joined: Sat Dec 31, 2016 10:10 am

FAH config doesn't take effect until FAHClient is restarted

Post by lknl »

I have set FAHClient to connect to the internet via proxy (because my country's firewall makes the connections to FAH servers very slow).
Because sometimes the proxy server is offline or slow, I frequently change the proxy.
Below is FAH configuration:

Code: Select all

01:25:32:<config>
01:25:32:  <!-- Folding Core -->
01:25:32:  <checkpoint v='30'/>
01:25:32:
01:25:32:  <!-- Folding Slot Configuration -->
01:25:32:  <disable-viz v='true'/>
01:25:32:
01:25:32:  <!-- GUI -->
01:25:32:  <gui-enabled v='false'/>
01:25:32:
01:25:32:  <!-- Network -->
01:25:32:  <proxy v='144.217.51.236:8080'/>
01:25:32:  <proxy-enable v='true'/>
01:25:32:
01:25:32:  <!-- Slot Control -->
01:25:32:  <power v='FULL'/>
01:25:32:
01:25:32:  <!-- User Information -->
01:25:32:  <passkey v='********************************'/>
01:25:32:  <team v='edited'/>
01:25:32:  <user v='edited'/>
01:25:32:
01:25:32:  <!-- Folding Slots -->
01:25:32:  <slot id='0' type='CPU'>
01:25:32:    <next-unit-percentage v='100'/>
01:25:32:  </slot>
01:25:32:  <slot id='1' type='GPU'>
01:25:32:    <next-unit-percentage v='100'/>
01:25:32:    <paused v='true'/>
01:25:32:  </slot>
01:25:32:</config>
Full log: http://pastebin.com/raw/ZwX6SLkG (too long to post directly here)
Below are the commands I use to change proxy:

Code: Select all

FAHClient.exe  --send-command "option proxy %new_proxy%"
FAHClient.exe  --send-command "option proxy-enable true"
The problem I have is: Although I have switched to other proxies or disabled the proxy as shown in the log, the old proxy 144.217.51.236:8080 keeps sticking to all new FAH connections.

Did I do something wrong?

Edit: solved. this is a known issue as mentioned by bruce here viewtopic.php?p=302569#p302569
Last edited by lknl on Thu Aug 09, 2018 4:04 pm, edited 4 times in total.
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: FAH Configuration: New proxy doesn't take effect

Post by SteveWillis »

Have you tried restarting the fah client? (easiest but not fastest way is to reboot). Just a thought as I don't use a proxy.
Image

1080 and 1080TI GPUs on Linux Mint
lknl
Posts: 26
Joined: Sat Dec 31, 2016 10:10 am

Re: FAH Configuration: New proxy doesn't take effect

Post by lknl »

SteveWillis wrote:Have you tried restarting the fah client? (easiest but not fastest way is to reboot). Just a thought as I don't use a proxy.
You are correct. I have just checked the old logs and found out that the new proxy setting didn't take effect untill I restarted my PC.
I have changed the topic title.
Is there any way to apply the settings without restarting FAHClient?
I change proxy frequently, so it's a bit troublesome to pause, quit and restart FAHClient eveytime.
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: FAH config doesn't take effect until FAHClient is restar

Post by bruce »

Some settings are applied immediately and others are only applied during startup.

Unfortunately, networking is initialized during startup and is not restarted until FAHClient, itself, is restarted. :(
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: FAH config doesn't take effect until FAHClient is restar

Post by bruce »

Getting a new feature added generally takes a long time, but I did open a ticket for it. https://github.com/FoldingAtHome/fah-issues/issues/1254
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: FAH config doesn't take effect until FAHClient is restar

Post by SteveWillis »

If you were on Linux I could give you my script that restarts the client without rebooting but I don't know how to do it on windows. I assume it can be done though.
Image

1080 and 1080TI GPUs on Linux Mint
lknl
Posts: 26
Joined: Sat Dec 31, 2016 10:10 am

Re: FAH config doesn't take effect until FAHClient is restar

Post by lknl »

bruce wrote:Some settings are applied immediately and others are only applied during startup.

Unfortunately, networking is initialized during startup and is not restarted until FAHClient, itself, is restarted. :(

Getting a new feature added generally takes a long time, but I did open a ticket for it. https://github.com/FoldingAtHome/fah-issues/issues/1254
thanks for clarification and the follow-up action. this feature is not frequently used, so i don't expect to see it soon.
SteveWillis wrote:If you were on Linux I could give you my script that restarts the client without rebooting but I don't know how to do it on windows. I assume it can be done though.
would you mind to share here so that other Linux users can benefit from it? i may get some ideas from it also.
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: FAH config doesn't take effect until FAHClient is restar

Post by SteveWillis »

Sure. The reason it repeats as that occasionally it takes more than one iteration but usually not. Usually on the first pass of the loop and almost always by the second.
#!/bin/ksh
# first install ksh sudo apt-get install ksh
if ! [ $(id -u) = 0 ]; then
echo "This script must be run as root (sudo path/restartclient.sh)"
exit 1
fi
set -x
for i in {1..5}
do
systemctl stop FAHClient || true
sleep 5
[[ $(pgrep -c FahCore) -gt 0 ]] && pkill -e -9 FahCore
[[ $(pgrep -c FAHClient) -gt 0 ]] && pkill -e -9 FAHClient
sleep 5
systemctl restart FAHClient || true
sleep 5
running=$(/etc/init.d/FAHClient status|grep -c "fahclient is running")
if [ $running == 1 ] #success
then
break
fi
sleep 10
done

exit
Image

1080 and 1080TI GPUs on Linux Mint
lknl
Posts: 26
Joined: Sat Dec 31, 2016 10:10 am

Re: FAH config doesn't take effect until FAHClient is restar

Post by lknl »

i would suggest to terminate FAHCore gracefully using pause and unpause after FAHClient is restarted, so that you don't lose any unsaved progress, and also to have assembly optimization enabled (fahcore will auto disable optimization in case of improper termination)

Code: Select all

FAHClient --send-pause
Pause a slot or all slots on an already running client.
FAHClient --send-unpause
Unpause a slot or all slots on an already running client.
To terminate FAHClient, I can't find any built-in command, so I use taskkill /IM /F FAHClient.exe
there are some other commands that i haven't tried

Code: Select all

Process Control:
  pid <boolean=false>
    Create PID file.

  pid-file <string=Folding@home Client.pid>
    Name of PID file.

  respawn <boolean=false>
    Run the application as a child process and respawn if it is killed or exits.
Joe_H
Site Admin
Posts: 7856
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: FAH config doesn't take effect until FAHClient is restar

Post by Joe_H »

lknl wrote:i would suggest to terminate FAHCore gracefully using pause and unpause after FAHClient is restarted, so that you don't lose any unsaved progress, and also to have assembly optimization enabled (fahcore will auto disable optimization in case of improper termination)
Pausing is preferable, but does not make much difference in how much progress is saved except if you happen to want to stop the core just as it is writing a checkpoint. Then there is the possibility of writing out an incomplete or corrupted checkpoint if the core process is just killed.

Checkpoint frequency for the GPU core is set by the project, pausing the core does not cause it to write out a new one. For the CPU cores it depends on whether you have an A4 or A7 project. The A4 core writes out a checkpoint at the interval set in FAHControl defaulting to every 15 minutes. The A7 core also defaults to every 15 minutes or uses the value set in FAHControl, but will attempt to write out a checkpoint when paused. That is not a guarantee one will be written though.

As for auto disabling of optimization, that has not been an issue since before the A4 core was released. It might have been even farther back, by that point the CPU core was only going to run on CPU's with SSE instructions. The message was still in the code though, and was eventually removed.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: FAH config doesn't take effect until FAHClient is restar

Post by SteveWillis »

I'm not convinced that progress doesn't get saved when the client is shut down. Any time I've shut down the client by having to do a computer hard shutdown I almost always get some bad work units when I restart. I've shut down and restarted the client with my script many dozens of times and NEVER had any noticeable problem. Besides, the times I've looked at checkpoint intervals for my GPUs they have only been about 5 minutes.
Image

1080 and 1080TI GPUs on Linux Mint
Joe_H
Site Admin
Posts: 7856
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: FAH config doesn't take effect until FAHClient is restar

Post by Joe_H »

The checkpoint intervals for GPU projects are set by the project manager. They are in terms that relate to the number of steps completed, that may or may not match up to 5minutes, and will be longer in time on a slower card. You can set the checkpoint interval in FAHControl all you want, it will not be used by the GPU core.

As for CPU projects, the checkpointing is done just the way I mentioned above. The core will use the interval set, and also on a clean shutdown vs. a hard shutdown the A7 core will attempt to write out a checkpoint. A hard shutdown that happens to coincide with a checkpoint being written or still cached pending being written out to disk will often result in a corrupted checkpoint. Sometimes in that case the WU will be started from the beginning, other times it will be dumped.

Now if there are other things going on that are messing up on your computer and interfere with the normal processing and writing out to disk of resulting data, then that also can lead to WU corruption. Those include things like memory errors, locking up the data buses, and other hardware or software faults. I have also seen this on systems that run out of RAM and have no page or swap space configuration.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: FAH config doesn't take effect until FAHClient is restar

Post by bruce »

In theory you don't need to terminate the FAHCore* processes.

The original design of FAHClient coordinated the shutdown process by tracking the PID and terminating when the partner process was no longer running. I have not done a recent test confirming that that code works, but I have reported several cases where it didn't work correctly. In any case, a running FAHCore will only test for a missing FAHClient at certain intervals -- which might be a while if it waits until the next checkpoint is written.

If you go that direction, please report whatever you learn.

I'm not suggesting that your script is wrong ... only that you might want to do some experimentation and see if there's a better way to do it.
SteveWillis
Posts: 409
Joined: Fri Apr 15, 2016 12:42 am
Hardware configuration: PC 1:
Linux Mint 17.3
three gtx 1080 GPUs One on a powered header
Motherboard = [MB-AM3-AS-SB-990FXR2] qty 1 Asus Sabertooth 990FX(+59.99)
CPU = [CPU-AM3-FX-8320BR] qty 1 AMD FX 8320 Eight Core 3.5GHz(+41.99)

PC2:
Linux Mint 18
Open air case
Motherboard: ASUS Crosshair V Formula-Z AM3+ AMD 990FX SATA 6Gb/s USB 3.0 ATX AMD
AMD FD6300WMHKBOX FX-6300 6-Core Processor Black Edition with Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan
three gtx 1080,
one gtx 1080 TI on a powered header

Re: FAH config doesn't take effect until FAHClient is restar

Post by SteveWillis »

The processes are stopped in my script because that is what is necessary to fix the 99% error, which is when there is a problem with a GPU, without rebooting. I should have removed those steps before posting since that didn't really apply but I was in a hurry and didn't think about it at the time.
Image

1080 and 1080TI GPUs on Linux Mint
Post Reply