Rejecting a4 core WUs

[code_glitch]

Apologies if this is in the wrong place. I am aware of this being solved by enabling vsyscall. In this instance, it is not practical to recompile and reboot.

Is there any way to either:
[*] solve the infinite loop that results from a segfault in trying to run a4 core on a system without sysvcall?
[*] not run WUs that would run into this problem in the first place?

[JimboPalmer]

viewtopic.php?f=24&t=26036 < how to tell us enough to help.

I do not know what OS, what bitness, and what version you are using
I do nat know what hardware you have.
I do not know what version of F@H you are using.
I do not know what configuration you have done.

The first 100 lines of your F@H log often have all this info. (The above link can help you post it)

Also include a portion of the log where the problem is obvious.

[code_glitch]

I've read about the previous problems with core 0xa4 and vsyscalls, and I have no doubt that its an easy enough fix for most cases to use emulated mode in the kernel options but in this case, thats an impractical solution.

If you want proof of this being the typical case I've read about multiple times over, heres the first chunk of the log for one of the nodes:

Code: Select all

*********************** Log Started 2018-11-10T17:28:38Z ***********************
17:28:38:************************* Folding@home Client *************************
17:28:38:    Website: https://foldingathome.org/
17:28:38:  Copyright: (c) 2009-2018 foldingathome.org
17:28:38:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:28:38:       Args:
17:28:38:     Config: <none>
17:28:38:******************************** Build ********************************
17:28:38:    Version: 7.5.1
17:28:38:       Date: May 11 2018
17:28:38:       Time: 19:59:04
17:28:38: Repository: Git
17:28:38:   Revision: 4705bf53c635f88b8fe85af7675557e15d491ff0
17:28:38:     Branch: master
17:28:38:   Compiler: GNU 6.3.0 20170516
17:28:38:    Options: -std=gnu++98 -O3 -funroll-loops
17:28:38:   Platform: linux2 4.14.0-3-amd64
17:28:38:       Bits: 64
17:28:38:       Mode: Release
17:28:38:******************************* System ********************************
17:28:38:        CPU: Six-Core AMD Opteron(tm) Processor 2435
17:28:38:     CPU ID: AuthenticAMD Family 16 Model 8 Stepping 0
17:28:38:       CPUs: 12
17:28:38:     Memory: 19.60GiB
17:28:38:Free Memory: 18.76GiB
17:28:38:    Threads: POSIX_THREADS
17:28:38: OS Version: 4.18
17:28:38:Has Battery: false
17:28:38: On Battery: false
17:28:38: UTC Offset: 0
17:28:38:        PID: 6601
17:28:38:        CWD: /home/mpi
17:28:38:         OS: Linux 4.18.12-arch1-1-ARCH x86_64
17:28:38:    OS Arch: AMD64
17:28:38:       GPUs: 1
17:28:38:      GPU 0: Bus:24 Slot:0 Func:0 NVIDIA:1 G94 [GeForce 9600 GT]
17:28:38:       CUDA: Not detected: Failed to open dynamic library 'libcuda.so':
17:28:38:             libcuda.so: cannot open shared object file: No such file or
17:28:38:             directory
17:28:38:     OpenCL: Not detected: Failed to open dynamic library 'libOpenCL.so':
17:28:38:             libOpenCL.so: cannot open shared object file: No such file or
17:28:38:             directory
17:28:38:***********************************************************************
17:28:38:<config>
17:28:38:  <!-- Folding Slots -->
17:28:38:</config>
17:28:38:Connecting to assign1.foldingathome.org:8080
17:28:39:Updated GPUs.txt
17:28:39:Read GPUs.txt
17:28:39:Trying to access database...
17:28:39:Successfully acquired database lock
17:28:39:Enabled folding slot 00: PAUSED cpu:11 (not configured)
17:29:40:Saving configuration to config.xml
17:29:40:<config>
17:29:40:  <!-- Folding Slots -->
17:29:40:  <slot id='0' type='CPU'/>
17:29:40:</config>
17:29:40:Set client configured
17:29:40:WU00:FS00:Connecting to 65.254.110.245:8080
17:29:40:WU00:FS00:Connecting to 65.254.110.245:8080
17:29:41:WU00:FS00:Assigned to work server 128.252.203.2
17:29:41:WU00:FS00:Requesting new work unit for slot 00: READY cpu:11 from 128.252.203.2
17:29:41:WU00:FS00:Connecting to 128.252.203.2:8080
17:29:41:WU00:FS00:Downloading 4.57MiB
17:29:44:WU00:FS00:Download complete
17:29:44:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13812 run:0 clone:1055 gen:291 core:0xa7 unit:0x0000015880fccb025ac12c77c85853ba
17:29:44:WU00:FS00:Downloading core from http://cores.foldingathome.org/Linux/AMD64/Core_a7.fah
17:29:44:WU00:FS00:Connecting to cores.foldingathome.org:80
17:29:44:WU00:FS00:FahCore a7: Downloading 7.64MiB
17:29:47:WU00:FS00:FahCore a7: Download complete
17:29:47:WU00:FS00:Valid core signature
17:29:47:WU00:FS00:Unpacked 18.42MiB to cores/cores.foldingathome.org/Linux/AMD64/Core_a7.fah/FahCore_a7
17:29:48:WU00:FS00:Starting
17:29:48:WARNING:WU00:FS00:AS lowered CPUs from 11 to 10
17:29:48:WU00:FS00:Running FahCore: /opt/fah/FAHCoreWrapper /home/mpi/cores/cores.foldingathome.org/Linux/AMD64/Core_a7.fah/FahCore_a7 -dir 00 -suffix 01 -version 705 -lifeline 6601 -checkpoint 15 -np 10
17:29:48:WU00:FS00:Started FahCore on PID 6610
17:29:48:WU00:FS00:Core PID:6614
17:29:48:WU00:FS00:FahCore 0xa7 started
17:29:48:WU00:FS00:0xa7:*********************** Log Started 2018-11-10T17:29:48Z ***********************
17:29:48:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
17:29:48:WU00:FS00:0xa7:       Type: 0xa7
17:29:48:WU00:FS00:0xa7:       Core: Gromacs
17:29:48:WU00:FS00:0xa7:    Website: https://foldingathome.org/
17:29:48:WU00:FS00:0xa7:  Copyright: (c) 2009-2018 foldingathome.org
17:29:48:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:29:48:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 705 -lifeline 6610 -checkpoint 15 -np
17:29:48:WU00:FS00:0xa7:             10
17:29:48:WU00:FS00:0xa7:     Config: <none>
17:29:48:WU00:FS00:0xa7:************************************ Build *************************************
17:29:48:WU00:FS00:0xa7:    Version: 0.0.17
17:29:48:WU00:FS00:0xa7:       Date: Apr 27 2018
17:29:48:WU00:FS00:0xa7:       Time: 19:09:25
17:29:48:WU00:FS00:0xa7: Repository: Git
17:29:48:WU00:FS00:0xa7:   Revision: 21359963583d09ec2063ef946399441c4df4ccd7
17:29:48:WU00:FS00:0xa7:     Branch: master
17:29:48:WU00:FS00:0xa7:   Compiler: GNU 6.3.0 20170516
17:29:48:WU00:FS00:0xa7:    Options: -std=gnu++98 -O3 -funroll-loops
17:29:48:WU00:FS00:0xa7:   Platform: linux2 4.14.0-3-amd64
17:29:48:WU00:FS00:0xa7:       Bits: 64
17:29:48:WU00:FS00:0xa7:       Mode: Release
17:29:48:WU00:FS00:0xa7:       SIMD: sse2
17:29:48:WU00:FS00:0xa7:************************************ System ************************************
17:29:48:WU00:FS00:0xa7:        CPU: Six-Core AMD Opteron(tm) Processor 2435
17:29:48:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 16 Model 8 Stepping 0
17:29:48:WU00:FS00:0xa7:       CPUs: 12
17:29:48:WU00:FS00:0xa7:     Memory: 19.60GiB
17:29:48:WU00:FS00:0xa7:Free Memory: 18.73GiB
17:29:48:WU00:FS00:0xa7:    Threads: POSIX_THREADS
17:29:48:WU00:FS00:0xa7: OS Version: 4.18
17:29:48:WU00:FS00:0xa7:Has Battery: false
17:29:48:WU00:FS00:0xa7: On Battery: false
17:29:48:WU00:FS00:0xa7: UTC Offset: 0
17:29:48:WU00:FS00:0xa7:        PID: 6614
17:29:48:WU00:FS00:0xa7:        CWD: /home/mpi/work
17:29:48:WU00:FS00:0xa7:         OS: Linux 4.18.12-arch1-1-ARCH x86_64
17:29:48:WU00:FS00:0xa7:    OS Arch: AMD64
17:29:48:WU00:FS00:0xa7:********************************************************************************
17:29:48:WU00:FS00:0xa7:Project: 13812 (Run 0, Clone 1055, Gen 291)
17:29:48:WU00:FS00:0xa7:Unit: 0x0000015880fccb025ac12c77c85853ba
17:29:48:WU00:FS00:0xa7:Reading tar file core.xml
17:29:48:WU00:FS00:0xa7:Reading tar file frame291.tpr
17:29:48:WU00:FS00:0xa7:Digital signatures verified
17:29:48:WU00:FS00:0xa7:Calling: mdrun -s frame291.tpr -o frame291.trr -x frame291.xtc -cpt 15 -nt 10
17:29:48:WU00:FS00:0xa7:Steps: first=72750000 total=250000
17:29:50:WU00:FS00:0xa7:Completed 1 out of 250000 steps (0%)
17:33:23:WU00:FS00:0xa7:Completed 2500 out of 250000 steps (1%)
***SNIP***


Eventually, it will get a WU that uses core 0xa4 and this will cause an infinite loop

Code: Select all

07:18:52:WU02:FS00:Removing old file './work/02/logfile_01-20181111-064652.txt'
07:18:52:WU02:FS00:Running FahCore: /opt/fah/FAHCoreWrapper /home/mpi/cores/cores.foldingathome.org/Linux/AMD64/Core_a4.fah/FahCore_a4 -dir 02 -suffix 01 -version 705 -lifeline 6601 -checkpoint 15 -np 10
07:18:52:WU02:FS00:Started FahCore on PID 13522
07:18:53:WU02:FS00:Core PID:13526
07:18:53:WU02:FS00:FahCore 0xa4 started
07:18:53:WU02:FS00:0xa4:
07:18:53:WU02:FS00:0xa4:*------------------------------*
07:18:53:WU02:FS00:0xa4:Folding@Home Gromacs GB Core
07:18:53:WU02:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
07:18:53:WU02:FS00:0xa4:
07:18:53:WU02:FS00:0xa4:Preparing to commence simulation
07:18:53:WU02:FS00:0xa4:- Ensuring status. Please wait.
07:19:02:WU02:FS00:0xa4:- Looking at optimizations...
07:19:02:WU02:FS00:0xa4:- Working with standard loops on this execution.
07:19:02:WU02:FS00:0xa4:Examination of work files indicates 8 consecutive improper terminations of core.
07:19:02:WU02:FS00:0xa4:- Expanded 826026 -> 1398024 (decompressed 169.2 percent)
07:19:02:WU02:FS00:0xa4:Called DecompressByteArray: compressed_data_size=826026 data_size=1398024, decompressed_data_size=1398024 diff=0
07:19:02:WU02:FS00:0xa4:- Digital signature verified
07:19:02:WU02:FS00:0xa4:
07:19:02:WU02:FS00:0xa4:Project: 9032 (Run 497, Clone 1, Gen 2183)
07:19:02:WU02:FS00:0xa4:
07:19:02:WU02:FS00:0xa4:Entering M.D.
07:19:09:WU02:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
07:19:53:WU02:FS00:Starting
07:19:53:WARNING:WU02:FS00:AS lowered CPUs from 11 to 10
07:19:53:WU02:FS00:Removing old file './work/02/logfile_01-20181111-064752.txt'
07:19:53:WU02:FS00:Running FahCore: /opt/fah/FAHCoreWrapper /home/mpi/cores/cores.foldingathome.org/Linux/AMD64/Core_a4.fah/FahCore_a4 -dir 02 -suffix 01 -version 705 -lifeline 6601 -checkpoint 15 -np 10
07:19:53:WU02:FS00:Started FahCore on PID 13537
07:19:53:WU02:FS00:Core PID:13541
07:19:53:WU02:FS00:FahCore 0xa4 started
07:19:53:WU02:FS00:0xa4:
07:19:53:WU02:FS00:0xa4:*------------------------------*
07:19:53:WU02:FS00:0xa4:Folding@Home Gromacs GB Core
07:19:53:WU02:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
07:19:53:WU02:FS00:0xa4:
07:19:53:WU02:FS00:0xa4:Preparing to commence simulation
07:19:53:WU02:FS00:0xa4:- Ensuring status. Please wait.
07:20:02:WU02:FS00:0xa4:- Looking at optimizations...
07:20:02:WU02:FS00:0xa4:- Working with standard loops on this execution.
07:20:02:WU02:FS00:0xa4:Examination of work files indicates 8 consecutive improper terminations of core.
07:20:02:WU02:FS00:0xa4:- Expanded 826026 -> 1398024 (decompressed 169.2 percent)
07:20:02:WU02:FS00:0xa4:Called DecompressByteArray: compressed_data_size=826026 data_size=1398024, decompressed_data_size=1398024 diff=0
07:20:02:WU02:FS00:0xa4:- Digital signature verified
07:20:02:WU02:FS00:0xa4:
07:20:02:WU02:FS00:0xa4:Project: 9032 (Run 497, Clone 1, Gen 2183)
07:20:02:WU02:FS00:0xa4:
07:20:02:WU02:FS00:0xa4:Entering M.D.
07:20:09:WU02:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
07:20:53:WU02:FS00:Starting
07:20:53:WARNING:WU02:FS00:AS lowered CPUs from 11 to 10
07:20:53:WU02:FS00:Removing old file './work/02/logfile_01-20181111-064852.txt'
07:20:53:WU02:FS00:Running FahCore: /opt/fah/FAHCoreWrapper /home/mpi/cores/cores.foldingathome.org/Linux/AMD64/Core_a4.fah/FahCore_a4 -dir 02 -suffix 01 -version 705 -lifeline 6601 -checkpoint 15 -np 10
07:20:53:WU02:FS00:Started FahCore on PID 13553
07:20:53:WU02:FS00:Core PID:13557
07:20:53:WU02:FS00:FahCore 0xa4 started
07:20:53:WU02:FS00:0xa4:
07:20:53:WU02:FS00:0xa4:*------------------------------*
07:20:53:WU02:FS00:0xa4:Folding@Home Gromacs GB Core
07:20:53:WU02:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
07:20:53:WU02:FS00:0xa4:
07:20:53:WU02:FS00:0xa4:Preparing to commence simulation
07:20:53:WU02:FS00:0xa4:- Ensuring status. Please wait.
07:21:02:WU02:FS00:0xa4:- Looking at optimizations...
07:21:02:WU02:FS00:0xa4:- Working with standard loops on this execution.
07:21:02:WU02:FS00:0xa4:Examination of work files indicates 8 consecutive improper terminations of core.
07:21:02:WU02:FS00:0xa4:- Expanded 826026 -> 1398024 (decompressed 169.2 percent)
07:21:02:WU02:FS00:0xa4:Called DecompressByteArray: compressed_data_size=826026 data_size=1398024, decompressed_data_size=1398024 diff=0
07:21:02:WU02:FS00:0xa4:- Digital signature verified
07:21:02:WU02:FS00:0xa4:
07:21:02:WU02:FS00:0xa4:Project: 9032 (Run 497, Clone 1, Gen 2183)
07:21:02:WU02:FS00:0xa4:
07:21:02:WU02:FS00:0xa4:Entering M.D.
07:21:09:WU02:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
07:21:53:WU02:FS00:Starting
07:21:53:WARNING:WU02:FS00:AS lowered CPUs from 11 to 10
07:21:53:WU02:FS00:Removing old file './work/02/logfile_01-20181111-064952.txt'
07:21:53:WU02:FS00:Running FahCore: /opt/fah/FAHCoreWrapper /home/mpi/cores/cores.foldingathome.org/Linux/AMD64/Core_a4.fah/FahCore_a4 -dir 02 -suffix 01 -version 705 -lifeline 6601 -checkpoint 15 -np 10
07:21:53:WU02:FS00:Started FahCore on PID 13568
07:21:53:WU02:FS00:Core PID:13572
07:21:53:WU02:FS00:FahCore 0xa4 started
07:21:53:WU02:FS00:0xa4:
07:21:53:WU02:FS00:0xa4:*------------------------------*
07:21:53:WU02:FS00:0xa4:Folding@Home Gromacs GB Core
07:21:53:WU02:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
07:21:53:WU02:FS00:0xa4:
07:21:53:WU02:FS00:0xa4:Preparing to commence simulation
07:21:53:WU02:FS00:0xa4:- Ensuring status. Please wait.
07:22:02:WU02:FS00:0xa4:- Looking at optimizations...
07:22:02:WU02:FS00:0xa4:- Working with standard loops on this execution.
07:22:02:WU02:FS00:0xa4:Examination of work files indicates 8 consecutive improper terminations of core.
07:22:02:WU02:FS00:0xa4:- Expanded 826026 -> 1398024 (decompressed 169.2 percent)
07:22:02:WU02:FS00:0xa4:Called DecompressByteArray: compressed_data_size=826026 data_size=1398024, decompressed_data_size=1398024 diff=0
07:22:02:WU02:FS00:0xa4:- Digital signature verified
07:22:02:WU02:FS00:0xa4:
07:22:02:WU02:FS00:0xa4:Project: 9032 (Run 497, Clone 1, Gen 2183)
07:22:02:WU02:FS00:0xa4:
07:22:02:WU02:FS00:0xa4:Entering M.D.
07:22:09:WU02:FS00:FahCore returned: INTERRUPTED (102 = 0x66)


meanwhile, dmesg is flooded with segfaults since core 0xa4 depends on vsyscall and that is all very standard fare -

Code: Select all

[Sun Nov 11 07:21:08 2018] Code: Bad RIP value.
[Sun Nov 11 07:21:08 2018] audit: type=1701 audit(1541920868.788:376): auid=1000 uid=1000 gid=1000 ses=1 pid=13557 comm="FahCore_a4" exe="/home/mpi/cores/cores.foldingathome.org/Linux/AMD64/Core_a4.fah/FahCore_a4" sig=11 res=1
[Sun Nov 11 07:21:08 2018] audit: type=1130 audit(1541920868.821:377): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@111-13561-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Sun Nov 11 07:21:09 2018] audit: type=1131 audit(1541920869.368:378): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@111-13561-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Sun Nov 11 07:22:08 2018] FahCore_a4[13573] vsyscall attempted with vsyscall=none ip:ffffffffff600400 cs:33 sp:7f55126bcd88 ax:ffffffffff600400 si:7f55126bcc2a di:7f55126bcdc8
[Sun Nov 11 07:22:08 2018] FahCore_a4[13573]: segfault at ffffffffff600400 ip ffffffffff600400 sp 00007f55126bcd88 error 15


[code_glitch]

Almost forgot:
The hardware is about 10 old dual-opteron boxes, some older fx boxes, a zen box, some intel desktop boxes of various generations... A bit of a mish-mash as its what I had on hand and was very very cheap (average about £1 per core, £0.80 per GB of memory all in. Couldnt really go faster for less from what I saw).

Its mainly used for OpenMPI for the engineering workloads and a bunch of custom code that deals with databases, finance, pricing models and that sort of thing. The software stack is lightly customised Arch that is mirrored on all systems. Key problem being a number of these nodes are about 200 days into their workloads, and not due to complete for another 50-100(?). Lately, a number of cores arent fully loaded as the jobs are more sporadic so I was hoping these could be use to help out with fah.

[JimboPalmer]

[I am just a user, like yourself: I need to own 6 laptops (my Theatre front office laptops need Win 7 or newer, while my BBQ back office laptops can't run anything newer than Win XP) My one desktop generates 3/4 of my points.]

So to use a core as old as a4 (which supports any CPU newer than a Pentium 3) you need to have sysvcall = native, but that involves a reboot of Linux. Some of your boxes won't be usable for 100 days. Do you normally reboot between workloads?

F@H does not allow us to choose between a4 and a7, (a much newer core) because if they did, almost everyone would hog a7 as they get more points. No new research is being started using a4, but some research takes years, so some existing research will continue to use a4.

Unrelated, and as an aside, on the node you show you have 12 threads, F@H detects an outdated GPU you are not using, and lowers that to 11. 11 is a 'large' prime number, so it then lowers to 10. All to support a GPU you are not using. If you hard code your CPU Slot configuration to 12 (from -1 which means automatic), you would get a minor bump in points.

[bruce]

... If you hard code your CPU Slot configuration to 12 (from -1 which means automatic), you would get a minor bump in points.

... or, if heat is a problem, you'll probably get dependable performance by forcing it down to 8 or 9 (although that will reduce your points). Turning off SMT could actually help both problems, though that might interfere with your OpenMP applications.

[code_glitch]

@JimboPalmer: Some more CFD oriented workloads take a few days to complete, but some other finance modelling takes on the order of months to complete a single run, hence why a reboot wont be feasible this winter. Since these machines are silent and have a reasonable TDP when combined I actually use them to heat the house.I'll keep the core reduction issue in mind, most have ancient quadros or equally ancient ATI GPUs just for the BIOS to be happy. The 9600GT was one of the few passively cooled cards I had in the bin.

Lately, the shorter workloads are getting a bit more sporadic, some nights things get a bit cold. fah would have been the solution to keeping everything warm as winter sets in, but since I'm not willing to give up on the last few months of progress on other workloads rebooting is not really possible right now. At the moment SSHing into a bunch of nodes to catch an infinite loop is also not practical either. Im really just surprised something like this hasnt been caught and fixed. I had a quick scan for the source to the FAHClient but nothing came up so fixing the problem myself wouldn't be easy at all.

@Bruce: No issues on heat. These are K10 opterons so no SMT and are factory water cooled so heat is a non issue.

[Joe_H]

As mentioned, the Assignment Servers will give both A4 and A7 WU's as they have no way of distinguishing whether a system lacks support for use of vsyscall. Since a restart to enable that is not being considered, I can think of one workaround. As you are not doing GPU folding, CPU folding will work just fine in a VM. Setting up a VM running the Linux of your choice with vsyscall enabled should no longer have A4 processing issues.

[code_glitch]

Thats unfortunate. Is this something that is on the roadmap? If it isn't, I've read briefly about the existence of some type of plugin support in FAHClient on the main fah website and that the forum would be the place to ask about this. The change I'm looking for is pretty basic so if such a thing exists I would definitely be interested in pursuing that option.

[JimboPalmer]

Is this something that is on the roadmap?

As I say, no new research is being started with a4. But the research already started deserves to finish.

We are users, if a roadmap exists, we would not know of it, let alone what is on it.

[code_glitch]

Guess no folding this winter then

[Joe_H]

There is one other folding option. You could try the NaCl client for Chrome on your system. That will fold on all OS's.

[bruce]

It won't do a lot of heating, but more than zero.

[bjmgeek]

I also use FaH for heating in the winter. I've had no problem with the NaCL client using all 4 cores on my laptop, or all 16 cores on my 16 core Xeon E5620. I have the same problem with the a4 core causing endless segfaults. While I can use the NaCL client on that too, I'd prefer to have a less manual solution. You mentioned vsyscall=native. Is that a kernel boot parameter, or does that go in my FAHClient xml file?

[Joe_H]

That is a kernel boot parameter for Linux.