FahCore_a7.exe fails on Windows Azure (hyper-v) server

Moderators: Site Moderators, FAHC Science Team

ERROR: rtdscp CPU instruction

Postby cs1818 » Fri Oct 21, 2016 12:21 am

Running as a service on Windows Server 2012 r2, clean install. Everything works initially, but ends up crashing every time. I can send the events logged in Windows, and the log file if needed. Please help! From the log file in program data directory:


ERROR:Source code file: /host/windows-cross-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212

The GROMACS executable was compiled to use the rdtscp CPU instruction. However, this is not supported by the current hardware and continuing would lead to a crash. Please rebuild GROMACS with the GMX_USE_RDTSCP=OFF CMake option.
Last edited by cs1818 on Fri Oct 21, 2016 12:33 am, edited 1 time in total.
cs1818
 
Posts: 2
Joined: Fri Oct 21, 2016 12:17 am

Re: ERROR: rtdscp CPU instruction

Postby cs1818 » Fri Oct 21, 2016 12:31 am

Additional Info: it's running in Hyper-V with NUMA, 32 cores, 48GB RAM, on a Dell R620 physical machine.
cs1818
 
Posts: 2
Joined: Fri Oct 21, 2016 12:17 am

Re: ERROR: rtdscp CPU instruction

Postby JimboPalmer » Fri Oct 21, 2016 3:17 am

this is for some other hypervisor, but illustrates the problem and how they solved it. https://xenbits.xen.org/docs/4.3-testin ... scmode.txt

I do not know what options Hyper V may need to correctly emulate rdtscp.
Tsar of all the Rushers
I tried to remain childlike, all I achieved was childish.
A friend to those who want no friends
JimboPalmer
 
Posts: 1809
Joined: Mon Feb 16, 2009 5:12 am
Location: Greenwood MS USA

FahCore_a7.exe fails on Windows Azure (hyper-v) server

Postby alius » Fri Dec 02, 2016 7:53 pm

Windows server 2012 R2 datacenter under Hyper-V. 2 CPU (Xeon E5-2673 v3)

Error as follows:
Code: Select all
18:32:14:WU00:FS00:0xa7:ERROR:The GROMACS executable was compiled to use the rdtscp CPU instruction. However, this is not supported by the current hardware and continuing would lead to a crash. Please rebuild GROMACS with the GMX_USE_RDTSCP=OFF CMake option.
18:32:14:WU00:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
18:32:14:WU00:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
18:32:14:WU00:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
18:32:14:WU00:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
18:32:14:WU00:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:15:WARNING:WU00:FS00:FahCore returned an unknown error code which probably indicates that it crashed
18:32:15:WARNING:WU00:FS00:FahCore returned: UNKNOWN_ENUM (-1073741784 = 0xc0000028)


System info:
Code: Select all
18:32:12:******************************* System ********************************
18:32:12:          CPU: Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
18:32:12:       CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
18:32:12:         CPUs: 16
18:32:12:       Memory: 32.00GiB
18:32:12:  Free Memory: 30.44GiB
18:32:12:      Threads: WINDOWS_THREADS
18:32:12:   OS Version: 6.2
18:32:12:  Has Battery: false
18:32:12:   On Battery: false
18:32:12:   UTC Offset: 0
18:32:12:          PID: 4740
18:32:12:          CWD: C:/Windows/system32
18:32:12:           OS: Windows Server 2012 R2 Datacenter
18:32:12:      OS Arch: AMD64
18:32:12:         GPUs: 0
18:32:12:         CUDA: Not detected
18:32:12:Win32 Service: true
18:32:12:***********************************************************************


Head of the log where the core tries to start:
Code: Select all
18:32:12:WU00:FS00:0xa7:*********************** Log Started 2016-12-02T18:32:12Z ***********************
18:32:12:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
18:32:12:WU00:FS00:0xa7:       Type: 0xa7
18:32:12:WU00:FS00:0xa7:       Core: Gromacs
18:32:12:WU00:FS00:0xa7:    Website: http://folding.stanford.edu/
18:32:12:WU00:FS00:0xa7:  Copyright: (c) 2009-2016 Stanford University
18:32:12:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:32:12:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 704 -lifeline 3032 -checkpoint 15 -np
18:32:12:WU00:FS00:0xa7:             16 -service
18:32:12:WU00:FS00:0xa7:     Config: <none>
18:32:12:WU00:FS00:0xa7:************************************ Build *************************************
18:32:12:WU00:FS00:0xa7:    Version: 0.0.11
18:32:12:WU00:FS00:0xa7:       Date: Sep 21 2016
18:32:12:WU00:FS00:0xa7:       Time: 01:43:48
18:32:12:WU00:FS00:0xa7: Repository: Git
18:32:12:WU00:FS00:0xa7:   Revision: 957bd90e68d95ddcf1594dc15ff6c64cc4555146
18:32:12:WU00:FS00:0xa7:     Branch: master
18:32:12:WU00:FS00:0xa7:   Compiler: GNU 4.2.1 Compatible Clang 3.9.0 (trunk 274080)
18:32:12:WU00:FS00:0xa7:    Options: -std=gnu++98 -O3 -funroll-loops -ffast-math -mfpmath=sse
18:32:12:WU00:FS00:0xa7:             -fno-unsafe-math-optimizations -msse2 -I/mingw64/include
18:32:12:WU00:FS00:0xa7:             -Wno-inconsistent-dllimport -Wno-parentheses-equality
18:32:12:WU00:FS00:0xa7:             -Wno-deprecated-register -Wno-unused-local-typedef
18:32:12:WU00:FS00:0xa7:   Platform: linux2 4.6.0-1-amd64
18:32:12:WU00:FS00:0xa7:       Bits: 64
18:32:12:WU00:FS00:0xa7:       Mode: Release
18:32:12:WU00:FS00:0xa7:       SIMD: avx_256
18:32:12:WU00:FS00:0xa7:************************************ System ************************************
18:32:12:WU00:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
18:32:12:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
18:32:12:WU00:FS00:0xa7:       CPUs: 16
18:32:12:WU00:FS00:0xa7:     Memory: 32.00GiB
18:32:12:WU00:FS00:0xa7:Free Memory: 30.46GiB
18:32:12:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
18:32:12:WU00:FS00:0xa7: OS Version: 6.2
18:32:12:WU00:FS00:0xa7:Has Battery: false
18:32:12:WU00:FS00:0xa7: On Battery: false
18:32:12:WU00:FS00:0xa7: UTC Offset: 0
18:32:12:WU00:FS00:0xa7:        PID: 2164
18:32:12:WU00:FS00:0xa7:        CWD: C:\Users\Alius\AppData\Roaming\FAHClient\work
18:32:12:WU00:FS00:0xa7:         OS: Windows Server 2012 R2 Datacenter
18:32:12:WU00:FS00:0xa7:    OS Arch: AMD64
18:32:12:WU00:FS00:0xa7:********************************************************************************
18:32:12:WU00:FS00:0xa7:Project: 11920 (Run 957, Clone 4, Gen 112)
18:32:12:WU00:FS00:0xa7:Unit: 0x00000089ab4041295809c36e95f41303
18:32:12:WU00:FS00:0xa7:Reading tar file core.xml
18:32:12:WU00:FS00:0xa7:Reading tar file frame112.tpr
18:32:12:WU00:FS00:0xa7:Digital signatures verified
18:32:12:WU00:FS00:0xa7:Calling: mdrun -s frame112.tpr -o frame112.trr -cpt 15 -nt 16
18:32:14:WU00:FS00:0xa7:Steps: first=8960000 total=80000
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
18:32:14:WU00:FS00:0xa7:WARNING:Unexpected exit() call
18:32:14:WU00:FS00:0xa7:ERROR:Source code file: /host/windows-cross-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:Source code file: /host/windows-cross-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
18:32:14:WU00:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
18:32:14:WU00:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:Fatal error:
18:32:14:WU00:FS00:0xa7:ERROR:Source code file: /host/windows-cross-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
18:32:14:WU00:FS00:0xa7:ERROR:Source code file: /host/windows-cross-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
18:32:14:WU00:FS00:0xa7:ERROR:Source code file: /host/windows-cross-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
18:32:14:WU00:FS00:0xa7:ERROR:Fatal error:
18:32:14:WU00:FS00:0xa7:ERROR:The GROMACS executable was compiled to use the rdtscp CPU instruction. However, this is not supported by the current hardware and continuing would lead to a crash. Please rebuild GROMACS with the GMX_USE_RDTSCP=OFF CMake option.


Every time this VM pulls down an a7 it fails the same way until it finds an a4. If there is any other information I can provide I would be happy to do so. Cheers
alius
 
Posts: 5
Joined: Tue Sep 13, 2011 2:11 am

Re: FahCore_a7.exe fails on Windows Azure (hyper-v) server

Postby alius » Sat Dec 03, 2016 7:57 pm

Unfortunately, I do not have control over the hyper-v hypervisor settings for this host, as this is a Windows Azure cloud server. If the situation arises that I try this with Hyper-V running on hardware and a host OS that I control, I will definitely poke around and report back if I find the right setting(s). This may even be a flaw in hyper-v itself, but I thought the FAH developers should be made aware of this issue.

There should be a way to detect that the client is running under hyper-v, and utilize the alternate GROMACS compiled using the GMX_USE_RDTSCP=OFF flag, so that donors running under hyper-v can successfully complete these WUs once the core gets put on non-test projects. https://blogs.technet.microsoft.com/ton ... side-a-vm/ may be helpful if that is an avenue that might be pursued.
Last edited by alius on Sat Dec 03, 2016 8:55 pm, edited 1 time in total.
alius
 
Posts: 5
Joined: Tue Sep 13, 2011 2:11 am

Re: FahCore_a7.exe fails on Windows Azure (hyper-v) server

Postby Joe_H » Sat Dec 03, 2016 8:17 pm

Unfortunately the recent releases of the Gromacs code do not include the ability to compile with conditional execution paths depending on the CPU hardware being used. So a folding core is built and compiled for each target platform's capabilities. Right now that is two separate downloads for each OS type, one for CPU's that support AVX2 and the other for older hardware that does not and therefore uses SSE2 computations. So the detection could not be built into the folding core but would need to be in the client to download and use a core suitable for use in a VM without full access to the detected type of CPU hardware.

We can bring this issue to the attention of the developer of the A7 core. He may be able to add support for this type of configuration, or to at least not assign to it, but that might not be easy or quick.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Joe_H
Site Admin
 
Posts: 6123
Joined: Tue Apr 21, 2009 5:41 pm
Location: W. MA

Re: FahCore_a7.exe fails on Windows Azure (hyper-v) server

Postby alius » Sat Dec 03, 2016 8:52 pm

Joe_H wrote:We can bring this issue to the attention of the developer of the A7 core. He may be able to add support for this type of configuration, or to at least not assign to it, but that might not be easy or quick.



These things rarely are! Just wanted to provide as much info as I could in the hope it might help. Thanks!
alius
 
Posts: 5
Joined: Tue Sep 13, 2011 2:11 am

Re: FahCore_a7.exe fails on Windows Azure (hyper-v) server

Postby bruce » Fri Dec 09, 2016 8:08 am

Thanks for reporting this.

I can inform you that code has been added to the upcoming v4.7.16 client to detect the rtdscp CPU instruction and code has been added to an upcoming version of the Assignment Server to process that report. These changes will be rolled out "soon"© and once both are in place (i.e. - at some point in the future) it'll be fixed.
bruce
 
Posts: 19142
Joined: Thu Nov 29, 2007 11:13 pm
Location: So. Cal.

Re: FahCore_a7.exe fails on Windows Azure (hyper-v) server

Postby JimboPalmer » Fri Dec 09, 2016 3:49 pm

Thank you for taking care of this, and of us!
JimboPalmer
 
Posts: 1809
Joined: Mon Feb 16, 2009 5:12 am
Location: Greenwood MS USA

Re: FahCore_a7.exe fails on Windows Azure (hyper-v) server

Postby alius » Mon Dec 12, 2016 11:53 pm

JimboPalmer wrote:Thank you for taking care of this, and of us!


Seconded, great news! Thank you!
alius
 
Posts: 5
Joined: Tue Sep 13, 2011 2:11 am


Return to CPU Projects - beta FAHCores (Currently _a7)

Who is online

Users browsing this forum: No registered users and 1 guest

cron