Project 16804 (Run 14, Clone 406, Gen 12) crashing at end.

Moderators: Site Moderators, FAHC Science Team

Post Reply
Smith6612
Posts: 13
Joined: Fri Jan 31, 2014 1:41 pm

Project 16804 (Run 14, Clone 406, Gen 12) crashing at end.

Post by Smith6612 »

Looks like I got a bad work unit here. This one has been crashing regularly throughout it's run and will not complete. The Folding@Home client starts work on another work unit, then picks up the problem work unit again from the incomplete queue. Here's some logs.

Code: Select all

13:34:50:WU00:FS00:Connecting to 65.254.110.245:80
13:34:50:WU00:FS00:Assigned to work server 130.237.11.145
13:34:50:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:9 from 130.237.11.145
13:34:50:WU00:FS00:Connecting to 130.237.11.145:8080
13:34:51:WU00:FS00:Downloading 3.39MiB
13:34:52:WU00:FS00:Download complete
13:34:52:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:16804 run:14 clone:406 gen:12 core:0xa7 unit:0x0000000e82ed0b915e9df53e2f37ed91
13:36:03:WU00:FS00:Starting
13:36:03:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Sean\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/avx/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 20764 -checkpoint 15 -np 9
13:36:03:WU00:FS00:Started FahCore on PID 3852
13:36:03:WU00:FS00:Core PID:9996
13:36:03:WU00:FS00:FahCore 0xa7 started
13:36:03:WU00:FS00:0xa7:*********************** Log Started 2020-05-02T13:36:03Z ***********************
13:36:03:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
13:36:03:WU00:FS00:0xa7:       Type: 0xa7
13:36:03:WU00:FS00:0xa7:       Core: Gromacs
13:36:03:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 3852 -checkpoint 15 -np 9
13:36:03:WU00:FS00:0xa7:************************************ CBang *************************************
13:36:03:WU00:FS00:0xa7:       Date: Oct 26 2019
13:36:03:WU00:FS00:0xa7:       Time: 01:38:25
13:36:03:WU00:FS00:0xa7:   Revision: c46a1a011a24143739ac7218c5a435f66777f62f
13:36:03:WU00:FS00:0xa7:     Branch: master
13:36:03:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
13:36:03:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
13:36:03:WU00:FS00:0xa7:   Platform: win32 10
13:36:03:WU00:FS00:0xa7:       Bits: 64
13:36:03:WU00:FS00:0xa7:       Mode: Release
13:36:03:WU00:FS00:0xa7:************************************ System ************************************
13:36:03:WU00:FS00:0xa7:        CPU: AMD Ryzen 5 1600 Six-Core Processor
13:36:03:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
13:36:03:WU00:FS00:0xa7:       CPUs: 12
13:36:03:WU00:FS00:0xa7:     Memory: 31.94GiB
13:36:03:WU00:FS00:0xa7:Free Memory: 25.07GiB
13:36:03:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
13:36:03:WU00:FS00:0xa7: OS Version: 6.2
13:36:03:WU00:FS00:0xa7:Has Battery: false
13:36:03:WU00:FS00:0xa7: On Battery: false
13:36:03:WU00:FS00:0xa7: UTC Offset: -4
13:36:03:WU00:FS00:0xa7:        PID: 9996
13:36:03:WU00:FS00:0xa7:        CWD: C:\Users\Sean\AppData\Roaming\FAHClient\work
13:36:04:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
13:36:04:WU00:FS00:0xa7:    Version: 0.0.18
13:36:04:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
13:36:04:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
13:36:04:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
13:36:04:WU00:FS00:0xa7:       Date: Oct 26 2019
13:36:04:WU00:FS00:0xa7:       Time: 01:52:30
13:36:04:WU00:FS00:0xa7:   Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
13:36:04:WU00:FS00:0xa7:     Branch: master
13:36:04:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
13:36:04:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
13:36:04:WU00:FS00:0xa7:   Platform: win32 10
13:36:04:WU00:FS00:0xa7:       Bits: 64
13:36:04:WU00:FS00:0xa7:       Mode: Release
13:36:04:WU00:FS00:0xa7:************************************ Build *************************************
13:36:04:WU00:FS00:0xa7:       SIMD: avx_256
13:36:04:WU00:FS00:0xa7:********************************************************************************
13:36:04:WU00:FS00:0xa7:Project: 16804 (Run 14, Clone 406, Gen 12)
13:36:04:WU00:FS00:0xa7:Unit: 0x0000000e82ed0b915e9df53e2f37ed91
13:36:04:WU00:FS00:0xa7:Reading tar file core.xml
13:36:04:WU00:FS00:0xa7:Reading tar file frame12.tpr
13:36:04:WU00:FS00:0xa7:Digital signatures verified
13:36:04:WU00:FS00:0xa7:Calling: mdrun -s frame12.tpr -o frame12.trr -cpt 15 -nt 9
13:36:04:WU00:FS00:0xa7:Steps: first=6000000 total=500000
13:36:05:WU00:FS00:0xa7:Completed 1 out of 500000 steps (0%)
13:40:17:WU00:FS00:0xa7:Completed 5000 out of 500000 steps (1%)
13:44:28:WU00:FS00:0xa7:Completed 10000 out of 500000 steps (2%)
13:48:41:WU00:FS00:0xa7:Completed 15000 out of 500000 steps (3%)
13:52:53:WU00:FS00:0xa7:Completed 20000 out of 500000 steps (4%)
13:57:06:WU00:FS00:0xa7:Completed 25000 out of 500000 steps (5%)
14:01:18:WU00:FS00:0xa7:Completed 30000 out of 500000 steps (6%)
14:05:29:WU00:FS00:0xa7:Completed 35000 out of 500000 steps (7%)
14:09:42:WU00:FS00:0xa7:Completed 40000 out of 500000 steps (8%)
14:13:53:WU00:FS00:0xa7:Completed 45000 out of 500000 steps (9%)
14:18:09:WU00:FS00:0xa7:Completed 50000 out of 500000 steps (10%)
14:22:29:WU00:FS00:0xa7:Completed 55000 out of 500000 steps (11%)
14:26:47:WU00:FS00:0xa7:Completed 60000 out of 500000 steps (12%)
14:31:08:WU00:FS00:0xa7:Completed 65000 out of 500000 steps (13%)
14:35:30:WU00:FS00:0xa7:Completed 70000 out of 500000 steps (14%)
14:39:43:WU00:FS00:0xa7:Completed 75000 out of 500000 steps (15%)
14:43:57:WU00:FS00:0xa7:Completed 80000 out of 500000 steps (16%)
14:48:10:WU00:FS00:0xa7:Completed 85000 out of 500000 steps (17%)
14:52:23:WU00:FS00:0xa7:Completed 90000 out of 500000 steps (18%)
14:56:36:WU00:FS00:0xa7:Completed 95000 out of 500000 steps (19%)
15:00:48:WU00:FS00:0xa7:Completed 100000 out of 500000 steps (20%)
15:05:00:WU00:FS00:0xa7:Completed 105000 out of 500000 steps (21%)
15:09:13:WU00:FS00:0xa7:Completed 110000 out of 500000 steps (22%)
15:13:25:WU00:FS00:0xa7:Completed 115000 out of 500000 steps (23%)
15:17:38:WU00:FS00:0xa7:Completed 120000 out of 500000 steps (24%)
15:21:51:WU00:FS00:0xa7:Completed 125000 out of 500000 steps (25%)
15:26:03:WU00:FS00:0xa7:Completed 130000 out of 500000 steps (26%)
15:30:15:WU00:FS00:0xa7:Completed 135000 out of 500000 steps (27%)
15:34:27:WU00:FS00:0xa7:Completed 140000 out of 500000 steps (28%)
15:38:40:WU00:FS00:0xa7:Completed 145000 out of 500000 steps (29%)
15:42:52:WU00:FS00:0xa7:Completed 150000 out of 500000 steps (30%)
15:47:05:WU00:FS00:0xa7:Completed 155000 out of 500000 steps (31%)
15:51:17:WU00:FS00:0xa7:Completed 160000 out of 500000 steps (32%)
15:55:29:WU00:FS00:0xa7:Completed 165000 out of 500000 steps (33%)
15:59:41:WU00:FS00:0xa7:Completed 170000 out of 500000 steps (34%)
16:03:53:WU00:FS00:0xa7:Completed 175000 out of 500000 steps (35%)
16:08:05:WU00:FS00:0xa7:Completed 180000 out of 500000 steps (36%)
16:12:17:WU00:FS00:0xa7:Completed 185000 out of 500000 steps (37%)
16:16:29:WU00:FS00:0xa7:Completed 190000 out of 500000 steps (38%)
16:20:42:WU00:FS00:0xa7:Completed 195000 out of 500000 steps (39%)
16:24:55:WU00:FS00:0xa7:Completed 200000 out of 500000 steps (40%)
16:29:07:WU00:FS00:0xa7:Completed 205000 out of 500000 steps (41%)
16:33:19:WU00:FS00:0xa7:Completed 210000 out of 500000 steps (42%)
16:37:30:WU00:FS00:0xa7:Completed 215000 out of 500000 steps (43%)
16:41:38:WU00:FS00:0xa7:Completed 220000 out of 500000 steps (44%)
16:45:48:WU00:FS00:0xa7:Completed 225000 out of 500000 steps (45%)
16:49:57:WU00:FS00:0xa7:Completed 230000 out of 500000 steps (46%)
16:54:07:WU00:FS00:0xa7:Completed 235000 out of 500000 steps (47%)
16:58:16:WU00:FS00:0xa7:Completed 240000 out of 500000 steps (48%)
16:58:23:WARNING:WU00:FS00:FahCore returned an unknown error code which probably indicates that it crashed
16:58:23:WARNING:WU00:FS00:FahCore returned: UNKNOWN_ENUM (-1073741819 = 0xc0000005)
16:58:23:WU00:FS00:Starting
16:58:23:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Sean\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/avx/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 20764 -checkpoint 15 -np 9
16:58:23:WU00:FS00:Started FahCore on PID 7408
16:58:23:WU00:FS00:Core PID:18852
16:58:23:WU00:FS00:FahCore 0xa7 started
16:58:23:WU00:FS00:0xa7:*********************** Log Started 2020-05-02T16:58:23Z ***********************
16:58:23:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
16:58:23:WU00:FS00:0xa7:       Type: 0xa7
16:58:23:WU00:FS00:0xa7:       Core: Gromacs
16:58:23:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 7408 -checkpoint 15 -np 9
16:58:23:WU00:FS00:0xa7:************************************ CBang *************************************
16:58:23:WU00:FS00:0xa7:       Date: Oct 26 2019
16:58:23:WU00:FS00:0xa7:       Time: 01:38:25
16:58:23:WU00:FS00:0xa7:   Revision: c46a1a011a24143739ac7218c5a435f66777f62f
16:58:23:WU00:FS00:0xa7:     Branch: master
16:58:23:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
16:58:23:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
16:58:23:WU00:FS00:0xa7:   Platform: win32 10
16:58:23:WU00:FS00:0xa7:       Bits: 64
16:58:23:WU00:FS00:0xa7:       Mode: Release
16:58:23:WU00:FS00:0xa7:************************************ System ************************************
16:58:23:WU00:FS00:0xa7:        CPU: AMD Ryzen 5 1600 Six-Core Processor
16:58:24:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
16:58:24:WU00:FS00:0xa7:       CPUs: 12
16:58:24:WU00:FS00:0xa7:     Memory: 31.94GiB
16:58:24:WU00:FS00:0xa7:Free Memory: 24.47GiB
16:58:24:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
16:58:24:WU00:FS00:0xa7: OS Version: 6.2
16:58:24:WU00:FS00:0xa7:Has Battery: false
16:58:24:WU00:FS00:0xa7: On Battery: false
16:58:24:WU00:FS00:0xa7: UTC Offset: -4
16:58:24:WU00:FS00:0xa7:        PID: 18852
16:58:24:WU00:FS00:0xa7:        CWD: C:\Users\Sean\AppData\Roaming\FAHClient\work
16:58:24:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
16:58:24:WU00:FS00:0xa7:    Version: 0.0.18
16:58:24:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:58:24:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
16:58:24:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
16:58:24:WU00:FS00:0xa7:       Date: Oct 26 2019
16:58:24:WU00:FS00:0xa7:       Time: 01:52:30
16:58:24:WU00:FS00:0xa7:   Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
16:58:24:WU00:FS00:0xa7:     Branch: master
16:58:24:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
16:58:24:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
16:58:24:WU00:FS00:0xa7:   Platform: win32 10
16:58:24:WU00:FS00:0xa7:       Bits: 64
16:58:24:WU00:FS00:0xa7:       Mode: Release
16:58:24:WU00:FS00:0xa7:************************************ Build *************************************
16:58:24:WU00:FS00:0xa7:       SIMD: avx_256
16:58:24:WU00:FS00:0xa7:********************************************************************************
16:58:24:WU00:FS00:0xa7:Project: 16804 (Run 14, Clone 406, Gen 12)
16:58:24:WU00:FS00:0xa7:Unit: 0x0000000e82ed0b915e9df53e2f37ed91
16:58:24:WU00:FS00:0xa7:Digital signatures verified
16:58:24:WU00:FS00:0xa7:Calling: mdrun -s frame12.tpr -o frame12.trr -cpi state.cpt -cpt 15 -nt 9
16:58:24:WU00:FS00:0xa7:Steps: first=6000000 total=500000
16:58:25:WU00:FS00:0xa7:Completed 231376 out of 500000 steps (46%)
17:01:26:WU00:FS00:0xa7:Completed 235000 out of 500000 steps (47%)
17:05:35:WU00:FS00:0xa7:Completed 240000 out of 500000 steps (48%)
17:09:45:WU00:FS00:0xa7:Completed 245000 out of 500000 steps (49%)
17:13:55:WU00:FS00:0xa7:Completed 250000 out of 500000 steps (50%)
17:16:21:WARNING:WU00:FS00:FahCore returned an unknown error code which probably indicates that it crashed
17:16:21:WARNING:WU00:FS00:FahCore returned: UNKNOWN_ENUM (-1073741819 = 0xc0000005)
17:16:21:WU00:FS00:Starting
17:16:21:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Sean\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/avx/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 20764 -checkpoint 15 -np 9
17:16:21:WU00:FS00:Started FahCore on PID 14916
17:16:21:WU00:FS00:Core PID:4184
17:16:21:WU00:FS00:FahCore 0xa7 started
17:16:22:WU00:FS00:0xa7:*********************** Log Started 2020-05-02T17:16:21Z ***********************
17:16:22:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
17:16:22:WU00:FS00:0xa7:       Type: 0xa7
17:16:22:WU00:FS00:0xa7:       Core: Gromacs
17:16:22:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 14916 -checkpoint 15 -np
17:16:22:WU00:FS00:0xa7:             9
17:16:22:WU00:FS00:0xa7:************************************ CBang *************************************
17:16:22:WU00:FS00:0xa7:       Date: Oct 26 2019
17:16:22:WU00:FS00:0xa7:       Time: 01:38:25
17:16:22:WU00:FS00:0xa7:   Revision: c46a1a011a24143739ac7218c5a435f66777f62f
17:16:22:WU00:FS00:0xa7:     Branch: master
17:16:22:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
17:16:22:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
17:16:22:WU00:FS00:0xa7:   Platform: win32 10
17:16:22:WU00:FS00:0xa7:       Bits: 64
17:16:22:WU00:FS00:0xa7:       Mode: Release
17:16:22:WU00:FS00:0xa7:************************************ System ************************************
17:16:22:WU00:FS00:0xa7:        CPU: AMD Ryzen 5 1600 Six-Core Processor
17:16:22:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
17:16:22:WU00:FS00:0xa7:       CPUs: 12
17:16:22:WU00:FS00:0xa7:     Memory: 31.94GiB
17:16:22:WU00:FS00:0xa7:Free Memory: 24.33GiB
17:16:22:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
17:16:22:WU00:FS00:0xa7: OS Version: 6.2
17:16:22:WU00:FS00:0xa7:Has Battery: false
17:16:22:WU00:FS00:0xa7: On Battery: false
17:16:22:WU00:FS00:0xa7: UTC Offset: -4
17:16:22:WU00:FS00:0xa7:        PID: 4184
17:16:22:WU00:FS00:0xa7:        CWD: C:\Users\Sean\AppData\Roaming\FAHClient\work
17:16:22:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
17:16:22:WU00:FS00:0xa7:    Version: 0.0.18
17:16:22:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:16:22:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
17:16:22:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
17:16:22:WU00:FS00:0xa7:       Date: Oct 26 2019
17:16:22:WU00:FS00:0xa7:       Time: 01:52:30
17:16:22:WU00:FS00:0xa7:   Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
17:16:22:WU00:FS00:0xa7:     Branch: master
17:16:22:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
17:16:22:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
17:16:22:WU00:FS00:0xa7:   Platform: win32 10
17:16:22:WU00:FS00:0xa7:       Bits: 64
17:16:22:WU00:FS00:0xa7:       Mode: Release
17:16:22:WU00:FS00:0xa7:************************************ Build *************************************
17:16:22:WU00:FS00:0xa7:       SIMD: avx_256
17:16:22:WU00:FS00:0xa7:********************************************************************************
17:16:22:WU00:FS00:0xa7:Project: 16804 (Run 14, Clone 406, Gen 12)
17:16:22:WU00:FS00:0xa7:Unit: 0x0000000e82ed0b915e9df53e2f37ed91
17:16:22:WU00:FS00:0xa7:Digital signatures verified
17:16:22:WU00:FS00:0xa7:Calling: mdrun -s frame12.tpr -o frame12.trr -cpi state.cpt -cpt 15 -nt 9
17:16:22:WU00:FS00:0xa7:Steps: first=6000000 total=500000
17:16:23:WU00:FS00:0xa7:Completed 249426 out of 500000 steps (49%)
17:16:52:WU00:FS00:0xa7:Completed 250000 out of 500000 steps (50%)
17:21:02:WU00:FS00:0xa7:Completed 255000 out of 500000 steps (51%)
17:25:12:WU00:FS00:0xa7:Completed 260000 out of 500000 steps (52%)
17:29:21:WU00:FS00:0xa7:Completed 265000 out of 500000 steps (53%)
17:33:36:WU00:FS00:0xa7:Completed 270000 out of 500000 steps (54%)
17:37:46:WU00:FS00:0xa7:Completed 275000 out of 500000 steps (55%)
17:41:56:WU00:FS00:0xa7:Completed 280000 out of 500000 steps (56%)
17:46:05:WU00:FS00:0xa7:Completed 285000 out of 500000 steps (57%)
17:50:16:WU00:FS00:0xa7:Completed 290000 out of 500000 steps (58%)
17:54:26:WU00:FS00:0xa7:Completed 295000 out of 500000 steps (59%)
17:58:36:WU00:FS00:0xa7:Completed 300000 out of 500000 steps (60%)
18:02:44:WU00:FS00:0xa7:Completed 305000 out of 500000 steps (61%)
18:06:53:WU00:FS00:0xa7:Completed 310000 out of 500000 steps (62%)
18:11:04:WU00:FS00:0xa7:Completed 315000 out of 500000 steps (63%)
18:15:14:WU00:FS00:0xa7:Completed 320000 out of 500000 steps (64%)
18:19:25:WU00:FS00:0xa7:Completed 325000 out of 500000 steps (65%)
18:23:36:WU00:FS00:0xa7:Completed 330000 out of 500000 steps (66%)
******************************* Date: 2020-05-02 *******************************
18:27:47:WU00:FS00:0xa7:Completed 335000 out of 500000 steps (67%)
18:31:58:WU00:FS00:0xa7:Completed 340000 out of 500000 steps (68%)
18:36:08:WU00:FS00:0xa7:Completed 345000 out of 500000 steps (69%)
18:40:18:WU00:FS00:0xa7:Completed 350000 out of 500000 steps (70%)
18:44:29:WU00:FS00:0xa7:Completed 355000 out of 500000 steps (71%)
18:48:41:WU00:FS00:0xa7:Completed 360000 out of 500000 steps (72%)
18:52:51:WU00:FS00:0xa7:Completed 365000 out of 500000 steps (73%)
18:57:02:WU00:FS00:0xa7:Completed 370000 out of 500000 steps (74%)
19:01:13:WU00:FS00:0xa7:Completed 375000 out of 500000 steps (75%)
19:05:24:WU00:FS00:0xa7:Completed 380000 out of 500000 steps (76%)
19:09:34:WU00:FS00:0xa7:Completed 385000 out of 500000 steps (77%)
19:13:45:WU00:FS00:0xa7:Completed 390000 out of 500000 steps (78%)
19:17:57:WU00:FS00:0xa7:Completed 395000 out of 500000 steps (79%)
19:22:09:WU00:FS00:0xa7:Completed 400000 out of 500000 steps (80%)
19:26:20:WU00:FS00:0xa7:Completed 405000 out of 500000 steps (81%)
19:30:31:WU00:FS00:0xa7:Completed 410000 out of 500000 steps (82%)
19:34:42:WU00:FS00:0xa7:Completed 415000 out of 500000 steps (83%)
19:38:53:WU00:FS00:0xa7:Completed 420000 out of 500000 steps (84%)
19:43:04:WU00:FS00:0xa7:Completed 425000 out of 500000 steps (85%)
19:47:14:WU00:FS00:0xa7:Completed 430000 out of 500000 steps (86%)
19:51:25:WU00:FS00:0xa7:Completed 435000 out of 500000 steps (87%)
19:55:36:WU00:FS00:0xa7:Completed 440000 out of 500000 steps (88%)
19:59:47:WU00:FS00:0xa7:Completed 445000 out of 500000 steps (89%)
20:03:57:WU00:FS00:0xa7:Completed 450000 out of 500000 steps (90%)
20:08:08:WU00:FS00:0xa7:Completed 455000 out of 500000 steps (91%)
20:12:18:WU00:FS00:0xa7:Completed 460000 out of 500000 steps (92%)
20:16:30:WU00:FS00:0xa7:Completed 465000 out of 500000 steps (93%)
20:20:41:WU00:FS00:0xa7:Completed 470000 out of 500000 steps (94%)
20:24:52:WU00:FS00:0xa7:Completed 475000 out of 500000 steps (95%)
20:29:02:WU00:FS00:0xa7:Completed 480000 out of 500000 steps (96%)
20:33:16:WU00:FS00:0xa7:Completed 485000 out of 500000 steps (97%)
20:37:26:WU00:FS00:0xa7:Completed 490000 out of 500000 steps (98%)
20:41:37:WU00:FS00:0xa7:Completed 495000 out of 500000 steps (99%)
20:44:14:WARNING:WU00:FS00:FahCore returned an unknown error code which probably indicates that it crashed
20:44:14:WARNING:WU00:FS00:FahCore returned: UNKNOWN_ENUM (-1073741819 = 0xc0000005)
13:34:50:WU00:FS00:Connecting to 65.254.110.245:80
13:34:50:WU00:FS00:Assigned to work server 130.237.11.145
13:34:50:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:9 from 130.237.11.145
13:34:50:WU00:FS00:Connecting to 130.237.11.145:8080
13:34:51:WU00:FS00:Downloading 3.39MiB
13:34:52:WU00:FS00:Download complete
13:34:52:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:16804 run:14 clone:406 gen:12 core:0xa7 unit:0x0000000e82ed0b915e9df53e2f37ed91
13:36:03:WU00:FS00:Starting
13:36:03:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Sean\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/avx/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 20764 -checkpoint 15 -np 9
13:36:03:WU00:FS00:Started FahCore on PID 3852
13:36:03:WU00:FS00:Core PID:9996
13:36:03:WU00:FS00:FahCore 0xa7 started
13:36:03:WU00:FS00:0xa7:*********************** Log Started 2020-05-02T13:36:03Z ***********************
13:36:03:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
13:36:03:WU00:FS00:0xa7:       Type: 0xa7
13:36:03:WU00:FS00:0xa7:       Core: Gromacs
13:36:03:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 3852 -checkpoint 15 -np 9
13:36:03:WU00:FS00:0xa7:************************************ CBang *************************************
13:36:03:WU00:FS00:0xa7:       Date: Oct 26 2019
13:36:03:WU00:FS00:0xa7:       Time: 01:38:25
13:36:03:WU00:FS00:0xa7:   Revision: c46a1a011a24143739ac7218c5a435f66777f62f
13:36:03:WU00:FS00:0xa7:     Branch: master
13:36:03:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
13:36:03:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
13:36:03:WU00:FS00:0xa7:   Platform: win32 10
13:36:03:WU00:FS00:0xa7:       Bits: 64
13:36:03:WU00:FS00:0xa7:       Mode: Release
13:36:03:WU00:FS00:0xa7:************************************ System ************************************
13:36:03:WU00:FS00:0xa7:        CPU: AMD Ryzen 5 1600 Six-Core Processor
13:36:03:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
13:36:03:WU00:FS00:0xa7:       CPUs: 12
13:36:03:WU00:FS00:0xa7:     Memory: 31.94GiB
13:36:03:WU00:FS00:0xa7:Free Memory: 25.07GiB
13:36:03:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
13:36:03:WU00:FS00:0xa7: OS Version: 6.2
13:36:03:WU00:FS00:0xa7:Has Battery: false
13:36:03:WU00:FS00:0xa7: On Battery: false
13:36:03:WU00:FS00:0xa7: UTC Offset: -4
13:36:03:WU00:FS00:0xa7:        PID: 9996
13:36:03:WU00:FS00:0xa7:        CWD: C:\Users\Sean\AppData\Roaming\FAHClient\work
13:36:04:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
13:36:04:WU00:FS00:0xa7:    Version: 0.0.18
13:36:04:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
13:36:04:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
13:36:04:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
13:36:04:WU00:FS00:0xa7:       Date: Oct 26 2019
13:36:04:WU00:FS00:0xa7:       Time: 01:52:30
13:36:04:WU00:FS00:0xa7:   Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
13:36:04:WU00:FS00:0xa7:     Branch: master
13:36:04:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
13:36:04:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
13:36:04:WU00:FS00:0xa7:   Platform: win32 10
13:36:04:WU00:FS00:0xa7:       Bits: 64
13:36:04:WU00:FS00:0xa7:       Mode: Release
13:36:04:WU00:FS00:0xa7:************************************ Build *************************************
13:36:04:WU00:FS00:0xa7:       SIMD: avx_256
13:36:04:WU00:FS00:0xa7:********************************************************************************
13:36:04:WU00:FS00:0xa7:Project: 16804 (Run 14, Clone 406, Gen 12)
13:36:04:WU00:FS00:0xa7:Unit: 0x0000000e82ed0b915e9df53e2f37ed91
13:36:04:WU00:FS00:0xa7:Reading tar file core.xml
13:36:04:WU00:FS00:0xa7:Reading tar file frame12.tpr
13:36:04:WU00:FS00:0xa7:Digital signatures verified
13:36:04:WU00:FS00:0xa7:Calling: mdrun -s frame12.tpr -o frame12.trr -cpt 15 -nt 9
13:36:04:WU00:FS00:0xa7:Steps: first=6000000 total=500000
13:36:05:WU00:FS00:0xa7:Completed 1 out of 500000 steps (0%)
13:40:17:WU00:FS00:0xa7:Completed 5000 out of 500000 steps (1%)
13:44:28:WU00:FS00:0xa7:Completed 10000 out of 500000 steps (2%)
13:48:41:WU00:FS00:0xa7:Completed 15000 out of 500000 steps (3%)
13:52:53:WU00:FS00:0xa7:Completed 20000 out of 500000 steps (4%)
13:57:06:WU00:FS00:0xa7:Completed 25000 out of 500000 steps (5%)
14:01:18:WU00:FS00:0xa7:Completed 30000 out of 500000 steps (6%)
14:05:29:WU00:FS00:0xa7:Completed 35000 out of 500000 steps (7%)
14:09:42:WU00:FS00:0xa7:Completed 40000 out of 500000 steps (8%)
14:13:53:WU00:FS00:0xa7:Completed 45000 out of 500000 steps (9%)
14:18:09:WU00:FS00:0xa7:Completed 50000 out of 500000 steps (10%)
14:22:29:WU00:FS00:0xa7:Completed 55000 out of 500000 steps (11%)
14:26:47:WU00:FS00:0xa7:Completed 60000 out of 500000 steps (12%)
14:31:08:WU00:FS00:0xa7:Completed 65000 out of 500000 steps (13%)
14:35:30:WU00:FS00:0xa7:Completed 70000 out of 500000 steps (14%)
14:39:43:WU00:FS00:0xa7:Completed 75000 out of 500000 steps (15%)
14:43:57:WU00:FS00:0xa7:Completed 80000 out of 500000 steps (16%)
14:48:10:WU00:FS00:0xa7:Completed 85000 out of 500000 steps (17%)
14:52:23:WU00:FS00:0xa7:Completed 90000 out of 500000 steps (18%)
14:56:36:WU00:FS00:0xa7:Completed 95000 out of 500000 steps (19%)
15:00:48:WU00:FS00:0xa7:Completed 100000 out of 500000 steps (20%)
15:05:00:WU00:FS00:0xa7:Completed 105000 out of 500000 steps (21%)
15:09:13:WU00:FS00:0xa7:Completed 110000 out of 500000 steps (22%)
15:13:25:WU00:FS00:0xa7:Completed 115000 out of 500000 steps (23%)
15:17:38:WU00:FS00:0xa7:Completed 120000 out of 500000 steps (24%)
15:21:51:WU00:FS00:0xa7:Completed 125000 out of 500000 steps (25%)
15:26:03:WU00:FS00:0xa7:Completed 130000 out of 500000 steps (26%)
15:30:15:WU00:FS00:0xa7:Completed 135000 out of 500000 steps (27%)
15:34:27:WU00:FS00:0xa7:Completed 140000 out of 500000 steps (28%)
15:38:40:WU00:FS00:0xa7:Completed 145000 out of 500000 steps (29%)
15:42:52:WU00:FS00:0xa7:Completed 150000 out of 500000 steps (30%)
15:47:05:WU00:FS00:0xa7:Completed 155000 out of 500000 steps (31%)
15:51:17:WU00:FS00:0xa7:Completed 160000 out of 500000 steps (32%)
15:55:29:WU00:FS00:0xa7:Completed 165000 out of 500000 steps (33%)
15:59:41:WU00:FS00:0xa7:Completed 170000 out of 500000 steps (34%)
16:03:53:WU00:FS00:0xa7:Completed 175000 out of 500000 steps (35%)
16:08:05:WU00:FS00:0xa7:Completed 180000 out of 500000 steps (36%)
16:12:17:WU00:FS00:0xa7:Completed 185000 out of 500000 steps (37%)
16:16:29:WU00:FS00:0xa7:Completed 190000 out of 500000 steps (38%)
16:20:42:WU00:FS00:0xa7:Completed 195000 out of 500000 steps (39%)
16:24:55:WU00:FS00:0xa7:Completed 200000 out of 500000 steps (40%)
16:29:07:WU00:FS00:0xa7:Completed 205000 out of 500000 steps (41%)
16:33:19:WU00:FS00:0xa7:Completed 210000 out of 500000 steps (42%)
16:37:30:WU00:FS00:0xa7:Completed 215000 out of 500000 steps (43%)
16:41:38:WU00:FS00:0xa7:Completed 220000 out of 500000 steps (44%)
16:45:48:WU00:FS00:0xa7:Completed 225000 out of 500000 steps (45%)
16:49:57:WU00:FS00:0xa7:Completed 230000 out of 500000 steps (46%)
16:54:07:WU00:FS00:0xa7:Completed 235000 out of 500000 steps (47%)
16:58:16:WU00:FS00:0xa7:Completed 240000 out of 500000 steps (48%)
16:58:23:WARNING:WU00:FS00:FahCore returned an unknown error code which probably indicates that it crashed
16:58:23:WARNING:WU00:FS00:FahCore returned: UNKNOWN_ENUM (-1073741819 = 0xc0000005)
16:58:23:WU00:FS00:Starting
16:58:23:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Sean\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/avx/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 20764 -checkpoint 15 -np 9
16:58:23:WU00:FS00:Started FahCore on PID 7408
16:58:23:WU00:FS00:Core PID:18852
16:58:23:WU00:FS00:FahCore 0xa7 started
16:58:23:WU00:FS00:0xa7:*********************** Log Started 2020-05-02T16:58:23Z ***********************
16:58:23:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
16:58:23:WU00:FS00:0xa7:       Type: 0xa7
16:58:23:WU00:FS00:0xa7:       Core: Gromacs
16:58:23:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 7408 -checkpoint 15 -np 9
16:58:23:WU00:FS00:0xa7:************************************ CBang *************************************
16:58:23:WU00:FS00:0xa7:       Date: Oct 26 2019
16:58:23:WU00:FS00:0xa7:       Time: 01:38:25
16:58:23:WU00:FS00:0xa7:   Revision: c46a1a011a24143739ac7218c5a435f66777f62f
16:58:23:WU00:FS00:0xa7:     Branch: master
16:58:23:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
16:58:23:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
16:58:23:WU00:FS00:0xa7:   Platform: win32 10
16:58:23:WU00:FS00:0xa7:       Bits: 64
16:58:23:WU00:FS00:0xa7:       Mode: Release
16:58:23:WU00:FS00:0xa7:************************************ System ************************************
16:58:23:WU00:FS00:0xa7:        CPU: AMD Ryzen 5 1600 Six-Core Processor
16:58:24:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
16:58:24:WU00:FS00:0xa7:       CPUs: 12
16:58:24:WU00:FS00:0xa7:     Memory: 31.94GiB
16:58:24:WU00:FS00:0xa7:Free Memory: 24.47GiB
16:58:24:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
16:58:24:WU00:FS00:0xa7: OS Version: 6.2
16:58:24:WU00:FS00:0xa7:Has Battery: false
16:58:24:WU00:FS00:0xa7: On Battery: false
16:58:24:WU00:FS00:0xa7: UTC Offset: -4
16:58:24:WU00:FS00:0xa7:        PID: 18852
16:58:24:WU00:FS00:0xa7:        CWD: C:\Users\Sean\AppData\Roaming\FAHClient\work
16:58:24:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
16:58:24:WU00:FS00:0xa7:    Version: 0.0.18
16:58:24:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:58:24:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
16:58:24:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
16:58:24:WU00:FS00:0xa7:       Date: Oct 26 2019
16:58:24:WU00:FS00:0xa7:       Time: 01:52:30
16:58:24:WU00:FS00:0xa7:   Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
16:58:24:WU00:FS00:0xa7:     Branch: master
16:58:24:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
16:58:24:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
16:58:24:WU00:FS00:0xa7:   Platform: win32 10
16:58:24:WU00:FS00:0xa7:       Bits: 64
16:58:24:WU00:FS00:0xa7:       Mode: Release
16:58:24:WU00:FS00:0xa7:************************************ Build *************************************
16:58:24:WU00:FS00:0xa7:       SIMD: avx_256
16:58:24:WU00:FS00:0xa7:********************************************************************************
16:58:24:WU00:FS00:0xa7:Project: 16804 (Run 14, Clone 406, Gen 12)
16:58:24:WU00:FS00:0xa7:Unit: 0x0000000e82ed0b915e9df53e2f37ed91
16:58:24:WU00:FS00:0xa7:Digital signatures verified
16:58:24:WU00:FS00:0xa7:Calling: mdrun -s frame12.tpr -o frame12.trr -cpi state.cpt -cpt 15 -nt 9
16:58:24:WU00:FS00:0xa7:Steps: first=6000000 total=500000
16:58:25:WU00:FS00:0xa7:Completed 231376 out of 500000 steps (46%)
17:01:26:WU00:FS00:0xa7:Completed 235000 out of 500000 steps (47%)
17:05:35:WU00:FS00:0xa7:Completed 240000 out of 500000 steps (48%)
17:09:45:WU00:FS00:0xa7:Completed 245000 out of 500000 steps (49%)
17:13:55:WU00:FS00:0xa7:Completed 250000 out of 500000 steps (50%)
17:16:21:WARNING:WU00:FS00:FahCore returned an unknown error code which probably indicates that it crashed
17:16:21:WARNING:WU00:FS00:FahCore returned: UNKNOWN_ENUM (-1073741819 = 0xc0000005)
17:16:21:WU00:FS00:Starting
17:16:21:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Sean\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/avx/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 20764 -checkpoint 15 -np 9
17:16:21:WU00:FS00:Started FahCore on PID 14916
17:16:21:WU00:FS00:Core PID:4184
17:16:21:WU00:FS00:FahCore 0xa7 started
17:16:22:WU00:FS00:0xa7:*********************** Log Started 2020-05-02T17:16:21Z ***********************
17:16:22:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
17:16:22:WU00:FS00:0xa7:       Type: 0xa7
17:16:22:WU00:FS00:0xa7:       Core: Gromacs
17:16:22:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 14916 -checkpoint 15 -np
17:16:22:WU00:FS00:0xa7:             9
17:16:22:WU00:FS00:0xa7:************************************ CBang *************************************
17:16:22:WU00:FS00:0xa7:       Date: Oct 26 2019
17:16:22:WU00:FS00:0xa7:       Time: 01:38:25
17:16:22:WU00:FS00:0xa7:   Revision: c46a1a011a24143739ac7218c5a435f66777f62f
17:16:22:WU00:FS00:0xa7:     Branch: master
17:16:22:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
17:16:22:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
17:16:22:WU00:FS00:0xa7:   Platform: win32 10
17:16:22:WU00:FS00:0xa7:       Bits: 64
17:16:22:WU00:FS00:0xa7:       Mode: Release
17:16:22:WU00:FS00:0xa7:************************************ System ************************************
17:16:22:WU00:FS00:0xa7:        CPU: AMD Ryzen 5 1600 Six-Core Processor
17:16:22:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
17:16:22:WU00:FS00:0xa7:       CPUs: 12
17:16:22:WU00:FS00:0xa7:     Memory: 31.94GiB
17:16:22:WU00:FS00:0xa7:Free Memory: 24.33GiB
17:16:22:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
17:16:22:WU00:FS00:0xa7: OS Version: 6.2
17:16:22:WU00:FS00:0xa7:Has Battery: false
17:16:22:WU00:FS00:0xa7: On Battery: false
17:16:22:WU00:FS00:0xa7: UTC Offset: -4
17:16:22:WU00:FS00:0xa7:        PID: 4184
17:16:22:WU00:FS00:0xa7:        CWD: C:\Users\Sean\AppData\Roaming\FAHClient\work
17:16:22:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
17:16:22:WU00:FS00:0xa7:    Version: 0.0.18
17:16:22:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:16:22:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
17:16:22:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
17:16:22:WU00:FS00:0xa7:       Date: Oct 26 2019
17:16:22:WU00:FS00:0xa7:       Time: 01:52:30
17:16:22:WU00:FS00:0xa7:   Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
17:16:22:WU00:FS00:0xa7:     Branch: master
17:16:22:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
17:16:22:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
17:16:22:WU00:FS00:0xa7:   Platform: win32 10
17:16:22:WU00:FS00:0xa7:       Bits: 64
17:16:22:WU00:FS00:0xa7:       Mode: Release
17:16:22:WU00:FS00:0xa7:************************************ Build *************************************
17:16:22:WU00:FS00:0xa7:       SIMD: avx_256
17:16:22:WU00:FS00:0xa7:********************************************************************************
17:16:22:WU00:FS00:0xa7:Project: 16804 (Run 14, Clone 406, Gen 12)
17:16:22:WU00:FS00:0xa7:Unit: 0x0000000e82ed0b915e9df53e2f37ed91
17:16:22:WU00:FS00:0xa7:Digital signatures verified
17:16:22:WU00:FS00:0xa7:Calling: mdrun -s frame12.tpr -o frame12.trr -cpi state.cpt -cpt 15 -nt 9
17:16:22:WU00:FS00:0xa7:Steps: first=6000000 total=500000
17:16:23:WU00:FS00:0xa7:Completed 249426 out of 500000 steps (49%)
17:16:52:WU00:FS00:0xa7:Completed 250000 out of 500000 steps (50%)
17:21:02:WU00:FS00:0xa7:Completed 255000 out of 500000 steps (51%)
17:25:12:WU00:FS00:0xa7:Completed 260000 out of 500000 steps (52%)
17:29:21:WU00:FS00:0xa7:Completed 265000 out of 500000 steps (53%)
17:33:36:WU00:FS00:0xa7:Completed 270000 out of 500000 steps (54%)
17:37:46:WU00:FS00:0xa7:Completed 275000 out of 500000 steps (55%)
17:41:56:WU00:FS00:0xa7:Completed 280000 out of 500000 steps (56%)
17:46:05:WU00:FS00:0xa7:Completed 285000 out of 500000 steps (57%)
17:50:16:WU00:FS00:0xa7:Completed 290000 out of 500000 steps (58%)
17:54:26:WU00:FS00:0xa7:Completed 295000 out of 500000 steps (59%)
17:58:36:WU00:FS00:0xa7:Completed 300000 out of 500000 steps (60%)
18:02:44:WU00:FS00:0xa7:Completed 305000 out of 500000 steps (61%)
18:06:53:WU00:FS00:0xa7:Completed 310000 out of 500000 steps (62%)
18:11:04:WU00:FS00:0xa7:Completed 315000 out of 500000 steps (63%)
18:15:14:WU00:FS00:0xa7:Completed 320000 out of 500000 steps (64%)
18:19:25:WU00:FS00:0xa7:Completed 325000 out of 500000 steps (65%)
18:23:36:WU00:FS00:0xa7:Completed 330000 out of 500000 steps (66%)
******************************* Date: 2020-05-02 *******************************
18:27:47:WU00:FS00:0xa7:Completed 335000 out of 500000 steps (67%)
18:31:58:WU00:FS00:0xa7:Completed 340000 out of 500000 steps (68%)
18:36:08:WU00:FS00:0xa7:Completed 345000 out of 500000 steps (69%)
18:40:18:WU00:FS00:0xa7:Completed 350000 out of 500000 steps (70%)
18:44:29:WU00:FS00:0xa7:Completed 355000 out of 500000 steps (71%)
18:48:41:WU00:FS00:0xa7:Completed 360000 out of 500000 steps (72%)
18:52:51:WU00:FS00:0xa7:Completed 365000 out of 500000 steps (73%)
18:57:02:WU00:FS00:0xa7:Completed 370000 out of 500000 steps (74%)
19:01:13:WU00:FS00:0xa7:Completed 375000 out of 500000 steps (75%)
19:05:24:WU00:FS00:0xa7:Completed 380000 out of 500000 steps (76%)
19:09:34:WU00:FS00:0xa7:Completed 385000 out of 500000 steps (77%)
19:13:45:WU00:FS00:0xa7:Completed 390000 out of 500000 steps (78%)
19:17:57:WU00:FS00:0xa7:Completed 395000 out of 500000 steps (79%)
19:22:09:WU00:FS00:0xa7:Completed 400000 out of 500000 steps (80%)
19:26:20:WU00:FS00:0xa7:Completed 405000 out of 500000 steps (81%)
19:30:31:WU00:FS00:0xa7:Completed 410000 out of 500000 steps (82%)
19:34:42:WU00:FS00:0xa7:Completed 415000 out of 500000 steps (83%)
19:38:53:WU00:FS00:0xa7:Completed 420000 out of 500000 steps (84%)
19:43:04:WU00:FS00:0xa7:Completed 425000 out of 500000 steps (85%)
19:47:14:WU00:FS00:0xa7:Completed 430000 out of 500000 steps (86%)
19:51:25:WU00:FS00:0xa7:Completed 435000 out of 500000 steps (87%)
19:55:36:WU00:FS00:0xa7:Completed 440000 out of 500000 steps (88%)
19:59:47:WU00:FS00:0xa7:Completed 445000 out of 500000 steps (89%)
20:03:57:WU00:FS00:0xa7:Completed 450000 out of 500000 steps (90%)
20:08:08:WU00:FS00:0xa7:Completed 455000 out of 500000 steps (91%)
20:12:18:WU00:FS00:0xa7:Completed 460000 out of 500000 steps (92%)
20:16:30:WU00:FS00:0xa7:Completed 465000 out of 500000 steps (93%)
20:20:41:WU00:FS00:0xa7:Completed 470000 out of 500000 steps (94%)
20:24:52:WU00:FS00:0xa7:Completed 475000 out of 500000 steps (95%)
20:29:02:WU00:FS00:0xa7:Completed 480000 out of 500000 steps (96%)
20:33:16:WU00:FS00:0xa7:Completed 485000 out of 500000 steps (97%)
20:37:26:WU00:FS00:0xa7:Completed 490000 out of 500000 steps (98%)
20:41:37:WU00:FS00:0xa7:Completed 495000 out of 500000 steps (99%)
20:44:14:WARNING:WU00:FS00:FahCore returned an unknown error code which probably indicates that it crashed
20:44:14:WARNING:WU00:FS00:FahCore returned: UNKNOWN_ENUM (-1073741819 = 0xc0000005)
23:38:10:WU00:FS00:Starting
23:38:10:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Sean\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/avx/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 20764 -checkpoint 15 -np 9
23:38:10:WU00:FS00:Started FahCore on PID 6160
23:38:10:WU00:FS00:Core PID:19184
23:38:10:WU00:FS00:FahCore 0xa7 started
23:38:10:WU00:FS00:0xa7:*********************** Log Started 2020-05-02T23:38:10Z ***********************
23:38:10:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
23:38:10:WU00:FS00:0xa7:       Type: 0xa7
23:38:10:WU00:FS00:0xa7:       Core: Gromacs
23:38:10:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 6160 -checkpoint 15 -np 9
23:38:10:WU00:FS00:0xa7:************************************ CBang *************************************
23:38:10:WU00:FS00:0xa7:       Date: Oct 26 2019
23:38:10:WU00:FS00:0xa7:       Time: 01:38:25
23:38:10:WU00:FS00:0xa7:   Revision: c46a1a011a24143739ac7218c5a435f66777f62f
23:38:10:WU00:FS00:0xa7:     Branch: master
23:38:10:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
23:38:10:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
23:38:10:WU00:FS00:0xa7:   Platform: win32 10
23:38:10:WU00:FS00:0xa7:       Bits: 64
23:38:10:WU00:FS00:0xa7:       Mode: Release
23:38:10:WU00:FS00:0xa7:************************************ System ************************************
23:38:10:WU00:FS00:0xa7:        CPU: AMD Ryzen 5 1600 Six-Core Processor
23:38:10:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
23:38:10:WU00:FS00:0xa7:       CPUs: 12
23:38:10:WU00:FS00:0xa7:     Memory: 31.94GiB
23:38:10:WU00:FS00:0xa7:Free Memory: 21.25GiB
23:38:10:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
23:38:10:WU00:FS00:0xa7: OS Version: 6.2
23:38:10:WU00:FS00:0xa7:Has Battery: false
23:38:10:WU00:FS00:0xa7: On Battery: false
23:38:10:WU00:FS00:0xa7: UTC Offset: -4
23:38:10:WU00:FS00:0xa7:        PID: 19184
23:38:10:WU00:FS00:0xa7:        CWD: C:\Users\Sean\AppData\Roaming\FAHClient\work
23:38:10:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
23:38:10:WU00:FS00:0xa7:    Version: 0.0.18
23:38:10:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
23:38:10:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
23:38:10:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
23:38:10:WU00:FS00:0xa7:       Date: Oct 26 2019
23:38:10:WU00:FS00:0xa7:       Time: 01:52:30
23:38:10:WU00:FS00:0xa7:   Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
23:38:10:WU00:FS00:0xa7:     Branch: master
23:38:10:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
23:38:10:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
23:38:10:WU00:FS00:0xa7:   Platform: win32 10
23:38:10:WU00:FS00:0xa7:       Bits: 64
23:38:10:WU00:FS00:0xa7:       Mode: Release
23:38:10:WU00:FS00:0xa7:************************************ Build *************************************
23:38:10:WU00:FS00:0xa7:       SIMD: avx_256
23:38:10:WU00:FS00:0xa7:********************************************************************************
23:38:10:WU00:FS00:0xa7:Project: 16804 (Run 14, Clone 406, Gen 12)
23:38:10:WU00:FS00:0xa7:Unit: 0x0000000e82ed0b915e9df53e2f37ed91
23:38:10:WU00:FS00:0xa7:Digital signatures verified
23:38:10:WU00:FS00:0xa7:Calling: mdrun -s frame12.tpr -o frame12.trr -cpi state.cpt -cpt 15 -nt 9
23:38:10:WU00:FS00:0xa7:Steps: first=6000000 total=500000
23:38:12:WU00:FS00:0xa7:Completed 482826 out of 500000 steps (96%)
23:40:40:WU00:FS00:0xa7:Completed 485000 out of 500000 steps (97%)
23:46:08:WU00:FS00:0xa7:Completed 490000 out of 500000 steps (98%)
23:51:52:WU00:FS00:0xa7:Completed 495000 out of 500000 steps (99%)
23:52:05:WARNING:WU00:FS00:FahCore returned an unknown error code which probably indicates that it crashed
23:52:05:WARNING:WU00:FS00:FahCore returned: UNKNOWN_ENUM (-1073741819 = 0xc0000005)
******************************* Date: 2020-05-03 *******************************
Top
Joe_H
Site Admin
Posts: 7870
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Project 16804 (Run 14, Clone 406, Gen 12) crashing at en

Post by Joe_H »

The folding client is passing through an error from the Windows OS. That particular error code indicates an issue accessing the filesystem. Do you have sufficient free space for the works files and checkpoint files? Can you check for other filesystem problems?
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Top
Smith6612
Posts: 13
Joined: Fri Jan 31, 2014 1:41 pm

Re: Project 16804 (Run 14, Clone 406, Gen 12) crashing at en

Post by Smith6612 »

Just ran chkdsk to make sure, and it is not reporting any errors with the file system. The drive the work folder for this work unit is on has 82GB free (256GB SSD). Drive S.M.A.R.T checks back clean, no read errors, bad blocks, or relocated blocks. Also checked file / folder permissions, and they are inheriting from the parent which has other work units in it which are downloading/folding/uploading without issue.

I did find this in the event viewer from the last crash:

Code: Select all

Faulting application name: FahCore_a7.exe, version: 0.0.0.0, time stamp: 0x5db40a96
Faulting module name: FahCore_a7.exe, version: 0.0.0.0, time stamp: 0x5db40a96
Exception code: 0xc0000005
Fault offset: 0x0000000000a7a558
Faulting process id: 0x4af0
Faulting application start time: 0x01d620dabd407b00
Faulting application path: C:\Users\Sean\AppData\Roaming\FAHClient\cores\cores.foldingathome.org\v7\win\64bit\avx\Core_a7.fah\FahCore_a7.exe
Faulting module path: C:\Users\Sean\AppData\Roaming\FAHClient\cores\cores.foldingathome.org\v7\win\64bit\avx\Core_a7.fah\FahCore_a7.exe
Report Id: 9b2d5ffb-2317-4e35-9bee-708d98131acc
Faulting package full name: 
Faulting package-relative application ID:
I'm seeing mention that this might be due to Data Execution Prevention because of an access violation, which is enabled on the machine and set to cover all programs (not just essential Windows programs). Although if DEP is messing something up, this would be the first time in years that I've seen such a thing happen with Folding@Home on this particular install.
Top
Smith6612
Posts: 13
Joined: Fri Jan 31, 2014 1:41 pm

Re: Project 16804 (Run 14, Clone 406, Gen 12) crashing at en

Post by Smith6612 »

I checked through the historica logs. Looks like this crashes periodically with the same exception code. I can go a few days or a week of 24/7 Folding without getting that exception. Let me exclude the FahCore_a7.exe binary and see if that helps.

EDIT: Nope. This work unit just bailed out after moving 2%.

Code: Select all

*********************** Log Started 2020-05-03T03:44:02Z ***********************
03:44:02:WU00:FS00:Starting
03:44:03:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Sean\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/avx/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 22352 -checkpoint 15 -np 9
03:44:03:WU00:FS00:Started FahCore on PID 18024
03:44:03:WU00:FS00:Core PID:16036
03:44:03:WU00:FS00:FahCore 0xa7 started
03:44:03:WU00:FS00:0xa7:*********************** Log Started 2020-05-03T03:44:03Z ***********************
03:44:03:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
03:44:03:WU00:FS00:0xa7:       Type: 0xa7
03:44:03:WU00:FS00:0xa7:       Core: Gromacs
03:44:03:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 18024 -checkpoint 15 -np
03:44:03:WU00:FS00:0xa7:             9
03:44:03:WU00:FS00:0xa7:************************************ CBang *************************************
03:44:03:WU00:FS00:0xa7:       Date: Oct 26 2019
03:44:03:WU00:FS00:0xa7:       Time: 01:38:25
03:44:03:WU00:FS00:0xa7:   Revision: c46a1a011a24143739ac7218c5a435f66777f62f
03:44:03:WU00:FS00:0xa7:     Branch: master
03:44:03:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
03:44:03:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
03:44:03:WU00:FS00:0xa7:   Platform: win32 10
03:44:03:WU00:FS00:0xa7:       Bits: 64
03:44:03:WU00:FS00:0xa7:       Mode: Release
03:44:03:WU00:FS00:0xa7:************************************ System ************************************
03:44:03:WU00:FS00:0xa7:        CPU: AMD Ryzen 5 1600 Six-Core Processor
03:44:03:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
03:44:03:WU00:FS00:0xa7:       CPUs: 12
03:44:03:WU00:FS00:0xa7:     Memory: 31.94GiB
03:44:03:WU00:FS00:0xa7:Free Memory: 25.20GiB
03:44:03:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
03:44:03:WU00:FS00:0xa7: OS Version: 6.2
03:44:03:WU00:FS00:0xa7:Has Battery: false
03:44:03:WU00:FS00:0xa7: On Battery: false
03:44:03:WU00:FS00:0xa7: UTC Offset: -4
03:44:03:WU00:FS00:0xa7:        PID: 16036
03:44:03:WU00:FS00:0xa7:        CWD: C:\Users\Sean\AppData\Roaming\FAHClient\work
03:44:03:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
03:44:03:WU00:FS00:0xa7:    Version: 0.0.18
03:44:03:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
03:44:03:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
03:44:03:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
03:44:03:WU00:FS00:0xa7:       Date: Oct 26 2019
03:44:03:WU00:FS00:0xa7:       Time: 01:52:30
03:44:03:WU00:FS00:0xa7:   Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
03:44:03:WU00:FS00:0xa7:     Branch: master
03:44:03:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
03:44:03:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
03:44:03:WU00:FS00:0xa7:   Platform: win32 10
03:44:03:WU00:FS00:0xa7:       Bits: 64
03:44:03:WU00:FS00:0xa7:       Mode: Release
03:44:03:WU00:FS00:0xa7:************************************ Build *************************************
03:44:03:WU00:FS00:0xa7:       SIMD: avx_256
03:44:03:WU00:FS00:0xa7:********************************************************************************
03:44:03:WU00:FS00:0xa7:Project: 16804 (Run 14, Clone 406, Gen 12)
03:44:03:WU00:FS00:0xa7:Unit: 0x0000000e82ed0b915e9df53e2f37ed91
03:44:03:WU00:FS00:0xa7:Digital signatures verified
03:44:03:WU00:FS00:0xa7:Calling: mdrun -s frame12.tpr -o frame12.trr -cpi state.cpt -cpt 15 -nt 9
03:44:04:WU00:FS00:0xa7:Steps: first=6000000 total=500000
03:44:10:WU00:FS00:0xa7:Completed 482826 out of 500000 steps (96%)
03:46:35:WU00:FS00:0xa7:Completed 485000 out of 500000 steps (97%)
03:49:07:WU00:FS00:0xa7:ERROR:
03:49:07:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
03:49:08:WU00:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20191026-456f0d636-unknown
03:49:08:WU00:FS00:0xa7:ERROR:Source code file: C:\build\fah\core-a7-avx-release\windows-10-64bit-core-a7-avx-release\gromacs-core\build\gromacs\src\gromacs\mdlib\pme.c, line: 754
03:49:08:WU00:FS00:0xa7:ERROR:
03:49:08:WU00:FS00:0xa7:ERROR:Fatal error:
03:49:08:WU00:FS00:0xa7:ERROR:9 particles communicated to PME rank 5 are more than 2/3 times the cut-off out of the domain decomposition cell of their charge group in dimension y.
03:49:08:WU00:FS00:0xa7:ERROR:This usually means that your system is not well equilibrated.
03:49:08:WU00:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
03:49:08:WU00:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
03:49:08:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
03:49:14:WARNING:WU00:FS00:FahCore returned an unknown error code which probably indicates that it crashed
03:49:14:WARNING:WU00:FS00:FahCore returned: UNKNOWN_ENUM (-1073741819 = 0xc0000005)
03:49:14:WARNING:WU00:FS00:Too many errors, failing
03:49:14:WU00:FS00:Sending unit results: id:00 state:SEND error:FAILED project:16804 run:14 clone:406 gen:12 core:0xa7 unit:0x0000000e82ed0b915e9df53e2f37ed91
Top
PantherX
Site Moderator
Posts: 7020
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:
Contact PantherX

Re: Project 16804 (Run 14, Clone 406, Gen 12) crashing at en

Post by PantherX »

Do you have any software (Anti-virus/anti-malware/anti-spyware/anti-ransomware) that could be causing those issues? If you do, ensure that you setup the following folder to be excluded:
C:\Users\Sean\AppData\Roaming\FAHClient
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Top
Smith6612
Posts: 13
Joined: Fri Jan 31, 2014 1:41 pm

Re: Project 16804 (Run 14, Clone 406, Gen 12) crashing at en

Post by Smith6612 »

PantherX wrote:Do you have any software (Anti-virus/anti-malware/anti-spyware/anti-ransomware) that could be causing those issues? If you do, ensure that you setup the following folder to be excluded:
C:\Users\Sean\AppData\Roaming\FAHClient
Just Windows Defender. I'll add an exclusion to the folder and monitor.
Top
Smith6612
Posts: 13
Joined: Fri Jan 31, 2014 1:41 pm

Re: Project 16804 (Run 14, Clone 406, Gen 12) crashing at en

Post by Smith6612 »

Okay, just as an update. The machine has been folding for about four days now without the the Core crashing out. It definitely seems Windows Defender was causing problems. Good to know, given it's more advanced than it used to be in the past.
Top
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Project 16804 (Run 14, Clone 406, Gen 12) crashing at en

Post by bruce »

As was suggested earlier, error (0xc0000005) is a pass-through error from Windows. It's a memory violation error.

Perhaps the program ran out of RAM (including VRAM). Do you have a maximum set for your VRAM allocation? The first thing I'd do is add an unlimited overflow area on a second disk.

Traditionally, FAH has not made huge demands on RAM, but recently I've seen cases where one of the FAHCore programs neede a lot of memory, and when you add that to the memory requirements of whatever else you might be running, it can be a lot bigger than you planned for when you set up your paging area.

The other possiblity is that you might have memory chips that are overheating or having errors. I'd definitely run a good memory diagnostic overnight.
Top
Post Reply

Return to “Issues with a specific WU”