Core_22 Problem 14300 Stuck in Core Oudated

[gordonbb]

Just got a slot stuck with this message:

WARNING:WU01:FS03:FahCore returned: CORE_OUTDATED (110 = 0x6e)

Code: Select all

07:16:16:WU02:FS03:0x22:*********************** Log Started 2020-04-23T07:16:15Z ***********************
07:16:16:WU02:FS03:0x22:*************************** Core22 Folding@home Core ***************************
07:16:16:WU02:FS03:0x22:       Type: 0x22
07:16:16:WU02:FS03:0x22:       Core: Core22
07:16:16:WU02:FS03:0x22:    Website: https://foldingathome.org/
07:16:16:WU02:FS03:0x22:  Copyright: (c) 2009-2018 foldingathome.org
07:16:16:WU02:FS03:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
07:16:16:WU02:FS03:0x22:             <rafal.wiewiora@choderalab.org>
07:16:16:WU02:FS03:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 18499 -checkpoint 15
07:16:16:WU02:FS03:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 4 -cuda-device
07:16:16:WU02:FS03:0x22:             4 -gpu 4
07:16:16:WU02:FS03:0x22:     Config: <none>
07:16:16:WU02:FS03:0x22:************************************ Build *************************************
07:16:16:WU02:FS03:0x22:    Version: 0.0.2
07:16:16:WU02:FS03:0x22:       Date: Dec 6 2019
07:16:16:WU02:FS03:0x22:       Time: 21:20:17
07:16:16:WU02:FS03:0x22: Repository: Git
07:16:16:WU02:FS03:0x22:   Revision: f87d92b58abdf7e6bf2e173cfbc4dc3e837c7042
07:16:16:WU02:FS03:0x22:     Branch: core22
07:16:16:WU02:FS03:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
07:16:16:WU02:FS03:0x22:    Options: -std=gnu++98 -O3 -funroll-loops
07:16:16:WU02:FS03:0x22:   Platform: linux2 4.9.87-linuxkit-aufs
07:16:16:WU02:FS03:0x22:       Bits: 64
07:16:16:WU02:FS03:0x22:       Mode: Release
07:16:16:WU02:FS03:0x22:************************************ System ************************************
07:16:16:WU02:FS03:0x22:        CPU: AMD Ryzen 7 2700 Eight-Core Processor
07:16:16:WU02:FS03:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
07:16:16:WU02:FS03:0x22:       CPUs: 16
07:16:16:WU02:FS03:0x22:     Memory: 31.37GiB
07:16:16:WU02:FS03:0x22:Free Memory: 19.35GiB
07:16:16:WU02:FS03:0x22:    Threads: POSIX_THREADS
07:16:16:WU02:FS03:0x22: OS Version: 5.3
07:16:16:WU02:FS03:0x22:Has Battery: false
07:16:16:WU02:FS03:0x22: On Battery: false
07:16:16:WU02:FS03:0x22: UTC Offset: -4
07:16:16:WU02:FS03:0x22:        PID: 18503
07:16:16:WU02:FS03:0x22:        CWD: /var/lib/fahclient/work
07:16:16:WU02:FS03:0x22:         OS: Linux 5.3.0-46-generic x86_64
07:16:16:WU02:FS03:0x22:    OS Arch: AMD64
07:16:16:WU02:FS03:0x22:********************************************************************************
07:16:16:WU02:FS03:0x22:Project: 16434 (Run 299, Clone 0, Gen 2)
07:16:16:WU02:FS03:0x22:Unit: 0x0000000503854c135e9cbacc1393f318
07:16:16:WU02:FS03:0x22:Reading tar file core.xml
07:16:16:WU02:FS03:0x22:Reading tar file integrator.xml
07:16:16:WU02:FS03:0x22:Reading tar file state.xml
07:16:16:WU02:FS03:0x22:Reading tar file system.xml
07:16:16:WU02:FS03:0x22:Digital signatures verified
07:16:16:WU02:FS03:0x22:Folding@home GPU Core22 Folding@home Core
07:16:16:WU02:FS03:0x22:Version 0.0.2
07:16:21:WU05:FS03:Upload 12.62%
07:16:26:WU02:FS03:0x22:Completed 0 out of 2500000 steps (0%)
07:16:26:WU02:FS03:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
07:16:27:WU05:FS03:Upload 26.13%
07:16:33:WU05:FS03:Upload 39.08%
07:16:39:WU05:FS03:Upload 52.59%
07:16:45:WU05:FS03:Upload 65.32%
07:16:51:WU05:FS03:Upload 78.71%
07:16:57:WU05:FS03:Upload 92.22%
07:17:10:WU05:FS03:Upload complete
07:17:10:WU05:FS03:Server responded WORK_ACK (400)
07:17:10:WU05:FS03:Final credit estimate, 124089.00 points
07:17:10:WU05:FS03:Cleaning up
07:19:48:WU02:FS03:0x22:Completed 25000 out of 2500000 steps (1%)
07:23:09:WU02:FS03:0x22:Completed 50000 out of 2500000 steps (2%)
07:26:30:WU02:FS03:0x22:Completed 75000 out of 2500000 steps (3%)
07:29:50:WU02:FS03:0x22:Completed 100000 out of 2500000 steps (4%)
13:52:20:WU06:FS03:0x22:Completed 115000 out of 500000 steps (23%)
13:54:49:WU06:FS03:0x22:Completed 120000 out of 500000 steps (24%)
13:57:33:WU06:FS03:0x22:Completed 125000 out of 500000 steps (25%)
14:00:01:WU06:FS03:0x22:Completed 130000 out of 500000 steps (26%)
14:02:31:WU06:FS03:0x22:Completed 135000 out of 500000 steps (27%)
14:05:00:WU06:FS03:0x22:Completed 140000 out of 500000 steps (28%)
14:07:43:WU06:FS03:0x22:Completed 145000 out of 500000 steps (29%)
14:10:11:WU06:FS03:0x22:Completed 150000 out of 500000 steps (30%)
14:12:41:WU06:FS03:0x22:Completed 155000 out of 500000 steps (31%)
14:15:10:WU06:FS03:0x22:Completed 160000 out of 500000 steps (32%)
14:17:52:WU06:FS03:0x22:Completed 165000 out of 500000 steps (33%)
14:20:21:WU06:FS03:0x22:Completed 170000 out of 500000 steps (34%)
14:22:50:WU06:FS03:0x22:Completed 175000 out of 500000 steps (35%)
14:25:19:WU06:FS03:0x22:Completed 180000 out of 500000 steps (36%)
14:28:03:WU06:FS03:0x22:Completed 185000 out of 500000 steps (37%)
14:30:32:WU06:FS03:0x22:Completed 190000 out of 500000 steps (38%)
14:33:00:WU06:FS03:0x22:Completed 195000 out of 500000 steps (39%)
14:35:29:WU06:FS03:0x22:Completed 200000 out of 500000 steps (40%)
14:38:13:WU06:FS03:0x22:Completed 205000 out of 500000 steps (41%)
14:40:42:WU06:FS03:0x22:Completed 210000 out of 500000 steps (42%)
14:43:10:WU06:FS03:0x22:Completed 215000 out of 500000 steps (43%)
14:45:39:WU06:FS03:0x22:Completed 220000 out of 500000 steps (44%)
14:48:22:WU06:FS03:0x22:Completed 225000 out of 500000 steps (45%)
14:50:51:WU06:FS03:0x22:Completed 230000 out of 500000 steps (46%)
14:53:20:WU06:FS03:0x22:Completed 235000 out of 500000 steps (47%)
14:55:49:WU06:FS03:0x22:Completed 240000 out of 500000 steps (48%)
14:58:30:WU06:FS03:0x22:Completed 245000 out of 500000 steps (49%)
15:00:59:WU06:FS03:0x22:Completed 250000 out of 500000 steps (50%)
15:03:29:WU06:FS03:0x22:Completed 255000 out of 500000 steps (51%)
15:05:58:WU06:FS03:0x22:Completed 260000 out of 500000 steps (52%)
15:08:40:WU06:FS03:0x22:Completed 265000 out of 500000 steps (53%)
15:11:10:WU06:FS03:0x22:Completed 270000 out of 500000 steps (54%)
15:13:38:WU06:FS03:0x22:Completed 275000 out of 500000 steps (55%)
15:16:07:WU06:FS03:0x22:Completed 280000 out of 500000 steps (56%)
15:18:50:WU06:FS03:0x22:Completed 285000 out of 500000 steps (57%)
15:21:19:WU06:FS03:0x22:Completed 290000 out of 500000 steps (58%)
15:23:48:WU06:FS03:0x22:Completed 295000 out of 500000 steps (59%)
15:26:17:WU06:FS03:0x22:Completed 300000 out of 500000 steps (60%)
15:29:00:WU06:FS03:0x22:Completed 305000 out of 500000 steps (61%)
15:31:29:WU06:FS03:0x22:Completed 310000 out of 500000 steps (62%)
15:33:58:WU06:FS03:0x22:Completed 315000 out of 500000 steps (63%)
15:36:27:WU06:FS03:0x22:Completed 320000 out of 500000 steps (64%)
15:39:10:WU06:FS03:0x22:Completed 325000 out of 500000 steps (65%)
15:41:39:WU06:FS03:0x22:Completed 330000 out of 500000 steps (66%)
15:44:09:WU06:FS03:0x22:Completed 335000 out of 500000 steps (67%)
15:46:37:WU06:FS03:0x22:Completed 340000 out of 500000 steps (68%)
15:49:20:WU06:FS03:0x22:Completed 345000 out of 500000 steps (69%)
15:51:49:WU06:FS03:0x22:Completed 350000 out of 500000 steps (70%)
15:54:18:WU06:FS03:0x22:Completed 355000 out of 500000 steps (71%)
15:56:47:WU06:FS03:0x22:Completed 360000 out of 500000 steps (72%)
15:59:28:WU06:FS03:0x22:Completed 365000 out of 500000 steps (73%)
16:01:57:WU06:FS03:0x22:Completed 370000 out of 500000 steps (74%)
16:04:26:WU06:FS03:0x22:Completed 375000 out of 500000 steps (75%)
16:06:55:WU06:FS03:0x22:Completed 380000 out of 500000 steps (76%)
16:09:37:WU06:FS03:0x22:Completed 385000 out of 500000 steps (77%)
16:12:06:WU06:FS03:0x22:Completed 390000 out of 500000 steps (78%)
16:14:34:WU06:FS03:0x22:Completed 395000 out of 500000 steps (79%)
16:17:03:WU06:FS03:0x22:Completed 400000 out of 500000 steps (80%)
16:19:47:WU06:FS03:0x22:Completed 405000 out of 500000 steps (81%)
16:22:16:WU06:FS03:0x22:Completed 410000 out of 500000 steps (82%)
16:24:45:WU06:FS03:0x22:Completed 415000 out of 500000 steps (83%)
16:27:14:WU06:FS03:0x22:Completed 420000 out of 500000 steps (84%)
16:29:57:WU06:FS03:0x22:Completed 425000 out of 500000 steps (85%)
16:32:26:WU06:FS03:0x22:Completed 430000 out of 500000 steps (86%)
16:34:55:WU06:FS03:0x22:Completed 435000 out of 500000 steps (87%)
16:37:23:WU06:FS03:0x22:Completed 440000 out of 500000 steps (88%)
16:40:06:WU06:FS03:0x22:Completed 445000 out of 500000 steps (89%)
16:42:35:WU06:FS03:0x22:Completed 450000 out of 500000 steps (90%)
16:45:04:WU06:FS03:0x22:Completed 455000 out of 500000 steps (91%)
16:47:33:WU06:FS03:0x22:Completed 460000 out of 500000 steps (92%)
16:50:15:WU06:FS03:0x22:Completed 465000 out of 500000 steps (93%)
16:52:44:WU06:FS03:0x22:Completed 470000 out of 500000 steps (94%)
16:55:13:WU06:FS03:0x22:Completed 475000 out of 500000 steps (95%)
16:57:42:WU06:FS03:0x22:Completed 480000 out of 500000 steps (96%)
17:00:24:WU06:FS03:0x22:Completed 485000 out of 500000 steps (97%)
17:02:53:WU06:FS03:0x22:Completed 490000 out of 500000 steps (98%)
17:05:22:WU06:FS03:0x22:Completed 495000 out of 500000 steps (99%)
17:05:22:WU01:FS03:Connecting to 65.254.110.245:80
17:05:22:WU01:FS03:Assigned to work server 18.188.125.154
17:05:22:WU01:FS03:Requesting new work unit for slot 03: RUNNING gpu:3:GP104 [GeForce GTX 1070 Ti] 8186 from 18.188.125.154
17:05:22:WU01:FS03:Connecting to 18.188.125.154:8080
17:05:24:WU01:FS03:Downloading 9.84MiB
17:05:26:WU01:FS03:Download complete
17:05:27:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13400 run:26 clone:0 gen:0 core:0x22 unit:0x0000000112bc7d9a5ea1be65d9bb4b42
17:07:50:WU06:FS03:0x22:Completed 500000 out of 500000 steps (100%)
17:08:04:WU06:FS03:0x22:Saving result file ../logfile_01.txt
17:08:04:WU06:FS03:0x22:Saving result file checkpointState.xml
17:08:18:WU06:FS03:0x22:Saving result file checkpt.crc
17:08:18:WU06:FS03:0x22:Saving result file positions.xtc
17:08:26:WU06:FS03:0x22:Saving result file science.log
17:08:26:WU06:FS03:0x22:Folding@home Core Shutdown: FINISHED_UNIT
17:08:27:WU06:FS03:FahCore returned: FINISHED_UNIT (100 = 0x64)
17:08:27:WU06:FS03:Sending unit results: id:06 state:SEND error:NO_ERROR project:14561 run:0 clone:673 gen:11 core:0x22 unit:0x0000001180fccb025e9615c0ae524fba
17:08:27:WU06:FS03:Uploading 66.59MiB to 128.252.203.2
17:08:27:WU06:FS03:Connecting to 128.252.203.2:8080
17:08:27:WU01:FS03:Starting
17:08:27:WU01:FS03:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/beta/Core_22.fah/FahCore_22 -dir 01 -suffix 01 -version 706 -lifeline 1642 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 4 -cuda-device 4 -gpu 4
17:08:27:WU01:FS03:Started FahCore on PID 13016
17:08:27:WU01:FS03:Core PID:13020
17:08:27:WU01:FS03:FahCore 0x22 started
17:08:27:WARNING:WU01:FS03:FahCore returned: CORE_OUTDATED (110 = 0x6e)
17:08:33:WU06:FS03:Upload 10.51%
17:08:39:WU06:FS03:Upload 21.31%
17:08:45:WU06:FS03:Upload 31.91%
17:08:51:WU06:FS03:Upload 42.24%
17:08:57:WU06:FS03:Upload 52.56%
17:09:03:WU06:FS03:Upload 63.26%
17:09:09:WU06:FS03:Upload 74.34%
17:09:15:WU06:FS03:Upload 85.04%
17:09:21:WU06:FS03:Upload 95.64%
17:09:35:WU06:FS03:Upload complete
17:09:35:WU06:FS03:Server responded WORK_ACK (400)
17:09:35:WU06:FS03:Final credit estimate, 154558.00 points
17:09:35:WU06:FS03:Cleaning up
17:36:26:FS03:Paused
17:36:29:FS03:Unpaused

This is on a six-gpu system and from previous posts it appears that the core will not update if other slots are using it. So I can either wait for the other five slots to finish or pause them and hope it doesn't corrupt the other WUs.

[gordonbb]

Pausing all 6 slots immediately put them all into Paused with updating core. Un-paused the affected slot and it immediately downloaded the new core and proceeded:

Code: Select all

17:47:05:FS03:Unpaused
17:47:05:WU01:FS03:Downloading core from http://cores.foldingathome.org/v7/lin/64bit/beta/Core_22.fah
17:47:05:WU01:FS03:Connecting to cores.foldingathome.org:80
17:47:05:WU01:FS03:FahCore 22: Downloading 3.59MiB
17:47:06:WU01:FS03:FahCore 22: Download complete
17:47:06:WU01:FS03:Valid core signature
17:47:06:WU01:FS03:Unpacked 9.32MiB to cores/cores.foldingathome.org/v7/lin/64bit/beta/Core_22.fah/FahCore_22
17:47:06:WU01:FS03:Starting
17:47:06:WU01:FS03:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/beta/Core_22.fah/FahCore_22 -dir 01 -suffix 01 -version 706 -lifeline 1642 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 4 -cuda-device 4 -gpu 4
17:47:06:WU01:FS03:Started FahCore on PID 29123
17:47:06:WU01:FS03:Core PID:29127
17:47:06:WU01:FS03:FahCore 0x22 started
17:47:07:WU01:FS03:0x22:*********************** Log Started 2020-04-23T17:47:06Z ***********************
17:47:07:WU01:FS03:0x22:*************************** Core22 Folding@home Core ***************************
17:47:07:WU01:FS03:0x22:       Type: 0x22
17:47:07:WU01:FS03:0x22:       Core: Core22
17:47:07:WU01:FS03:0x22:    Website: https://foldingathome.org/
17:47:07:WU01:FS03:0x22:  Copyright: (c) 2009-2018 foldingathome.org
17:47:07:WU01:FS03:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
17:47:07:WU01:FS03:0x22:             <rafal.wiewiora@choderalab.org>
17:47:07:WU01:FS03:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 29123 -checkpoint 15
17:47:07:WU01:FS03:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 4 -cuda-device
17:47:07:WU01:FS03:0x22:             4 -gpu 4
17:47:07:WU01:FS03:0x22:     Config: <none>
17:47:07:WU01:FS03:0x22:************************************ Build *************************************
17:47:07:WU01:FS03:0x22:    Version: 0.0.5
17:47:07:WU01:FS03:0x22:       Date: Apr 22 2020
17:47:07:WU01:FS03:0x22:       Time: 03:57:11
17:47:07:WU01:FS03:0x22: Repository: Git
17:47:07:WU01:FS03:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
17:47:07:WU01:FS03:0x22:     Branch: HEAD
17:47:07:WU01:FS03:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
17:47:07:WU01:FS03:0x22:    Options: -std=c++11 -O3 -funroll-loops
17:47:07:WU01:FS03:0x22:   Platform: linux2 4.19.76-linuxkit
17:47:07:WU01:FS03:0x22:       Bits: 64
17:47:07:WU01:FS03:0x22:       Mode: Release
17:47:07:WU01:FS03:0x22:************************************ System ************************************
17:47:07:WU01:FS03:0x22:        CPU: AMD Ryzen 7 2700 Eight-Core Processor
17:47:07:WU01:FS03:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
17:47:07:WU01:FS03:0x22:       CPUs: 16
17:47:07:WU01:FS03:0x22:     Memory: 31.37GiB
17:47:07:WU01:FS03:0x22:Free Memory: 27.75GiB
17:47:07:WU01:FS03:0x22:    Threads: POSIX_THREADS
17:47:07:WU01:FS03:0x22: OS Version: 5.3
17:47:07:WU01:FS03:0x22:Has Battery: false
17:47:07:WU01:FS03:0x22: On Battery: false
17:47:07:WU01:FS03:0x22: UTC Offset: -4
17:47:07:WU01:FS03:0x22:        PID: 29127
17:47:07:WU01:FS03:0x22:        CWD: /var/lib/fahclient/work
17:47:07:WU01:FS03:0x22:         OS: Linux 5.3.0-46-generic x86_64
17:47:07:WU01:FS03:0x22:    OS Arch: AMD64
17:47:07:WU01:FS03:0x22:********************************************************************************
17:47:07:WU01:FS03:0x22:Project: 13400 (Run 26, Clone 0, Gen 0)
17:47:07:WU01:FS03:0x22:Unit: 0x0000000112bc7d9a5ea1be65d9bb4b42
17:47:07:WU01:FS03:0x22:Reading tar file core.xml
17:47:07:WU01:FS03:0x22:Reading tar file integrator.xml
17:47:07:WU01:FS03:0x22:Reading tar file state.xml
17:47:07:WU01:FS03:0x22:Reading tar file system.xml
17:47:07:WU01:FS03:0x22:Digital signatures verified
17:47:07:WU01:FS03:0x22:Folding@home GPU Core22 Folding@home Core
17:47:07:WU01:FS03:0x22:Version 0.0.5
17:47:20:WU01:FS03:0x22:Completed 0 out of 2000000 steps (0%)
17:47:20:WU01:FS03:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900

and then un-paused the rest and they resumed where they left off.

[JohnChodera]

Thanks for the update! You caught us when we were deploying core22 0.0.5 to beta/adv today! Hope you weren't stuck for too long!

[tbonse]

00:41:05:WU03:FS01:Connecting to 65.254.110.245:80
00:41:05:WU03:FS01:Assigned to work server 18.188.125.154
00:41:05:WU03:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 18.188.125.154
00:41:05:WU03:FS01:Connecting to 18.188.125.154:8080
00:41:07:WU03:FS01:Downloading 9.31MiB
00:41:08:WU03:FS01:Download complete
00:41:08:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13400 run:73 clone:0 gen:1 core:0x22 unit:0x0000000112bc7d9a5ea1be650d394b33
00:57:30:WU03:FS01:Starting
00:57:30:WU03:FS01:Running FahCore: "D:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" D:\tbonse\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 03 -suffix 01 -version 706 -lifeline 10668 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
00:57:30:WU03:FS01:Started FahCore on PID 22060
00:57:30:WU03:FS01:Core PID:2924
00:57:30:WU03:FS01:FahCore 0x22 started
00:57:31:WARNING:WU03:FS01:FahCore returned: CORE_OUTDATED (110 = 0x6e)
******************************* Date: 2020-04-23 *******************************
00:41:05:WU03:FS01:Connecting to 65.254.110.245:80
00:41:05:WU03:FS01:Assigned to work server 18.188.125.154
00:41:05:WU03:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 18.188.125.154
00:41:05:WU03:FS01:Connecting to 18.188.125.154:8080
00:41:07:WU03:FS01:Downloading 9.31MiB
00:41:08:WU03:FS01:Download complete
00:41:08:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13400 run:73 clone:0 gen:1 core:0x22 unit:0x0000000112bc7d9a5ea1be650d394b33
00:57:30:WU03:FS01:Starting
00:57:30:WU03:FS01:Running FahCore: "D:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" D:\tbonse\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 03 -suffix 01 -version 706 -lifeline 10668 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
00:57:30:WU03:FS01:Started FahCore on PID 22060
00:57:30:WU03:FS01:Core PID:2924
00:57:30:WU03:FS01:FahCore 0x22 started
00:57:31:WARNING:WU03:FS01:FahCore returned: CORE_OUTDATED (110 = 0x6e)
01:07:57:WU03:FS01:Starting
01:07:57:WU03:FS01:Running FahCore: "D:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" D:\tbonse\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 03 -suffix 01 -version 706 -lifeline 10668 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
01:07:57:WU03:FS01:Started FahCore on PID 21524
01:07:57:WU03:FS01:Core PID:12092
01:07:57:WU03:FS01:FahCore 0x22 started
01:08:04:WARNING:WU03:FS01:FahCore returned: CORE_OUTDATED (110 = 0x6e)

[tbonse]

The 18.188.125.154 server is handing out nothing but bad jobs.

Code: Select all

*********************** Log Started 2020-04-24T01:26:35Z ***********************
01:26:43:WU01:FS01:Connecting to 65.254.110.245:80
01:26:43:WU01:FS01:Assigned to work server 18.188.125.154
01:26:43:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 18.188.125.154
01:26:43:WU01:FS01:Connecting to 18.188.125.154:8080
01:26:55:WU01:FS01:Downloading 9.84MiB
01:26:56:WU01:FS01:Download complete
01:26:57:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13400 run:58 clone:5 gen:0 core:0x22 unit:0x0000000012bc7d9a5ea1be659719271b
01:26:57:WU01:FS01:Starting
01:26:57:WU01:FS01:Running FahCore: "D:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" D:\tbonse\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 19676 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
01:26:57:WU01:FS01:Started FahCore on PID 9404
01:26:57:WU01:FS01:Core PID:12048
01:26:57:WU01:FS01:FahCore 0x22 started
01:26:58:WARNING:WU01:FS01:FahCore returned: CORE_OUTDATED (110 = 0x6e)

[tbonse]

It appears to be projected 13400 that is producing these bad cores as well.

[JohnChodera]

Thanks for the update. I'll pause that server (aws3.foldingathome.org) from ADVANCED until we can figure out what is going on.

Can you try removing your cores/ directory and see if it correctly auto-downloads core22 0.0.5?

[tbonse]

And again with another bad core for project 13400 served by 18.188.125.154.

I have to keep stopping my folding at home, deleting the job then restart, and I keep getting these garbage jobs that cannot be worked, will try 3 more time then i'll block 18.188.125.154.

Code: Select all

*********************** Log Started 2020-04-24T01:31:11Z ***********************
01:31:11:WU01:FS01:Cleaning up
01:31:49:WU01:FS01:Connecting to 65.254.110.245:80
01:31:49:WU01:FS01:Assigned to work server 18.188.125.154
01:31:49:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 18.188.125.154
01:31:49:WU01:FS01:Connecting to 18.188.125.154:8080
01:31:51:WU01:FS01:Downloading 9.84MiB
01:31:52:WU01:FS01:Download complete
01:31:52:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13400 run:25 clone:5 gen:0 core:0x22 unit:0x0000000112bc7d9a5ea1be65707cb80e
01:31:52:WU01:FS01:Starting
01:31:52:WU01:FS01:Running FahCore: "D:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" D:\tbonse\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 8032 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
01:31:52:WU01:FS01:Started FahCore on PID 21312
01:31:52:WU01:FS01:Core PID:23032
01:31:52:WU01:FS01:FahCore 0x22 started
01:31:53:WARNING:WU01:FS01:FahCore returned: CORE_OUTDATED (110 = 0x6e)

[tbonse]

I have other slots that are working fine, it was only project 13400 coming out of 18.188.125.154 that was causing issues.

I just got a new job from a different server and it works fine.

Code: Select all

*********************** Log Started 2020-04-24T01:34:37Z ***********************
01:34:37:WU01:FS01:Cleaning up
01:34:38:WU01:FS01:Connecting to 65.254.110.245:80
01:34:38:WU01:FS01:Assigned to work server 128.252.203.2
01:34:38:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 128.252.203.2
01:34:38:WU01:FS01:Connecting to 128.252.203.2:8080
*********************** Log Started 2020-04-24T01:34:37Z ***********************
01:34:37:WU01:FS01:Cleaning up
01:34:38:WU01:FS01:Connecting to 65.254.110.245:80
01:34:38:WU01:FS01:Assigned to work server 128.252.203.2
01:34:38:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 128.252.203.2
01:34:38:WU01:FS01:Connecting to 128.252.203.2:8080
01:36:46:WU01:FS01:Downloading 33.06MiB
01:36:49:WU01:FS01:Download complete
01:36:50:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14561 run:0 clone:315 gen:7 core:0x22 unit:0x0000000a80fccb025e9615c1df3bbb46
01:36:50:WU01:FS01:Starting
01:36:50:WU01:FS01:Running FahCore: "D:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" D:\tbonse\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 19332 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
01:36:50:WU01:FS01:Started FahCore on PID 21916
01:36:50:WU01:FS01:Core PID:7644
01:36:50:WU01:FS01:FahCore 0x22 started
01:36:50:WU01:FS01:0x22:*********************** Log Started 2020-04-24T01:36:50Z ***********************
01:36:50:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
01:36:50:WU01:FS01:0x22:       Type: 0x22
01:36:50:WU01:FS01:0x22:       Core: Core22
01:36:50:WU01:FS01:0x22:    Website: https://foldingathome.org/
01:36:50:WU01:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
01:36:50:WU01:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
01:36:50:WU01:FS01:0x22:             <rafal.wiewiora@choderalab.org>
01:36:50:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 21916 -checkpoint 15
01:36:50:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
01:36:50:WU01:FS01:0x22:             1 -gpu 1
01:36:50:WU01:FS01:0x22:     Config: <none>
01:36:50:WU01:FS01:0x22:************************************ Build *************************************
01:36:50:WU01:FS01:0x22:    Version: 0.0.2
01:36:50:WU01:FS01:0x22:       Date: Dec 6 2019
01:36:50:WU01:FS01:0x22:       Time: 21:30:31
01:36:50:WU01:FS01:0x22: Repository: Git
01:36:50:WU01:FS01:0x22:   Revision: abeb39247cc72df5af0f63723edafadb23d5dfbe
01:36:50:WU01:FS01:0x22:     Branch: HEAD
01:36:50:WU01:FS01:0x22:   Compiler: Visual C++ 2008
01:36:50:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
01:36:50:WU01:FS01:0x22:   Platform: win32 10
01:36:50:WU01:FS01:0x22:       Bits: 64
01:36:50:WU01:FS01:0x22:       Mode: Release
01:36:50:WU01:FS01:0x22:************************************ System ************************************
01:36:50:WU01:FS01:0x22:        CPU: AMD Ryzen 7 2700 Eight-Core Processor
01:36:50:WU01:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
01:36:50:WU01:FS01:0x22:       CPUs: 16
01:36:50:WU01:FS01:0x22:     Memory: 31.93GiB
01:36:50:WU01:FS01:0x22:Free Memory: 19.54GiB
01:36:50:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
01:36:50:WU01:FS01:0x22: OS Version: 6.2
01:36:50:WU01:FS01:0x22:Has Battery: false
01:36:50:WU01:FS01:0x22: On Battery: false
01:36:50:WU01:FS01:0x22: UTC Offset: -4
01:36:50:WU01:FS01:0x22:        PID: 7644
01:36:50:WU01:FS01:0x22:        CWD: D:\tbonse\AppData\Roaming\FAHClient\work
01:36:50:WU01:FS01:0x22:         OS: Windows 10 Pro
01:36:50:WU01:FS01:0x22:    OS Arch: AMD64
01:36:50:WU01:FS01:0x22:********************************************************************************
01:36:50:WU01:FS01:0x22:Project: 14561 (Run 0, Clone 315, Gen 7)
01:36:50:WU01:FS01:0x22:Unit: 0x0000000a80fccb025e9615c1df3bbb46
01:36:50:WU01:FS01:0x22:Reading tar file core.xml
01:36:50:WU01:FS01:0x22:Reading tar file integrator.xml
01:36:50:WU01:FS01:0x22:Reading tar file state.xml
01:36:57:WU01:FS01:0x22:Reading tar file system.xml
01:37:02:WU01:FS01:0x22:Digital signatures verified
01:37:02:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
01:37:02:WU01:FS01:0x22:Version 0.0.2
01:38:18:WU01:FS01:0x22:Completed 0 out of 500000 steps (0%)
01:38:18:WU01:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900

[schapman1978]

I've restarted the client and the usual troubleshooting steps etc., but this unit is stuck on a GPU and I can't move on to a working WU. Any idea guys?
Been doing this for half an hour or so, here's a clip of the log

Code: Select all

01:31:29:WU01:FS01:0x22:Project: 13400 (Run 139, Clone 3, Gen 0)
01:31:29:WU01:FS01:0x22:Unit: 0x0000000112bc7d9a5ea1be655bd96ea1
01:31:29:WU01:FS01:0x22:ERROR:110: Need version 0.0.5
01:31:29:WU01:FS01:0x22:Folding@home Core Shutdown: CORE_OUTDATED
01:31:30:WARNING:WU01:FS01:FahCore returned: CORE_OUTDATED (110 = 0x6e)
01:32:28:WU01:FS01:Downloading core from http://cores.foldingathome.org/v7/win/64bit/Core_22.fah
01:32:28:WU01:FS01:Connecting to cores.foldingathome.org:80
01:32:28:WU01:FS01:FahCore 22: Downloading 4.04MiB
01:32:29:WU01:FS01:FahCore 22: Download complete
01:32:29:WU01:FS01:Valid core signature
01:32:29:WARNING:WU01:FS01:FahCore has not changed since last download, aborting core update
01:32:29:WU01:FS01:Starting
01:32:29:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\crcha\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 632 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
01:32:29:WU01:FS01:Started FahCore on PID 12668
01:32:29:WU01:FS01:Core PID:13872
01:32:29:WU01:FS01:FahCore 0x22 started
01:32:30:WARNING:WU01:FS01:FahCore returned: CORE_OUTDATED (110 = 0x6e)
01:34:05:WU01:FS01:Downloading core from http://cores.foldingathome.org/v7/win/64bit/Core_22.fah
01:34:05:WU01:FS01:Connecting to cores.foldingathome.org:80
01:34:06:WU01:FS01:FahCore 22: Downloading 4.04MiB
01:34:06:WU01:FS01:FahCore 22: Download complete
01:34:06:WU01:FS01:Valid core signature
01:34:06:WARNING:WU01:FS01:FahCore has not changed since last download, aborting core update
01:34:06:WU01:FS01:Starting
01:34:06:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\crcha\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 632 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
01:34:06:WU01:FS01:Started FahCore on PID 1624
01:34:06:WU01:FS01:Core PID:13640
01:34:06:WU01:FS01:FahCore 0x22 started
01:34:07:WARNING:WU01:FS01:FahCore returned: CORE_OUTDATED (110 = 0x6e)
01:36:43:WU01:FS01:Downloading core from http://cores.foldingathome.org/v7/win/64bit/Core_22.fah
01:36:43:WU01:FS01:Connecting to cores.foldingathome.org:80
01:36:43:WU01:FS01:FahCore 22: Downloading 4.04MiB
01:36:43:WU01:FS01:FahCore 22: Download complete
01:36:43:WU01:FS01:Valid core signature
01:36:43:WARNING:WU01:FS01:FahCore has not changed since last download, aborting core update
01:36:43:WU01:FS01:Starting
01:36:43:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\crcha\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 632 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
01:36:43:WU01:FS01:Started FahCore on PID 5880
01:36:43:WU01:FS01:Core PID:2212
01:36:43:WU01:FS01:FahCore 0x22 started
01:36:44:WARNING:WU01:FS01:FahCore returned: CORE_OUTDATED (110 = 0x6e)

Mod Edit: Added Code Tags - PantherX

[tbonse]

So from the limited sample size, I'd say that there are still a few kinks to work out with Core22 0.0.5. Just in case it's not captured, I had already updated the client to the Windows version 7.6.9 of the F@H client. Similar version upgrades to 7.6.9 were also completed on 2 out of 3 of my Linux servers. The last one hasn't been updated yet due to time constraints.

[JimboPalmer]

To the best of my knowledge, 0.0.2 is the latest production version.

You aren't playing with Beta flags without being on the Beta team are you?

[bruce]

This was an issue for those participating in internal testing of an unreleased FAHCore. I'm sorry, but support in this forum is for WUs and FAHCores that have been released to the public (including beta testing, provided you apply for beta test registration)


CORRECTION: The Up-to-date FAHCore was prepared for distributeion at the time this was all happening. There's nothing wrong with the project. Please PAUSE it, wait a few bit, and then resume folding. A new FAHCore will download and you'll be fine.

[alien88]

My windows F@H client is folding CPU units fine, but for GPU I stopped getting them and I am in a loop of it trying to update the core but not succeeding. I've tried pausing/unpausing specific WUs, all WUs, rebooting, deleting the cores, but still struggling:

Code: Select all

01:57:28:WU00:FS01:Downloading core from http://cores.foldingathome.org/v7/win/64bit/Core_22.fah
01:57:28:WU00:FS01:Connecting to cores.foldingathome.org:80
01:57:29:WU00:FS01:FahCore 22: Downloading 4.04MiB
01:57:30:WU00:FS01:FahCore 22: Download complete
01:57:30:WU00:FS01:Valid core signature
01:57:30:WARNING:WU00:FS01:FahCore has not changed since last download, aborting core update
01:57:30:WU00:FS01:Starting
01:57:30:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:\Users\Mike Garrison\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe" -dir 00 -suffix 01 -version 706 -lifeline 17252 -checkpoint 20 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
01:57:30:WU00:FS01:Started FahCore on PID 4396
01:57:30:WU00:FS01:Core PID:19360
01:57:30:WU00:FS01:FahCore 0x22 started
01:57:30:WARNING:WU00:FS01:FahCore returned: CORE_OUTDATED (110 = 0x6e)

Config:

Code: Select all

01:57:42:<config>
01:57:42:  <!-- Folding Core -->
01:57:42:  <checkpoint v='20'/>
01:57:42:  <core-priority v='low'/>
01:57:42:
01:57:42:  <!-- Network -->
01:57:42:  <proxy v=':8080'/>
01:57:42:
01:57:42:  <!-- Slot Control -->
01:57:42:  <power v='full'/>
01:57:42:
01:57:42:  <!-- User Information -->
01:57:42:  <passkey v='*****'/>
01:57:42:  <team v='237601'/>
01:57:42:  <user v='Alien88'/>
01:57:42:
01:57:42:  <!-- Folding Slots -->
01:57:42:  <slot id='0' type='CPU'>
01:57:42:    <client-type v='advanced'/>
01:57:42:  </slot>
01:57:42:  <slot id='1' type='GPU'>
01:57:42:    <client-type v='advanced'/>
01:57:42:  </slot>
01:57:42:</config>

Any idea how I can get it out of this state? My linux machines all seem fine, so just this one Windows computer..

[tbonse]

This was an issue for those participating in internal testing of an unreleased FAHCore. I'm sorry, but support in this forum is for WUs and FAHCores that have been released to the public (including beta testing, provided you apply for beta test registration)

Discard that project.

Topic closed.

Bruce, that's just it, I don't have the beta flag set, which you can see from the output logs from my postings that this beta work got sent to a non-beta channel. Separately gordonbb's work is from the beta channel (same project none the less [13400]).