17424 r0 c1536 g151 - ERROR:Force RMSE

Moderators: Site Moderators, FAHC Science Team

Post Reply
n1np
Posts: 32
Joined: Sat Mar 14, 2009 2:16 pm
Hardware configuration: HP DL380g8 x12
Location: Virginia
Contact:

17424 r0 c1536 g151 - ERROR:Force RMSE

Post by n1np »

I saw this today, not seen before (ERROR:Force RMSE error of 6.46327 with threshold of 5):

Code: Select all

*********************** Log Started 2021-01-21T16:14:08Z ***********************
16:14:08:******************************* libFAH ********************************
16:14:08:           Date: Oct 20 2020
16:14:08:           Time: 20:36:39
16:14:08:       Revision: 5ca109d295a6245e2a2f590b3d0085ad5e567aeb
16:14:08:         Branch: master
16:14:08:       Compiler: GNU 8.3.0
16:14:08:        Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
16:14:08:                 -fdata-sections -O3 -funroll-loops -fno-pie
16:14:08:       Platform: linux2 5.8.0-1-amd64
16:14:08:           Bits: 64
16:14:08:           Mode: Release
16:14:08:****************************** FAHClient ******************************
16:14:08:        Version: 7.6.21
16:14:08:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:14:08:      Copyright: 2020 foldingathome.org
16:14:08:       Homepage: https://foldingathome.org/
16:14:08:           Date: Oct 20 2020
16:14:08:           Time: 20:39:00
16:14:08:       Revision: 6efbf0e138e22d3963e6a291f78dcb9c6422a278
16:14:08:         Branch: master
16:14:08:       Compiler: GNU 8.3.0
16:14:08:        Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
16:14:08:                 -fdata-sections -O3 -funroll-loops -fno-pie
16:14:08:       Platform: linux2 5.8.0-1-amd64
16:14:08:           Bits: 64
16:14:08:           Mode: Release
16:14:08:           Args: --child /etc/fahclient/config.xml --run-as fahclient
16:14:08:                 --pid-file=/var/run/fahclient.pid --daemon
16:14:08:         Config: /etc/fahclient/config.xml
16:14:08:******************************** CBang ********************************
16:14:08:           Date: Oct 20 2020
16:14:08:           Time: 18:37:59
16:14:08:       Revision: 7e4ce85225d7eaeb775e87c31740181ca603de60
16:14:08:         Branch: master
16:14:08:       Compiler: GNU 8.3.0
16:14:08:        Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
16:14:08:                 -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
16:14:08:       Platform: linux2 5.8.0-1-amd64
16:14:08:           Bits: 64
16:14:08:           Mode: Release
16:14:08:******************************* System ********************************
16:14:08:            CPU: Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
16:14:08:         CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
16:14:08:           CPUs: 32
16:14:08:         Memory: 62.87GiB
16:14:08:    Free Memory: 61.71GiB
16:14:08:        Threads: POSIX_THREADS
16:14:08:     OS Version: 5.4
16:14:08:    Has Battery: false
16:14:08:     On Battery: false
16:14:08:     UTC Offset: -5
16:14:08:            PID: 1594
16:14:08:            CWD: /var/lib/fahclient
16:14:08:             OS: Linux 5.4.0-62-generic x86_64
16:14:08:        OS Arch: AMD64
16:14:08:           GPUs: 2
16:14:08:          GPU 0: Bus:4 Slot:0 Func:0 NVIDIA:3 GK104 [Quadro K4000]
16:14:08:          GPU 1: Bus:33 Slot:0 Func:0 NVIDIA:3 GK104 [Quadro K4000]
16:14:08:  CUDA Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:3.0 Driver:11.0
16:14:08:  CUDA Device 1: Platform:0 Device:1 Bus:33 Slot:0 Compute:3.0 Driver:11.0
16:14:08:OpenCL Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:1.2 Driver:450.102
16:14:08:OpenCL Device 1: Platform:0 Device:1 Bus:33 Slot:0 Compute:1.2 Driver:450.102
16:14:08:***********************************************************************
16:14:08:<config>
16:14:08:  <!-- Client Control -->
16:14:08:  <fold-anon v='true'/>
16:14:08:
16:14:08:  <!-- HTTP Server -->
16:14:08:  <allow v='192.168.0.1/24'/>
16:14:08:
16:14:08:  <!-- Network -->
16:14:08:  <proxy v=':8080'/>
16:14:08:
16:14:08:  <!-- Remote Command Server -->
16:14:08:  <password v='*****'/>
16:14:08:
16:14:08:  <!-- User Information -->
16:14:08:  <passkey v='*****'/>
16:14:08:  <team v='12912'/>
16:14:08:  <user v='n1np'/>
16:14:08:
16:14:08:  <!-- Folding Slots -->
16:14:08:  <slot id='0' type='CPU'>
16:14:08:    <cpus v='28'/>
16:14:08:    <paused v='true'/>
16:14:08:  </slot>
16:14:08:  <slot id='1' type='GPU'>
16:14:08:    <paused v='true'/>
16:14:08:    <pci-bus v='4'/>
16:14:08:    <pci-slot v='0'/>
16:14:08:  </slot>
16:14:08:  <slot id='2' type='GPU'>
16:14:08:    <paused v='true'/>
16:14:08:    <pci-bus v='33'/>
16:14:08:    <pci-slot v='0'/>
16:14:08:  </slot>
16:14:08:</config>
16:14:08:Trying to access database...
16:14:08:Successfully acquired database lock
16:14:08:FS00:Initialized folding slot 00: cpu:28
16:14:08:FS01:Initialized folding slot 01: gpu:4:0 GK104 [Quadro K4000]
16:14:08:FS02:Initialized folding slot 02: gpu:33:0 GK104 [Quadro K4000]
16:16:08:FS00:Unpaused
16:16:08:FS01:Unpaused
16:16:08:FS02:Unpaused


**** EDIT ****

20:45:35:WU00:FS01:0x22:Completed 1250000 out of 1250000 steps (100%)
20:45:35:WU00:FS01:0x22:Average performance: 256.38 ns/day
20:45:35:WU00:FS01:0x22:Checkpoint completed at step 1250000
20:45:42:WU00:FS01:0x22:Saving result file ../logfile_01.txt
20:45:42:WU00:FS01:0x22:Saving result file checkpointIntegrator.xml.bz2
20:45:42:WU00:FS01:0x22:Saving result file checkpointState.xml.bz2
20:45:42:WU00:FS01:0x22:Saving result file positions.xtc
20:45:42:WU00:FS01:0x22:Saving result file science.log
20:45:42:WU00:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
20:45:43:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
20:45:43:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:17313 run:0 clone:1900 gen:133 core:0x22 unit:0x0000076c00000085000043a100000000
20:45:43:WU00:FS01:Uploading 3.81MiB to 140.163.4.200
20:45:43:WU00:FS01:Connecting to 140.163.4.200:8080
20:45:49:WU00:FS01:Upload 80.47%
20:45:51:WU00:FS01:Upload complete
20:45:51:WU00:FS01:Server responded WORK_ACK (400)
20:45:51:WU00:FS01:Final credit estimate, 13518.00 points
20:45:51:WU00:FS01:Cleaning up
20:46:57:WU04:FS02:0x22:Completed 2500000 out of 5000000 steps (50%)
20:46:57:WU04:FS02:0x22:Checkpoint completed at step 2500000
20:48:03:WU02:FS01:Connecting to assign1.foldingathome.org:80
20:48:03:WU02:FS01:Assigned to work server 206.223.170.146
20:48:03:WU02:FS01:Requesting new work unit for slot 01: gpu:4:0 GK104 [Quadro K4000] from 206.223.170.146
20:48:03:WU02:FS01:Connecting to 206.223.170.146:8080
20:48:04:WU02:FS01:Downloading 13.44MiB
20:48:10:WU02:FS01:Download 82.32%
20:48:11:WU02:FS01:Download complete
20:48:11:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:17424 run:0 clone:1536 gen:151 core:0x22 unit:0x00000600000000970000441000000000
20:48:11:WU02:FS01:Starting
20:48:11:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/FahCore_22 -dir 02 -suffix 01 -version 706 -lifeline 1594 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
20:48:11:WU02:FS01:Started FahCore on PID 37006
20:48:11:WU02:FS01:Core PID:37010
20:48:11:WU02:FS01:FahCore 0x22 started
20:48:12:WU02:FS01:0x22:*********************** Log Started 2021-02-11T20:48:11Z ***********************
20:48:12:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
20:48:12:WU02:FS01:0x22:       Core: Core22
20:48:12:WU02:FS01:0x22:       Type: 0x22
20:48:12:WU02:FS01:0x22:    Version: 0.0.13
20:48:12:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
20:48:12:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
20:48:12:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
20:48:12:WU02:FS01:0x22:       Date: Sep 19 2020
20:48:12:WU02:FS01:0x22:       Time: 01:10:35
20:48:12:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
20:48:12:WU02:FS01:0x22:     Branch: core22-0.0.13
20:48:12:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
20:48:12:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
20:48:12:WU02:FS01:0x22:             -funroll-loops -DOPENMM_GIT_HASH=\"\\\"189320d0\\\"\"
20:48:12:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
20:48:12:WU02:FS01:0x22:       Bits: 64
20:48:12:WU02:FS01:0x22:       Mode: Release
20:48:12:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
20:48:12:WU02:FS01:0x22:             <peastman@stanford.edu>
20:48:12:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 37006 -checkpoint 15
20:48:12:WU02:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
20:48:12:WU02:FS01:0x22:             nvidia -gpu 0 -gpu-usage 100
20:48:12:WU02:FS01:0x22:************************************ libFAH ************************************
20:48:12:WU02:FS01:0x22:       Date: Sep 15 2020
20:48:12:WU02:FS01:0x22:       Time: 05:14:43
20:48:12:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
20:48:12:WU02:FS01:0x22:     Branch: HEAD
20:48:12:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
20:48:12:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
20:48:12:WU02:FS01:0x22:             -funroll-loops
20:48:12:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
20:48:12:WU02:FS01:0x22:       Bits: 64
20:48:12:WU02:FS01:0x22:       Mode: Release
20:48:12:WU02:FS01:0x22:************************************ CBang *************************************
20:48:12:WU02:FS01:0x22:       Date: Sep 15 2020
20:48:12:WU02:FS01:0x22:       Time: 05:11:04
20:48:12:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
20:48:12:WU02:FS01:0x22:     Branch: HEAD
20:48:12:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
20:48:12:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
20:48:12:WU02:FS01:0x22:             -funroll-loops -fPIC
20:48:12:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
20:48:12:WU02:FS01:0x22:       Bits: 64
20:48:12:WU02:FS01:0x22:       Mode: Release
20:48:12:WU02:FS01:0x22:************************************ System ************************************
20:48:12:WU02:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
20:48:12:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
20:48:12:WU02:FS01:0x22:       CPUs: 32
20:48:12:WU02:FS01:0x22:     Memory: 62.87GiB
20:48:12:WU02:FS01:0x22:Free Memory: 704.75MiB
20:48:12:WU02:FS01:0x22:    Threads: POSIX_THREADS
20:48:12:WU02:FS01:0x22: OS Version: 5.4
20:48:12:WU02:FS01:0x22:Has Battery: false
20:48:12:WU02:FS01:0x22: On Battery: false
20:48:12:WU02:FS01:0x22: UTC Offset: -5
20:48:12:WU02:FS01:0x22:        PID: 37010
20:48:12:WU02:FS01:0x22:        CWD: /var/lib/fahclient/work
20:48:12:WU02:FS01:0x22:************************************ OpenMM ************************************
20:48:12:WU02:FS01:0x22:   Revision: 189320d0
20:48:12:WU02:FS01:0x22:********************************************************************************
20:48:12:WU02:FS01:0x22:Project: 17424 (Run 0, Clone 1536, Gen 151)
20:48:12:WU02:FS01:0x22:Unit: 0x00000000000000000000000000000000
20:48:12:WU02:FS01:0x22:Reading tar file core.xml
20:48:12:WU02:FS01:0x22:Reading tar file integrator.xml.bz2
20:48:12:WU02:FS01:0x22:Reading tar file state.xml.bz2
20:48:12:WU02:FS01:0x22:Reading tar file system.xml.bz2
20:48:12:WU02:FS01:0x22:Digital signatures verified
20:48:12:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
20:48:12:WU02:FS01:0x22:Version 0.0.13
20:48:12:WU02:FS01:0x22:  Checkpoint write interval: 25000 steps (2%) [50 total]
20:48:12:WU02:FS01:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
20:48:12:WU02:FS01:0x22:  XTC frame write interval: 10000 steps (0.8%) [125 total]
20:48:12:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
20:48:12:WU02:FS01:0x22:There are 4 platforms available.
20:48:12:WU02:FS01:0x22:Platform 0: Reference
20:48:12:WU02:FS01:0x22:Platform 1: CPU
20:48:12:WU02:FS01:0x22:Platform 2: OpenCL
20:48:12:WU02:FS01:0x22:  opencl-device 0 specified
20:48:12:WU02:FS01:0x22:Platform 3: CUDA
20:48:12:WU02:FS01:0x22:  cuda-device 0 specified
20:48:31:WU02:FS01:0x22:Attempting to create CUDA context:
20:48:31:WU02:FS01:0x22:  Configuring platform CUDA

*** PROBLEM HERE:

20:48:37:WU02:FS01:0x22:ERROR:Force RMSE error of 6.46327 with threshold of 5
20:48:37:WU02:FS01:0x22:Saving result file ../logfile_01.txt
20:48:37:WU02:FS01:0x22:Saving result file science.log
20:48:37:WU02:FS01:0x22:Saving result file state.xml.bz2
20:48:37:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
\x1b[93m20:48:37:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)\x1b[0m
20:48:37:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:17424 run:0 clone:1536 gen:151 core:0x22 unit:0x00000600000000970000441000000000

*** /PROBLEM

20:48:37:WU02:FS01:Uploading 11.73MiB to 206.223.170.146
20:48:37:WU02:FS01:Connecting to 206.223.170.146:8080
20:48:37:WU00:FS01:Connecting to assign1.foldingathome.org:80
\x1b[93m20:48:38:WARNING:WU00:FS01:Failed to get assignment from 'assign1.foldingathome.org:80': No WUs available for this configuration\x1b[0m
20:48:38:WU00:FS01:Connecting to assign2.foldingathome.org:80
\x1b[93m20:48:39:WARNING:WU00:FS01:Failed to get assignment from 'assign2.foldingathome.org:80': No WUs available for this configuration\x1b[0m
20:48:39:WU00:FS01:Connecting to assign3.foldingathome.org:80
\x1b[93m20:48:39:WARNING:WU00:FS01:Failed to get assignment from 'assign3.foldingathome.org:80': No WUs available for this configuration\x1b[0m
20:48:39:WU00:FS01:Connecting to assign4.foldingathome.org:80
\x1b[93m20:48:40:WARNING:WU00:FS01:Failed to get assignment from 'assign4.foldingathome.org:80': No WUs available for this configuration\x1b[0m
\x1b[91m20:48:40:ERROR:WU00:FS01:Exception: Could not get an assignment\x1b[0m
20:48:40:WU00:FS01:Connecting to assign1.foldingathome.org:80
\x1b[93m20:48:40:WARNING:WU00:FS01:Failed to get assignment from 'assign1.foldingathome.org:80': No WUs available for this configuration\x1b[0m
20:48:40:WU00:FS01:Connecting to assign2.foldingathome.org:80
\x1b[93m20:48:41:WARNING:WU00:FS01:Failed to get assignment from 'assign2.foldingathome.org:80': No WUs available for this configuration\x1b[0m
20:48:41:WU00:FS01:Connecting to assign3.foldingathome.org:80
\x1b[93m20:48:42:WARNING:WU00:FS01:Failed to get assignment from 'assign3.foldingathome.org:80': No WUs available for this configuration\x1b[0m
20:48:42:WU00:FS01:Connecting to assign4.foldingathome.org:80
\x1b[93m20:48:42:WARNING:WU00:FS01:Failed to get assignment from 'assign4.foldingathome.org:80': No WUs available for this configuration\x1b[0m
\x1b[91m20:48:42:ERROR:WU00:FS01:Exception: Could not get an assignment\x1b[0m
20:48:43:WU02:FS01:Upload 28.24%
20:48:49:WU02:FS01:Upload 58.62%
20:48:55:WU02:FS01:Upload 88.46%
20:48:58:WU02:FS01:Upload complete
20:48:58:WU02:FS01:Server responded WORK_ACK (400)
20:48:58:WU02:FS01:Cleaning up
20:49:38:WU01:FS00:0xa8:Completed 1900000 out of 5000000 steps (38%)
20:49:40:WU00:FS01:Connecting to assign1.foldingathome.org:80
\x1b[93m20:49:40:WARNING:WU00:FS01:Failed to get assignment from 'assign1.foldingathome.org:80': No WUs available for this configuration\x1b[0m
20:49:40:WU00:FS01:Connecting to assign2.foldingathome.org:80
Something for me to wory about?

Thanks,

Ben N1NP
Antonomasia Productions
New Release 2022: Re-Entrant
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: 17424 r0 c1536 g151 - ERROR:Force RMSE

Post by bruce »

Nope.

Something for the project owner to worry about.
https://apps.foldingathome.org/wu#proje ... 36&gen=151
n1np
Posts: 32
Joined: Sat Mar 14, 2009 2:16 pm
Hardware configuration: HP DL380g8 x12
Location: Virginia
Contact:

Re: 17424 r0 c1536 g151 - ERROR:Force RMSE

Post by n1np »

Thanks bruce!

Ben N1NP
Antonomasia Productions
New Release 2022: Re-Entrant
Post Reply