Page 1 of 1

anyone else getting less Moonshot projects?

Posted: Tue Sep 15, 2020 10:30 pm
by Knish
During Sprint 3 I had near 100% Moonshot WUs going anytime I checked all my folding clients.

The past 7 days it has been 25%. That's only 1 or 2 from a pool of 7 to 10 instances running FAH. They're all on Tesla P100s

Code: Select all

******************* Log Started 2020-09-15T21:56:50Z ****************
21:56:50:Trying to access database...
21:56:50:Successfully acquired database lock
21:56:50:Downloading GPUs.txt from assign1.foldingathome.org:80
21:56:50:Connecting to assign1.foldingathome.org:80
21:56:51:Read GPUs.txt
21:56:51:Enabled folding slot 00: PAUSED cpu:4 (by user)
21:56:53:Enabled folding slot 01: READY gpu:0:GP100GL [Tesla P100 16GB] 9526
21:56:53:***************************** FAHClient *********************
21:56:53:        Version: 7.6.13
21:56:53:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
21:56:53:      Copyright: 2020 foldingathome.org
21:56:53:       Homepage: https://foldingathome.org/
21:56:53:           Date: Apr 28 2020
21:56:53:           Time: 04:20:16
21:56:53:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
21:56:53:         Branch: master
21:56:53:       Compiler: GNU 8.3.0
21:56:53:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
21:56:53:                 -funroll-loops -fno-pie
21:56:53:       Platform: linux2 4.19.0-5-amd64
21:56:53:           Bits: 64
21:56:53:           Mode: Release
21:56:53:           Args: --child /etc/fahclient/config.xml --run-as fahclient
21:56:53:                 --pid-file=/var/run/fahclient.pid --daemon
21:56:53:         Config: /etc/fahclient/config.xml
21:56:53:******************************** CBang **********************
21:56:53:           Date: Apr 25 2020
21:56:53:           Time: 00:07:53
21:56:53:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
21:56:53:         Branch: master
21:56:53:       Compiler: GNU 8.3.0
21:56:53:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
21:56:53:                 -funroll-loops -fno-pie -fPIC
21:56:53:       Platform: linux2 4.19.0-5-amd64
21:56:53:           Bits: 64
21:56:53:           Mode: Release
21:56:53:******************************* System *********************
21:56:53:            CPU: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
21:56:53:         CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
21:56:53:           CPUs: 6
21:56:53:         Memory: 110.17GiB
21:56:53:    Free Memory: 109.40GiB
21:56:53:        Threads: POSIX_THREADS
21:56:53:     OS Version: 4.19
21:56:53:    Has Battery: false
21:56:53:     On Battery: false
21:56:53:     UTC Offset: 0
21:56:53:            PID: 584
21:56:53:            CWD: /var/lib/fahclient
21:56:53:             OS: Linux 4.19.0-10-cloud-amd64 x86_64
21:56:53:        OS Arch: AMD64
21:56:53:           GPUs: 1
21:56:53:          GPU 0: Bus:0 Slot:0 Func:0 NVIDIA:5 GP100GL [Tesla P100 16GB] 9526
21:56:53:  CUDA Device 0: Platform:0 Device:0 Bus:0 Slot:0 Compute:6.0 Driver:11.0
21:56:53:OpenCL Device 0: Platform:0 Device:0 Bus:0 Slot:0 Compute:1.2 Driver:450.51
21:56:53:******************************* libFAH ***********************
21:56:53:           Date: Apr 15 2020
21:56:53:           Time: 21:43:24
21:56:53:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
21:56:53:         Branch: master
21:56:53:       Compiler: GNU 8.3.0
21:56:53:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
21:56:53:                 -funroll-loops -fno-pie
21:56:53:       Platform: linux2 4.19.0-5-amd64
21:56:53:           Bits: 64
21:56:53:           Mode: Release
21:56:53:*************************************************************
21:56:53:<config>
21:56:53:  <!-- Client Control -->
21:56:53:  <fold-anon v='true'/>
21:56:53:
21:56:53:  <!-- Folding Slot Configuration -->
21:56:53:  <cpus v='4'/>
21:56:53:
21:56:53:  <!-- HTTP Server -->

21:56:53:  <!-- Folding Slots -->
21:56:53:  <slot id='0' type='CPU'>
21:56:53:    <paused v='true'/>
21:56:53:  </slot>
21:56:53:  <slot id='1' type='GPU'/>
21:56:53:</config>
21:56:53:WU01:FS01:Connecting to assign1.foldingathome.org:80
21:56:54:WU01:FS01:Assigned to work server 66.170.111.50
21:56:54:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP100GL [Tesla P100 16GB] 9526 from 66.170.111.50
21:56:54:WU01:FS01:Connecting to 66.170.111.50:8080
21:56:54:WU01:FS01:Downloading 9.28MiB
21:56:57:WU01:FS01:Download complete
21:56:57:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14488 run:0 clone:96 gen:27 core:0x22 unit:0x0000002242aa6f325f45dea88488499e
21:56:57:WU01:FS01:Starting
21:56:57:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.11/Core_22.fah/FahCore_22 -dir 01 -suffix 01 -version 706 -lifeline 584 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
21:56:57:WU01:FS01:Started FahCore on PID 685
21:56:57:WU01:FS01:Core PID:689
21:56:57:WU01:FS01:FahCore 0x22 started
21:56:58:WU01:FS01:0x22:*********************** Log Started 2020-09-15T21:56:57Z ***********************
21:56:58:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
21:56:58:WU01:FS01:0x22:       Core: Core22
21:56:58:WU01:FS01:0x22:       Type: 0x22
21:56:58:WU01:FS01:0x22:    Version: 0.0.11
21:56:58:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
21:56:58:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
21:56:58:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
21:56:58:WU01:FS01:0x22:       Date: Jun 27 2020
21:56:58:WU01:FS01:0x22:       Time: 22:50:00
21:56:58:WU01:FS01:0x22:   Revision: cfc2940c5dd1aa80f60daa6e28d4a2a417f74edb
21:56:58:WU01:FS01:0x22:     Branch: core22-0.0.11
21:56:58:WU01:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
21:56:58:WU01:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
21:56:58:WU01:FS01:0x22:             -funroll-loops
21:56:58:WU01:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
21:56:58:WU01:FS01:0x22:       Bits: 64
21:56:58:WU01:FS01:0x22:       Mode: Release
21:56:58:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
21:56:58:WU01:FS01:0x22:             <peastman@stanford.edu>
21:56:58:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 685 -checkpoint 15
21:56:58:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
21:56:58:WU01:FS01:0x22:             0 -gpu 0
21:56:58:WU01:FS01:0x22:************************************ libFAH ************************************
21:56:58:WU01:FS01:0x22:       Date: Jun 27 2020
21:56:58:WU01:FS01:0x22:       Time: 22:11:04
21:56:58:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
21:56:58:WU01:FS01:0x22:     Branch: HEAD
21:56:58:WU01:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
21:56:58:WU01:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
21:56:58:WU01:FS01:0x22:             -funroll-loops
21:56:58:WU01:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
21:56:58:WU01:FS01:0x22:       Bits: 64
21:56:58:WU01:FS01:0x22:       Mode: Release
21:56:58:WU01:FS01:0x22:************************************ CBang *************************************
21:56:58:WU01:FS01:0x22:       Date: Jun 27 2020
21:56:58:WU01:FS01:0x22:       Time: 22:10:11
21:56:58:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
21:56:58:WU01:FS01:0x22:     Branch: HEAD
21:56:58:WU01:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
21:56:58:WU01:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
21:56:58:WU01:FS01:0x22:             -funroll-loops -fPIC
21:56:58:WU01:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
21:56:58:WU01:FS01:0x22:       Bits: 64
21:56:58:WU01:FS01:0x22:       Mode: Release
21:56:58:WU01:FS01:0x22:************************************ System ************************************
21:56:58:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
21:56:58:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
21:56:58:WU01:FS01:0x22:       CPUs: 6
21:56:58:WU01:FS01:0x22:     Memory: 110.17GiB
21:56:58:WU01:FS01:0x22:Free Memory: 109.28GiB
21:56:58:WU01:FS01:0x22:    Threads: POSIX_THREADS
21:56:58:WU01:FS01:0x22: OS Version: 4.19
21:56:58:WU01:FS01:0x22:Has Battery: false
21:56:58:WU01:FS01:0x22: On Battery: false
21:56:58:WU01:FS01:0x22: UTC Offset: 0
21:56:58:WU01:FS01:0x22:        PID: 689
21:56:58:WU01:FS01:0x22:        CWD: /var/lib/fahclient/work
21:56:58:WU01:FS01:0x22:********************************************************************************
21:56:58:WU01:FS01:0x22:Project: 14488 (Run 0, Clone 96, Gen 27)
21:56:58:WU01:FS01:0x22:Unit: 0x0000002242aa6f325f45dea88488499e
21:56:58:WU01:FS01:0x22:Reading tar file core.xml
21:56:58:WU01:FS01:0x22:Reading tar file integrator.xml.bz2
21:56:58:WU01:FS01:0x22:Reading tar file state.xml.bz2
21:56:58:WU01:FS01:0x22:Reading tar file system.xml.bz2
21:56:58:WU01:FS01:0x22:Digital signatures verified
21:56:58:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
21:56:58:WU01:FS01:0x22:Version 0.0.11
21:56:58:WU01:FS01:0x22:  Checkpoint write interval: 25000 steps (2%) [50 total]
21:56:58:WU01:FS01:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
21:56:58:WU01:FS01:0x22:  XTC frame write interval: 10000 steps (0.8%) [125 total]
21:56:58:WU01:FS01:0x22:  Global context and integrator variables write interval: disabled
21:57:09:WU01:FS01:0x22:Completed 0 out of 1250000 steps (0%)
21:58:01:WU01:FS01:0x22:Completed 12500 out of 1250000 steps (1%)
21:58:52:WU01:FS01:0x22:Completed 25000 out of 1250000 steps (2%)

Re: anyone else getting less Moonshot projects?

Posted: Wed Sep 16, 2020 12:33 am
by PantherX
Earlier in the week, the Server hosting Moonshot WUs had technical difficulties and it took a bit of time to resolve. However, it is now fully operational so you will hopefully see a steady stream of Moonshot WUs :)

Re: anyone else getting less Moonshot projects?

Posted: Wed Sep 16, 2020 2:16 am
by JohnChodera
Right now, aws3.foldingathome.org (the COVID Moonshot server hosted on AWS) is seeing about ~1/3 the total core22 throughput of FAH.
I just realized I had neglected to configure the work server during some prior service to prefer clients that have set the Cause to COVID_19, so I've just done that.
I believe this will now result in those clients that have set Cause to COVID_19 preferentially picking up Moonshot WUs, so if you haven't set your Cause already, that should help!

Thanks for bringing this to our attention, and huge thanks for all of your support!

~ John Chodera // MSKCC

Re: anyone else getting less Moonshot projects?

Posted: Wed Sep 16, 2020 5:04 am
by JohnChodera
We're up to ~2/3 of the core22 assignments now, so I think this is helping.

Thanks again for the heads up!

~ John Chodera // MSKCC

Re: anyone else getting less Moonshot projects?

Posted: Thu Sep 17, 2020 2:28 pm
by bruce
Development is also untangling some issues releasing a new version of FAHCore_22. That should help, too.

Re: anyone else getting less Moonshot projects?

Posted: Thu Sep 17, 2020 4:00 pm
by HaloJones
Only getting 1490x units here