Most credits missing

Moderators: Site Moderators, FAHC Science Team

Post Reply
ThorstenR
Posts: 3
Joined: Wed Jan 27, 2021 5:08 pm

Most credits missing

Post by ThorstenR »

Hello all,

for a few days now, most of the credits seems to be missing - since Monday morning, only 4 WUs out of 13 got credited.
This seems to be related either to servers 155.247.166.219 and 129.32.209.201 or to projects 14188 and 16927. The results are successfully uploaded, the server sends WORK_ACK, but they are never credited. Here's are some example snippets from the log file:

Code: Select all

01:30:45:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14188 run:5 clone:1491 gen:8 core:0xa7 unit:0x000005d3000000080000376c00000005
01:32:56:WU00:FS00:Starting
01:32:56:WU00:FS00:Running FahCore: /usr/local/bin/FAHCoreWrapper "/Library/Application Support/FAHClient/cores/cores.foldingathome.org/osx/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7" -dir 00 -suffix 01 -version 706 -lifeline 68368 -checkpoint 15 -np 4
01:32:56:WU00:FS00:Started FahCore on PID 2037
01:32:56:WU00:FS00:Core PID:2038
01:32:56:WU00:FS00:FahCore 0xa7 started
01:32:57:WU00:FS00:0xa7:*********************** Log Started 2021-01-25T01:32:56Z ***********************
01:32:57:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
01:32:57:WU00:FS00:0xa7:       Type: 0xa7
01:32:57:WU00:FS00:0xa7:       Core: Gromacs
01:32:57:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 2037 -checkpoint 15 -np 4
01:32:57:WU00:FS00:0xa7:************************************ CBang *************************************
01:32:57:WU00:FS00:0xa7:       Date: Nov 27 2019
01:32:57:WU00:FS00:0xa7:       Time: 03:27:01
01:32:57:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
01:32:57:WU00:FS00:0xa7:     Branch: master
01:32:57:WU00:FS00:0xa7:   Compiler: GNU 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.8)
01:32:57:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -mmacosx-version-min=10.7
01:32:57:WU00:FS00:0xa7:             -Wno-unused-local-typedefs -stdlib=libc++ -fPIC
01:32:57:WU00:FS00:0xa7:   Platform: darwin 19.0.0
01:32:57:WU00:FS00:0xa7:       Bits: 64
01:32:57:WU00:FS00:0xa7:       Mode: Release
01:32:57:WU00:FS00:0xa7:************************************ System ************************************
01:32:57:WU00:FS00:0xa7:        CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
01:32:57:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 70 Stepping 1
01:32:57:WU00:FS00:0xa7:       CPUs: 8
01:32:57:WU00:FS00:0xa7:     Memory: 16.00GiB
01:32:57:WU00:FS00:0xa7:Free Memory: 9.86GiB
01:32:57:WU00:FS00:0xa7:    Threads: POSIX_THREADS
01:32:57:WU00:FS00:0xa7: OS Version: 10.11
01:32:57:WU00:FS00:0xa7:Has Battery: true
01:32:57:WU00:FS00:0xa7: On Battery: false
01:32:57:WU00:FS00:0xa7: UTC Offset: 1
01:32:57:WU00:FS00:0xa7:        PID: 2038
01:32:57:WU00:FS00:0xa7:        CWD: /Library/Application Support/FAHClient/work
01:32:57:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
01:32:57:WU00:FS00:0xa7:    Version: 0.0.19
01:32:57:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
01:32:57:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
01:32:57:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
01:32:57:WU00:FS00:0xa7:       Date: Nov 25 2019
01:32:57:WU00:FS00:0xa7:       Time: 16:41:59
01:32:57:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
01:32:57:WU00:FS00:0xa7:     Branch: master
01:32:57:WU00:FS00:0xa7:   Compiler: GNU 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.8)
01:32:57:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -mmacosx-version-min=10.7
01:32:57:WU00:FS00:0xa7:             -Wno-unused-local-typedefs -stdlib=libc++
01:32:57:WU00:FS00:0xa7:   Platform: darwin 19.0.0
01:32:57:WU00:FS00:0xa7:       Bits: 64
01:32:57:WU00:FS00:0xa7:       Mode: Release
01:32:57:WU00:FS00:0xa7:************************************ Build *************************************
01:32:57:WU00:FS00:0xa7:       SIMD: avx_256
01:32:57:WU00:FS00:0xa7:********************************************************************************
01:32:57:WU00:FS00:0xa7:Project: 14188 (Run 5, Clone 1491, Gen 8)
01:32:57:WU00:FS00:0xa7:Unit: 0x00000000000000000000000000000000
01:32:57:WU00:FS00:0xa7:Reading tar file core.xml
01:32:57:WU00:FS00:0xa7:Reading tar file frame8.tpr
01:32:57:WU00:FS00:0xa7:Digital signatures verified
01:32:57:WU00:FS00:0xa7:Calling: mdrun -s frame8.tpr -o frame8.trr -cpt 15 -nt 4
01:32:57:WU00:FS00:0xa7:Steps: first=10000000 total=1250000
<snip>
22:29:33:WU00:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
22:29:34:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:29:34:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:14188 run:5 clone:1491 gen:8 core:0xa7 unit:0x000005d3000000080000376c00000005
22:29:34:WU00:FS00:Uploading 11.93MiB to 155.247.166.219
22:29:34:WU00:FS00:Connecting to 155.247.166.219:8080
22:29:40:WU00:FS00:Upload 13.62%
22:29:46:WU00:FS00:Upload 26.19%
22:29:52:WU00:FS00:Upload 48.19%
22:29:58:WU00:FS00:Upload 62.34%
22:30:04:WU00:FS00:Upload 71.24%
22:30:10:WU00:FS00:Upload 77.53%
22:30:16:WU00:FS00:Upload 80.67%
22:30:22:WU00:FS00:Upload 86.96%
22:30:28:WU00:FS00:Upload 98.48%
22:30:32:WU00:FS00:Upload complete
22:30:32:WU00:FS00:Server responded WORK_ACK (400)
22:30:32:WU00:FS00:Final credit estimate, 22772.00 points
22:30:32:WU00:FS00:Cleaning up

Code: Select all

07:33:16:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
07:33:16:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16927 run:10 clone:1115 gen:2 core:0xa7 unit:0x0000045b000000020000421f0000000a
07:33:16:WU00:FS00:Uploading 6.30MiB to 129.32.209.201
07:33:16:WU00:FS00:Connecting to 129.32.209.201:8080
07:33:22:WU00:FS00:Upload 13.89%
07:33:28:WU00:FS00:Upload 49.61%
07:33:34:WU00:FS00:Upload 86.32%
07:33:36:WU00:FS00:Upload complete
07:33:36:WU00:FS00:Server responded WORK_ACK (400)
07:33:36:WU00:FS00:Final credit estimate, 8970.00 points
07:33:36:WU00:FS00:Cleaning up
Anything I can do to help?

Cheers,
Thorsten
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Most credits missing

Post by bruce »

ThorstenR wrote:
Anything I can do to help?

Cheers,
Thorsten
No. (unfortunately)

There are multiple reports of multiple servers failing to transfer the information about points and WUs to the stats system. I expect this is a temporary condition and they'll get there eventually. The problem has been escalated and I see no responses from the folks that can fix that type of problem.
ThorstenR
Posts: 3
Joined: Wed Jan 27, 2021 5:08 pm

Re: Most credits missing

Post by ThorstenR »

Just one little observation: both projects (and servers) that failed are hosted at Temple in Philly. Maybe there's a networking thing between Temple and the FAH stats servers?
JimboPalmer
Posts: 2573
Joined: Mon Feb 16, 2009 4:12 am
Location: Greenwood MS USA

Re: Most credits missing

Post by JimboPalmer »

ThorstenR wrote:Just one little observation: both projects (and servers) that failed are hosted at Temple in Philly. Maybe there's a networking thing between Temple and the FAH stats servers?
Just a comment: this can be one reason these can be so slow to resolve, watch the bureaucracy: F@H team at Temple, IT Department at Temple, Help Desk at ISP A, Help Desk at ISP B, IT Department at University with Stats Server, F@H team at University with Stats Server. You know that not all of these groups actually talk to each other. This game of 'Telephone' can tale weeks.

https://en.wikipedia.org/wiki/Chinese_whispers
Tsar of all the Rushers
I tried to remain childlike, all I achieved was childish.
A friend to those who want no friends
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Most credits missing

Post by Neil-B »

.. and even once the issue is resolved the clean up exercise of identifying all the "missed" stats and catching them up takes time
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Post Reply