Failed to send results to work server - 158.130.86.56

[CHiMUC05]

Since a couple of days, I get error messages for sending results to 158.130.86.56

Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database

here the last lines from the log file:
******************************* Date: 2024-02-02 *******************************
07:48:21:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:18806 run:25 clone:55 gen:43 core:0xa9 unit:0x000000370000002b0000497600000019
07:48:21:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
07:48:21:WU00:FS00:Trying to send results to collection server
07:48:21:ERROR:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
08:16:45:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:18806 run:70 clone:7 gen:125 core:0xa9 unit:0x000000070000007d0000497600000046
08:16:45:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
08:16:45:WU01:FS00:Trying to send results to collection server
08:16:45:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
08:48:21:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:18806 run:25 clone:55 gen:43 core:0xa9 unit:0x000000370000002b0000497600000019
08:48:21:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
08:48:21:WU00:FS00:Trying to send results to collection server
08:48:21:ERROR:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
09:16:45:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:18806 run:70 clone:7 gen:125 core:0xa9 unit:0x000000070000007d0000497600000046
09:16:45:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
09:16:45:WU01:FS00:Trying to send results to collection server
09:16:45:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
09:48:22:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:18806 run:25 clone:55 gen:43 core:0xa9 unit:0x000000370000002b0000497600000019
09:48:22:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
09:48:22:WU00:FS00:Trying to send results to collection server
09:48:22:ERROR:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
10:16:45:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:18806 run:70 clone:7 gen:125 core:0xa9 unit:0x000000070000007d0000497600000046
10:16:45:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
10:16:45:WU01:FS00:Trying to send results to collection server
10:16:45:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
10:48:22:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:18806 run:25 clone:55 gen:43 core:0xa9 unit:0x000000370000002b0000497600000019
10:48:22:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
10:48:22:WU00:FS00:Trying to send results to collection server
10:48:22:ERROR:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
11:16:45:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:18806 run:70 clone:7 gen:125 core:0xa9 unit:0x000000070000007d0000497600000046
11:16:45:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
11:16:45:WU01:FS00:Trying to send results to collection server
11:16:45:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database


Any idea how to solve this problem?
I assume it needs to be solved on the collection server.

Regards

[BobWilliams757]

Can you find the part of the log that shows the connection IP of the server?

[jjmiller]

Wanted to check in - are you still seeing this issue? Usually 158.... servers are the Penn servers, but we don't have any at that IP address. Echoing BobWilliams757's request- if you send along the full log that has the IP address included we may be able to sort things out a bit more!

[CHiMUC05]

Thanks of the replay. The log file is to big to post in this message.
However I check if I could find the IP address in the log file, but I could find the sending IP address, that's strange.

Here parts of the log file:

11:46:53:WU01:FS00:0xa9:*********************** Log Started 2024-01-31T11:46:52Z ***********************
11:46:53:WU01:FS00:0xa9:************************** Gromacs Folding@home Core ***************************
11:46:53:WU01:FS00:0xa9: Core: Gromacs
11:46:53:WU01:FS00:0xa9: Type: 0xa9
11:46:53:WU01:FS00:0xa9: Version: 0.0.12
11:46:53:WU01:FS00:0xa9: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
11:46:53:WU01:FS00:0xa9: Copyright: 2022 foldingathome.org
11:46:53:WU01:FS00:0xa9: Homepage: https://foldingathome.org/
11:46:53:WU01:FS00:0xa9: Date: Nov 9 2022
11:46:53:WU01:FS00:0xa9: Time: 15:02:46
11:46:53:WU01:FS00:0xa9: Revision: dd7eea37225ad21fddc0eaee26a6cf216f7e9766
11:46:53:WU01:FS00:0xa9: Branch: master
11:46:53:WU01:FS00:0xa9: Compiler: GNU Apple LLVM 14.0.0 (clang-1400.0.29.202)
11:46:53:WU01:FS00:0xa9: Options: -std=c++17 -fsigned-char -stdlib=libc++ -faligned-new -O3
11:46:53:WU01:FS00:0xa9: -funroll-loops -fno-pie -arch x86_64 -mmacosx-version-min=10.14
11:46:53:WU01:FS00:0xa9: Platform: darwin 21.6.0
11:46:53:WU01:FS00:0xa9: Bits: 64
11:46:53:WU01:FS00:0xa9: Mode: Release
11:46:53:WU01:FS00:0xa9: SIMD: avx2_256
11:46:53:WU01:FS00:0xa9: OpenMP: ON
11:46:53:WU01:FS00:0xa9: CUDA: OFF
11:46:53:WU01:FS00:0xa9: OpenCL: OFF
11:46:53:WU01:FS00:0xa9: Args: -dir 01 -suffix 01 -version 706 -lifeline 8920 -checkpoint 15 -np 4
11:46:53:WU01:FS00:0xa9:************************************ libFAH ************************************
11:46:53:WU01:FS00:0xa9: Date: Nov 9 2022
11:46:53:WU01:FS00:0xa9: Time: 15:01:25
11:46:53:WU01:FS00:0xa9: Revision: 6839156e07c373b19985c8ec346a2c61157bd2cf
11:46:53:WU01:FS00:0xa9: Branch: master
11:46:53:WU01:FS00:0xa9: Compiler: GNU Apple LLVM 14.0.0 (clang-1400.0.29.202)
11:46:53:WU01:FS00:0xa9: Options: -std=c++17 -fsigned-char -stdlib=libc++ -faligned-new -O3
11:46:53:WU01:FS00:0xa9: -funroll-loops -fno-pie -arch x86_64 -mmacosx-version-min=10.14
11:46:53:WU01:FS00:0xa9: Platform: darwin 21.6.0
11:46:53:WU01:FS00:0xa9: Bits: 64
11:46:53:WU01:FS00:0xa9: Mode: Release
11:46:53:WU01:FS00:0xa9:************************************ CBang *************************************
11:46:53:WU01:FS00:0xa9: Date: Nov 15 2022
11:46:53:WU01:FS00:0xa9: Time: 05:18:22
11:46:53:WU01:FS00:0xa9: Revision: a79801284795bd9a66b1891ad3ffa3860928f9a4
11:46:53:WU01:FS00:0xa9: Branch: master
11:46:53:WU01:FS00:0xa9: Compiler: GNU Apple LLVM 14.0.0 (clang-1400.0.29.202)
11:46:53:WU01:FS00:0xa9: Options: -std=c++17 -fsigned-char -stdlib=libc++ -faligned-new -O3
11:46:53:WU01:FS00:0xa9: -funroll-loops -fno-pie -arch x86_64 -mmacosx-version-min=10.14
11:46:53:WU01:FS00:0xa9: -fPIC
11:46:53:WU01:FS00:0xa9: Platform: darwin 21.6.0
11:46:53:WU01:FS00:0xa9: Bits: 64
11:46:53:WU01:FS00:0xa9: Mode: Release
11:46:53:WU01:FS00:0xa9:************************************ System ************************************
11:46:53:WU01:FS00:0xa9: CPU: Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
11:46:53:WU01:FS00:0xa9: CPU ID: GenuineIntel Family 6 Model 158 Stepping 9
11:46:53:WU01:FS00:0xa9: CPUs: 4
11:46:53:WU01:FS00:0xa9: Memory: 40.00GiB
11:46:53:WU01:FS00:0xa9:Free Memory: 26.96GiB
11:46:53:WU01:FS00:0xa9: Threads: POSIX_THREADS
11:46:53:WU01:FS00:0xa9: OS Version: 13.6.4
11:46:53:WU01:FS00:0xa9:Has Battery: false
11:46:53:WU01:FS00:0xa9: On Battery: false
11:46:53:WU01:FS00:0xa9: UTC Offset: 1
11:46:53:WU01:FS00:0xa9: PID: 8921
11:46:53:WU01:FS00:0xa9: CWD: /Library/Application Support/FAHClient/work
11:46:53:WU01:FS00:0xa9: Exec: /Library/Application
11:46:53:WU01:FS00:0xa9: Support/FAHClient/cores/cores.foldingathome.org/gromacs-core-a9/macos-12-64bit/cpu-avx2_256-release/0xa9-0.0.12/Core_a9.fah/FahCore_a9
11:46:53:WU01:FS00:0xa9:********************************************************************************
11:46:53:WU01:FS00:0xa9:Project: 18806 (Run 70, Clone 7, Gen 125)
11:46:53:WU01:FS00:0xa9:Reading tar file core.xml
11:46:53:WU01:FS00:0xa9:Reading tar file frame125.tpr
11:46:53:WU01:FS00:0xa9:Digital signatures verified
11:46:53:WU01:FS00:0xa9:Calling: mdrun -c frame125.gro -s frame125.tpr -x frame125.xtc -cpt 15 -nt 4 -ntmpi 1 -update cpu -nb cpu -bonded cpu -pme cpu -pmefft cpu
11:46:53:WU01:FS00:0xa9:Steps: first=31250000 total=31500000
11:47:00:WU01:FS00:0xa9:Completed 1 out of 250000 steps (0%)
11:47:53:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:18806 run:25 clone:55 gen:43 core:0xa9 unit:0x000000370000002b0000497600000019
11:47:53:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
11:47:53:WU00:FS00:Trying to send results to collection server
11:47:53:ERROR:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
11:49:30:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:18806 run:25 clone:55 gen:43 core:0xa9 unit:0x000000370000002b0000497600000019
11:49:30:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
11:49:30:WU00:FS00:Trying to send results to collection server
11:49:30:ERROR:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
11:52:07:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:18806 run:25 clone:55 gen:43 core:0xa9 unit:0x000000370000002b0000497600000019
11:52:07:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
11:52:07:WU00:FS00:Trying to send results to collection server
11:52:07:ERROR:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
11:56:21:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:18806 run:25 clone:55 gen:43 core:0xa9 unit:0x000000370000002b0000497600000019
11:56:21:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
11:56:21:WU00:FS00:Trying to send results to collection server
11:56:21:ERROR:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
11:56:47:WU01:FS00:0xa9:Completed 2500 out of 250000 steps (1%)
11:56:48:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
11:56:48:WARNING:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
12:03:13:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:18806 run:25 clone:55 gen:43 core:0xa9 unit:0x000000370000002b0000497600000019
12:03:13:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
12:03:13:WU00:FS00:Trying to send results to collection server
12:03:13:ERROR:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
12:06:38:WU01:FS00:0xa9:Completed 5000 out of 250000 steps (2%)
12:06:39:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
12:06:39:WARNING:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
12:14:18:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:18806 run:25 clone:55 gen:43 core:0xa9 unit:0x000000370000002b0000497600000019
12:14:18:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
12:14:18:WU00:FS00:Trying to send results to collection server
12:14:18:ERROR:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
12:16:23:WU01:FS00:0xa9:Completed 7500 out of 250000 steps (3%)

...

04:05:20:WU01:FS00:0xa9:Completed 247500 out of 250000 steps (99%)
04:05:21:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
04:05:21:WARNING:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
04:15:07:WU01:FS00:0xa9:Completed 250000 out of 250000 steps (100%)
04:15:07:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
04:15:07:WARNING:WU00:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
04:15:17:WU01:FS00:0xa9:Saving result file ../logfile_01.txt
04:15:17:WU01:FS00:0xa9:Saving result file frame125.gro
04:15:18:WU01:FS00:0xa9:Saving result file frame125.xtc
04:15:18:WU01:FS00:0xa9:Saving result file md.log
04:15:18:WU01:FS00:0xa9:Saving result file science.log
04:15:18:WU01:FS00:0xa9:Saving result file state.cpt
04:15:18:WU01:FS00:0xa9:Folding@home Core Shutdown: FINISHED_UNIT
04:15:18:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
04:15:18:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
04:15:18:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:18806 run:70 clone:7 gen:125 core:0xa9 unit:0x000000070000007d0000497600000046
04:15:18:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
04:15:18:WU01:FS00:Trying to send results to collection server
04:15:18:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
04:15:19:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:18806 run:70 clone:7 gen:125 core:0xa9 unit:0x000000070000007d0000497600000046
04:15:19:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
04:15:19:WU01:FS00:Trying to send results to collection server
04:15:19:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
04:16:19:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:18806 run:70 clone:7 gen:125 core:0xa9 unit:0x000000070000007d0000497600000046
04:16:19:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
04:16:19:WU01:FS00:Trying to send results to collection server
04:16:19:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
04:17:56:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:18806 run:70 clone:7 gen:125 core:0xa9 unit:0x000000070000007d0000497600000046
04:17:56:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
04:17:56:WU01:FS00:Trying to send results to collection server
04:17:56:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
04:20:33:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:18806 run:70 clone:7 gen:125 core:0xa9 unit:0x000000070000007d0000497600000046
04:20:33:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database
04:20:33:WU01:FS00:Trying to send results to collection server
04:20:33:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database
04:24:47:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:18806 run:70 clone:7 gen:125 core:0xa9 unit:0x000000070000007d0000497600000046
04:24:47:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to advance statement result: Attempt to write a readonly database


hope this helps.

[CHiMUC05]

just an addition, this was the download server, and the first exception already came after the download:

19:15:53:WU01:FS00:Connecting to assign1.foldingathome.org:80
19:15:54:WU01:FS00:Assigned to work server 144.121.86.56
19:15:54:WU01:FS00:Requesting new work unit for slot 00: cpu:4 from 144.121.86.56
19:15:54:WU01:FS00:Connecting to 144.121.86.56:8080
19:15:55:WU01:FS00:Downloading 21.80MiB
19:16:01:WU01:FS00:Download 88.29%
19:16:01:WU01:FS00:Download complete
19:16:02:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:18806 run:70 clone:7 gen:125 core:0xa9 unit:0x000000070000007d0000497600000046
19:16:02:ERROR:WU01:FS00:Exception: Failed to advance statement result: Attempt to write a readonly database


Maybe I need to reinstall the Folding Client...

[Joe_H]

I have never seen the error you are getting before, so not certain what is causing it. But it is occurring on your system before an upload connection is opened. From an older log of mine here is an example of the normal sequence:

Code: Select all

19:34:13:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16977 run:74 clone:731 gen:228 core:0xa8 unit:0x000002db000000e4000042510000004a
19:34:13:WU00:FS00:Uploading 2.77MiB to 129.32.209.202
19:34:13:WU00:FS00:Connecting to 129.32.209.202:8080
19:34:16:WU00:FS00:Upload complete
19:34:16:WU00:FS00:Server responded WORK_ACK (400)
19:34:16:WU00:FS00:Final credit estimate, 5687.00 points
19:34:16:WU00:FS00:Cleaning up

After the "Sending unit results..." line the log shows which WS IP address the client is connecting to. Instead you get the error abut not being able to write to a database. So a SQL database used by the client to track various operations has either become readonly or is locked by another process for some reason. Reinstalling might fix the first, candidates for the second can include anti-malware software scanning the F@h client data directory or a second FAHClient process running on your system.

[CHiMUC05]

Thanks for your replay.
Yes, I found a this sequence in an old log file as well.

I agree with you, that this has something to do with the access right of the client.
I will double check this and will let you know if I could fix it.


Gabriel

[CHiMUC05]

I couldn't find the exact issue of the problem with the access rights.
So I completely removed the f@h client and its settings and install it again.
Now it works again and I could process and upload a unit without any error

The only issue I have now, that the FAHControl window will disconnect from time to time, showing updating and will reconnect again after some time... But this is not critical.