New system wont fold

[v00d00]

I have built a temporary system due to a parts failure. It comprises of the following.

AMD A8-9600
Gigabyte A320-S2H (latest BIOS F50)
8GB DDR4 2133
Nvidia GTX 1050 Ti

I went to start folding on the Nvidia and it failed with a bad workunit, mayybe 3 times before I shut it down. I've tried 3 different driver versions, to no avail. I've tried downgrading the pcie slot to 2.0 (and 1.0). Reinstalling opencl.

log.txt

Code: Select all

*********************** Log Started 2020-06-20T03:02:53Z ***********************
03:02:53:Trying to access database...
03:02:53:Successfully acquired database lock
03:02:53:Read GPUs.txt
03:02:55:Enabled folding slot 01: READY gpu:1:GP107 [GeForce GTX 1050 Ti]  2138
03:02:55:****************************** FAHClient ******************************
03:02:55:        Version: 7.6.13
03:02:55:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
03:02:55:      Copyright: 2020 foldingathome.org
03:02:55:       Homepage: https://foldingathome.org/
03:02:55:           Date: Apr 28 2020
03:02:55:           Time: 04:20:16
03:02:55:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
03:02:55:         Branch: master
03:02:55:       Compiler: GNU 8.3.0
03:02:55:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
03:02:55:                 -funroll-loops -fno-pie
03:02:55:       Platform: linux2 4.19.0-5-amd64
03:02:55:           Bits: 64
03:02:55:           Mode: Release
03:02:55:         Config: /opt/fah/fahclient-1050/config.xml
03:02:55:******************************** CBang ********************************
03:02:55:           Date: Apr 25 2020
03:02:55:           Time: 00:07:53
03:02:55:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
03:02:55:         Branch: master
03:02:55:       Compiler: GNU 8.3.0
03:02:55:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
03:02:55:                 -funroll-loops -fno-pie -fPIC
03:02:55:       Platform: linux2 4.19.0-5-amd64
03:02:55:           Bits: 64
03:02:55:           Mode: Release
03:02:55:******************************* System ********************************
03:02:55:            CPU: AMD A8-9600 RADEON R7, 10 COMPUTE CORES 4C+6G
03:02:55:         CPU ID: AuthenticAMD Family 21 Model 101 Stepping 1
03:02:55:           CPUs: 4
03:02:55:         Memory: 7.26GiB
03:02:55:    Free Memory: 5.72GiB
03:02:55:        Threads: POSIX_THREADS
03:02:55:     OS Version: 4.9
03:02:55:    Has Battery: false
03:02:55:     On Battery: false
03:02:55:     UTC Offset: 1
03:02:55:            PID: 16685
03:02:55:            CWD: /opt/fah/fahclient-1050
03:02:55:             OS: Linux 4.9.0-7-amd64 x86_64
03:02:55:        OS Arch: AMD64
03:02:55:           GPUs: 2
03:02:55:          GPU 0: Bus:0 Slot:1 Func:0 AMD:5 Carrizo [Radeon R7/R6/R5 Series]
03:02:55:          GPU 1: Bus:7 Slot:0 Func:0 NVIDIA:7 GP107 [GeForce GTX 1050 Ti] 2138
03:02:55:  CUDA Device 0: Platform:0 Device:0 Bus:7 Slot:0 Compute:6.1 Driver:9.0
03:02:55:OpenCL Device 0: Platform:0 Device:0 Bus:7 Slot:0 Compute:1.2 Driver:384.130
03:02:55:OpenCL Device 1: Platform:1 Device:0 Bus:NA Slot:NA Compute:1.1 Driver:13.0
03:02:55:******************************* libFAH ********************************
03:02:55:           Date: Apr 15 2020
03:02:55:           Time: 21:43:24
03:02:55:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
03:02:55:         Branch: master
03:02:55:       Compiler: GNU 8.3.0
03:02:55:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
03:02:55:                 -funroll-loops -fno-pie
03:02:55:       Platform: linux2 4.19.0-5-amd64
03:02:55:           Bits: 64
03:02:55:           Mode: Release
03:02:55:***********************************************************************
03:02:55:<config>
03:02:55:  <!-- Folding Core -->
03:02:55:  <checkpoint v='10'/>
03:02:55:
03:02:55:  <!-- Folding Slot Configuration -->
03:02:55:  <client-type v='advanced'/>
03:02:55:  <extra-core-args v='  -service   '/>
03:02:55:
03:02:55:  <!-- HTTP Server -->
03:02:55:  <http-addresses v='127.0.0.1:7395'/>
03:02:55:
03:02:55:  <!-- Slot Control -->
03:02:55:  <power v='FULL'/>
03:02:55:
03:02:55:  <!-- User Information -->
03:02:55:  <passkey v='*****'/>
03:02:55:  <team v='32896'/>
03:02:55:  <user v='v00d00101'/>
03:02:55:
03:02:55:  <!-- Work Unit Control -->
03:02:55:  <next-unit-percentage v='100'/>
03:02:55:
03:02:55:  <!-- Folding Slots -->
03:02:55:  <slot id='1' type='GPU'>
03:02:55:    <!--cuda-index v='1'/-->
03:02:55:    <gpu-index v='1'/>
03:02:55:    <!--opencl-index v='1'/-->
03:02:55:  </slot>
03:02:55:</config>
03:02:56:WU00:FS01:Connecting to assign1.foldingathome.org:80
03:02:57:WU00:FS01:Assigned to work server 66.170.111.50
03:02:57:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:GP107 [GeForce GTX 1050 Ti]  2138 from 66.170.111.50
03:02:57:WU00:FS01:Connecting to 66.170.111.50:8080
03:02:58:ERROR:WU00:FS01:Exception: Server did not assign work unit
03:02:59:WU00:FS01:Connecting to assign1.foldingathome.org:80
03:03:00:WU00:FS01:Assigned to work server 3.133.76.19
03:03:00:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:GP107 [GeForce GTX 1050 Ti]  2138 from 3.133.76.19
03:03:00:WU00:FS01:Connecting to 3.133.76.19:8080
03:03:05:WU00:FS01:Downloading 19.15MiB
03:03:11:WU00:FS01:Download 43.07%
03:03:17:WU00:FS01:Download 96.58%
03:03:17:WU00:FS01:Download complete
03:03:17:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14458 run:0 clone:267 gen:40 core:0x22 unit:0x0000003d03854c135eb39a2ec67e163d
03:03:17:WU00:FS01:Starting
03:03:17:WU00:FS01:Running FahCore: /opt/fah/fahclient-1050/FAHCoreWrapper /opt/fah/fahclient-1050/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 00 -suffix 01 -version 706 -lifeline 16685 -checkpoint 10 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1 -service
03:03:17:WU00:FS01:Started FahCore on PID 16718
03:03:17:WU00:FS01:Core PID:16722
03:03:17:WU00:FS01:FahCore 0x22 started
03:03:18:WU00:FS01:0x22:*********************** Log Started 2020-06-20T03:03:17Z ***********************
03:03:18:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
03:03:18:WU00:FS01:0x22:       Core: Core22
03:03:18:WU00:FS01:0x22:       Type: 0x22
03:03:18:WU00:FS01:0x22:    Version: 0.0.10
03:03:18:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
03:03:18:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
03:03:18:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
03:03:18:WU00:FS01:0x22:       Date: Jun 16 2020
03:03:18:WU00:FS01:0x22:       Time: 15:55:31
03:03:18:WU00:FS01:0x22:   Revision: 147051aad40bcbec7d4b25105bbedfab425f1dc2
03:03:18:WU00:FS01:0x22:     Branch: core22-0.0.10
03:03:18:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
03:03:18:WU00:FS01:0x22:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
03:03:18:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
03:03:18:WU00:FS01:0x22:       Bits: 64
03:03:18:WU00:FS01:0x22:       Mode: Release
03:03:18:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
03:03:18:WU00:FS01:0x22:             <peastman@stanford.edu>
03:03:18:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 16718 -checkpoint 10
03:03:18:WU00:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
03:03:18:WU00:FS01:0x22:             1 -gpu 1 -service
03:03:18:WU00:FS01:0x22:************************************ libFAH ************************************
03:03:18:WU00:FS01:0x22:       Date: Jun 2 2020
03:03:18:WU00:FS01:0x22:       Time: 00:07:31
03:03:18:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
03:03:18:WU00:FS01:0x22:     Branch: HEAD
03:03:18:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
03:03:18:WU00:FS01:0x22:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
03:03:18:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
03:03:18:WU00:FS01:0x22:       Bits: 64
03:03:18:WU00:FS01:0x22:       Mode: Release
03:03:18:WU00:FS01:0x22:************************************ CBang *************************************
03:03:18:WU00:FS01:0x22:       Date: May 31 2020
03:03:18:WU00:FS01:0x22:       Time: 20:16:34
03:03:18:WU00:FS01:0x22:   Revision: 75fcee0b8e713cb47f5191a3689d5f4f07244c7f
03:03:18:WU00:FS01:0x22:     Branch: HEAD
03:03:18:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
03:03:18:WU00:FS01:0x22:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
03:03:18:WU00:FS01:0x22:             -fPIC
03:03:18:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
03:03:18:WU00:FS01:0x22:       Bits: 64
03:03:18:WU00:FS01:0x22:       Mode: Release
03:03:18:WU00:FS01:0x22:************************************ System ************************************
03:03:18:WU00:FS01:0x22:        CPU: AMD A8-9600 RADEON R7, 10 COMPUTE CORES 4C+6G
03:03:18:WU00:FS01:0x22:     CPU ID: AuthenticAMD Family 21 Model 101 Stepping 1
03:03:18:WU00:FS01:0x22:       CPUs: 4
03:03:18:WU00:FS01:0x22:     Memory: 7.26GiB
03:03:18:WU00:FS01:0x22:Free Memory: 5.62GiB
03:03:18:WU00:FS01:0x22:    Threads: POSIX_THREADS
03:03:18:WU00:FS01:0x22: OS Version: 4.9
03:03:18:WU00:FS01:0x22:Has Battery: false
03:03:18:WU00:FS01:0x22: On Battery: false
03:03:18:WU00:FS01:0x22: UTC Offset: 1
03:03:18:WU00:FS01:0x22:        PID: 16722
03:03:18:WU00:FS01:0x22:        CWD: /opt/fah/fahclient-1050/work
03:03:18:WU00:FS01:0x22:********************************************************************************
03:03:18:WU00:FS01:0x22:Project: 14458 (Run 0, Clone 267, Gen 40)
03:03:18:WU00:FS01:0x22:Unit: 0x0000003d03854c135eb39a2ec67e163d
03:03:18:WU00:FS01:0x22:Reading tar file core.xml
03:03:18:WU00:FS01:0x22:Reading tar file integrator.xml
03:03:18:WU00:FS01:0x22:Reading tar file state.xml
03:03:18:WU00:FS01:0x22:Reading tar file system.xml
03:03:18:WU00:FS01:0x22:Digital signatures verified
03:03:18:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
03:03:18:WU00:FS01:0x22:Version 0.0.10
03:03:18:WU00:FS01:0x22:  Checkpoint write interval: 100000 steps (5%) [20 total]
03:03:18:WU00:FS01:0x22:  JSON viewer frame write interval: 20000 steps (1%) [100 total]
03:03:18:WU00:FS01:0x22:  XTC frame write interval: 20000 steps (1%) [100 total]
03:03:18:WU00:FS01:0x22:  Global context and integrator variables write interval: disabled
03:03:33:WU00:FS01:0x22:ERROR:exception: Illegal value for DeviceIndex: 1
03:03:33:WU00:FS01:0x22:Saving result file ../logfile_01.txt
03:03:33:WU00:FS01:0x22:Saving result file science.log
03:03:33:WU00:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
03:03:33:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:03:33:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:14458 run:0 clone:267 gen:40 core:0x22 unit:0x0000003d03854c135eb39a2ec67e163d
03:03:33:WU00:FS01:Uploading 2.47KiB to 3.133.76.19
03:03:33:WU00:FS01:Connecting to 3.133.76.19:8080
03:03:34:WU00:FS01:Upload complete
03:03:34:WU01:FS01:Connecting to assign1.foldingathome.org:80
03:03:34:WU00:FS01:Server responded WORK_ACK (400)
03:03:34:WU00:FS01:Cleaning up
03:03:35:WU01:FS01:Assigned to work server 66.170.111.50
03:03:35:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GP107 [GeForce GTX 1050 Ti]  2138 from 66.170.111.50
03:03:35:WU01:FS01:Connecting to 66.170.111.50:8080
03:03:41:WU01:FS01:Downloading 22.41MiB
03:03:42:Caught signal SIGINT(2) on PID 16685
03:03:42:Exiting, please wait. . .
03:03:42:ERROR:WU01:FS01:Exception: Transfer failed
03:03:43:Clean exit

So I ran it as root. it works, but I would obviously prefer to not have to run it as root.. This copy of debian was folding on an AM3+ board without issue 3 days ago. the only thing that has changed is the cpu and motherboard. So why wont this one fold as a user?

Anyone got any ideas, as im kind of out of them. For now I will run it as root to get around whatever issue has befallen it.

[bruce]

Why are you starting the FAHClient in the /opt/fah/fahclient-1050 directory?

[JimboPalmer]

I am in over my head but lets try until an expert gets here!

03:02:55: GPUs: 2
03:02:55: GPU 0: Bus:0 Slot:1 Func:0 AMD:5 Carrizo [Radeon R7/R6/R5 Series]
03:02:55: GPU 1: Bus:7 Slot:0 Func:0 NVIDIA:7 GP107 [GeForce GTX 1050 Ti] 2138
03:02:55: CUDA Device 0: Platform:0 Device:0 Bus:7 Slot:0 Compute:6.1 Driver:9.0
03:02:55:OpenCL Device 0: Platform:0 Device:0 Bus:7 Slot:0 Compute:1.2 Driver:384.130
03:02:55:OpenCL Device 1: Platform:1 Device:0 Bus:NA Slot:NA Compute:1.1 Driver:13.0

GPU 0 has no OpenCL driver F@H can see, so we are done with GPU 0

GPU 1 should use CUDA device 0 and OpenCL Device 0

If I had to guess, OpenCL device 1 is an Intel driver

Instead you have:

03:02:55:
03:02:55: <!-- Folding Slots -->
03:02:55: <slot id='1' type='GPU'>
03:02:55: <!--cuda-index v='1'/-->
03:02:55: <gpu-index v='1'/>
03:02:55: <!--opencl-index v='1'/-->
03:02:55: </slot>

So I think the Nvidia card would fold with

03:02:55:
03:02:55: <!-- Folding Slots -->
03:02:55: <slot id='1' type='GPU'>
03:02:55: <gpu-index v='1'/>
03:02:55: <--cuda-index v='0'/-->
03:02:55: <--opencl-index v='0'/-->
03:02:55: </slot>

But that is just a guess.

That is an old Nvidia driver.

[v00d00]

Why are you starting the FAHClient in the /opt/fah/fahclient-1050 directory?

Because thats where all folding directories live (im not being arsey, its just how where it its installed, it should not make any real difference where you install software as long as the core files reside in the same directory). Im not so bothered about the particulars of that since thats how I've run folding for the last 20 years without issue. Its miscellaneous software and that mountpoint is a hard drive not SSD, so it doesnt suffer from wear in the same. Seemed logical to put it there. I suppose in the grand shceme of things I could have used /usr/local, but I chose /opt instead. it is also running under user fah from group fah. It also means in that home directory you would find fahclient-970 and fahclient-smp. Each one does one specific folding process, either gpu or cpu. Makes debugging stuff easier,

I am in over my head but lets try until an expert gets here!

03:02:55: GPUs: 2
03:02:55: GPU 0: Bus:0 Slot:1 Func:0 AMD:5 Carrizo [Radeon R7/R6/R5 Series]
03:02:55: GPU 1: Bus:7 Slot:0 Func:0 NVIDIA:7 GP107 [GeForce GTX 1050 Ti] 2138
03:02:55: CUDA Device 0: Platform:0 Device:0 Bus:7 Slot:0 Compute:6.1 Driver:9.0
03:02:55:OpenCL Device 0: Platform:0 Device:0 Bus:7 Slot:0 Compute:1.2 Driver:384.130
03:02:55:OpenCL Device 1: Platform:1 Device:0 Bus:NA Slot:NA Compute:1.1 Driver:13.0

GPU 0 has no OpenCL driver F@H can see, so we are done with GPU 0

GPU 1 should use CUDA device 0 and OpenCL Device 0

If I had to guess, OpenCL device 1 is an Intel driver

Instead you have:

03:02:55:
03:02:55: <!-- Folding Slots -->
03:02:55: <slot id='1' type='GPU'>
03:02:55: <!--cuda-index v='1'/-->
03:02:55: <gpu-index v='1'/>
03:02:55: <!--opencl-index v='1'/-->
03:02:55: </slot>

So I think the Nvidia card would fold with

03:02:55:
03:02:55: <!-- Folding Slots -->
03:02:55: <slot id='1' type='GPU'>
03:02:55: <gpu-index v='1'/>
03:02:55: <--cuda-index v='0'/-->
03:02:55: <--opencl-index v='0'/-->
03:02:55: </slot>

But that is just a guess.

That is an old Nvidia driver.

I reverted the driver from 440 to 384.130 which was working fine 3 days ago. Ignore the Carrizo R7 for now. While I would love to have that fold, for now the hassle of making it do so, isn't worth touching.

As a user, If i run opencl-index 1, which is what the nvidia should use, I get an error in the log and lots of Bad workunits, but if I remove it, it doesnt work full stop.

The only opencl-drivers currently installed are:

Code: Select all

$ locate opencl | grep .so
/usr/lib/i386-linux-gnu/libnvidia-opencl.so.1
/usr/lib/i386-linux-gnu/libnvidia-opencl.so.384.130
/usr/lib/i386-linux-gnu/wine/opencl.dll.so
/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.384.130
/usr/lib/x86_64-linux-gnu/wine/opencl.dll.so

I will be upgrading tomorrow to Devuan 10 tomorrow and picking up the latest kernel from backports, so hopefully that will fix the problem. At the same time it could be some other part of the system, but with never versions of software, it might resolve.

My main conundrum is why does it not fold as a user today when it did 4 days back. The OS is the same amd hasnt been updated in a week. The only thing that changed was some of the hardware. Unless of course some feature in the new cpu is causing the issue.

If you cant help, dont worry, it may be that I have to seriously hack on this problem and read a lot of google. At the same time I now have my 1050 runnign at full power, albeit as root, but it is folding once again. With time I will work out how to get it back on to its own user.

Looking at the log again (after sleeping) I notice this mapping that makes no sense, but explains the cuda/opencli errors. The fact that the Carrizo, which is gpu 0, maps to opencl-index 1 makes no sense at all. Thus the nvidia maps to opencl-index 0.

Code: Select all

03:02:55:          GPU 0: Bus:0 Slot:1 Func:0 AMD:5 Carrizo [Radeon R7/R6/R5 Series]
03:02:55:          GPU 1: Bus:7 Slot:0 Func:0 NVIDIA:7 GP107 [GeForce GTX 1050 Ti] 2138
03:02:55:  CUDA Device 0: Platform:0 Device:0 Bus:7 Slot:0 Compute:6.1 Driver:9.0
03:02:55:OpenCL Device 0: Platform:0 Device:0 Bus:7 Slot:0 Compute:1.2 Driver:384.130
03:02:55:OpenCL Device 1: Platform:1 Device:0 Bus:NA Slot:NA Compute:1.1 Driver:13.0

Definitely reinstall everything time. That shouldnt look like that at all.

[v00d00]

LOL. Ok its fixed. The cause of the problem was the fact that the opencl/cuda indexes didnt map to there logical adapters correctly. It now runs as a user like it did before.

Will probably still reinstall everything and see if I can get those mappings back to normal.

Thanks for your help guys.