Project: 11713 (Run 5, Clone 269, Gen 18)

Moderators: Site Moderators, FAHC Science Team

Post Reply
rwh202
Posts: 425
Joined: Mon Nov 15, 2010 8:51 pm
Hardware configuration: 8x GTX 1080
3x GTX 1080 Ti
3x GTX 1060
Various other bits and pieces
Location: South Coast, UK

Project: 11713 (Run 5, Clone 269, Gen 18)

Post by rwh202 »

I don't know whether this rig was just having a hissy fit, but the problem with this WU persisted across a restart with the system folding happily before and after on different WUs.

Code: Select all

11:13:08:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:11713 run:5 clone:269 gen:18 core:0x21 unit:0x0000001b8ca304e75adf74183284c129
11:13:08:WU00:FS01:Starting
11:13:08:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1149 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
11:13:08:WU00:FS01:Started FahCore on PID 5000
11:13:08:WU00:FS01:Core PID:5004
11:13:08:WU00:FS01:FahCore 0x21 started
11:13:09:WU00:FS01:0x21:*********************** Log Started 2018-05-21T11:13:08Z ***********************
11:13:09:WU00:FS01:0x21:Project: 11713 (Run 5, Clone 269, Gen 18)
11:13:09:WU00:FS01:0x21:Unit: 0x0000001b8ca304e75adf74183284c129
11:13:09:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:13:09:WU00:FS01:0x21:Machine: 1
11:13:09:WU00:FS01:0x21:Reading tar file core.xml
11:13:09:WU00:FS01:0x21:Reading tar file integrator.xml
11:13:09:WU00:FS01:0x21:Reading tar file state.xml
11:13:09:WU00:FS01:0x21:Reading tar file system.xml
11:13:09:WU00:FS01:0x21:Digital signatures verified
11:13:09:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:13:09:WU00:FS01:0x21:Version 0.0.18
11:13:10:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
11:13:10:WU00:FS01:Starting
11:13:10:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1149 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
11:13:10:WU00:FS01:Started FahCore on PID 5007
11:13:10:WU00:FS01:Core PID:5011
11:13:10:WU00:FS01:FahCore 0x21 started
11:13:10:WU00:FS01:0x21:*********************** Log Started 2018-05-21T11:13:10Z ***********************
11:13:10:WU00:FS01:0x21:Project: 11713 (Run 5, Clone 269, Gen 18)
11:13:10:WU00:FS01:0x21:Unit: 0x0000001b8ca304e75adf74183284c129
11:13:10:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:13:10:WU00:FS01:0x21:Machine: 1
11:13:10:WU00:FS01:0x21:Reading tar file core.xml
11:13:10:WU00:FS01:0x21:Reading tar file integrator.xml
11:13:10:WU00:FS01:0x21:Reading tar file state.xml
11:13:10:WU00:FS01:0x21:Reading tar file system.xml
11:13:10:WU00:FS01:0x21:Digital signatures verified
11:13:10:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:13:10:WU00:FS01:0x21:Version 0.0.18
11:13:11:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
11:14:10:WU00:FS01:Starting
11:14:10:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1149 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
11:14:10:WU00:FS01:Started FahCore on PID 5020
11:14:10:WU00:FS01:Core PID:5024
11:14:10:WU00:FS01:FahCore 0x21 started
11:14:11:WU00:FS01:0x21:*********************** Log Started 2018-05-21T11:14:10Z ***********************
11:14:11:WU00:FS01:0x21:Project: 11713 (Run 5, Clone 269, Gen 18)
11:14:11:WU00:FS01:0x21:Unit: 0x0000001b8ca304e75adf74183284c129
11:14:11:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:14:11:WU00:FS01:0x21:Machine: 1
11:14:11:WU00:FS01:0x21:Reading tar file core.xml
11:14:11:WU00:FS01:0x21:Reading tar file integrator.xml
11:14:11:WU00:FS01:0x21:Reading tar file state.xml
11:14:11:WU00:FS01:0x21:Reading tar file system.xml
11:14:11:WU00:FS01:0x21:Digital signatures verified
11:14:11:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:14:11:WU00:FS01:0x21:Version 0.0.18
11:14:12:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
11:15:10:WU00:FS01:Starting
11:15:10:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1149 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
11:15:10:WU00:FS01:Started FahCore on PID 5032
11:15:10:WU00:FS01:Core PID:5036
11:15:10:WU00:FS01:FahCore 0x21 started
11:15:11:WU00:FS01:0x21:*********************** Log Started 2018-05-21T11:15:10Z ***********************
11:15:11:WU00:FS01:0x21:Project: 11713 (Run 5, Clone 269, Gen 18)
11:15:11:WU00:FS01:0x21:Unit: 0x0000001b8ca304e75adf74183284c129
11:15:11:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:15:11:WU00:FS01:0x21:Machine: 1
11:15:11:WU00:FS01:0x21:Reading tar file core.xml
11:15:11:WU00:FS01:0x21:Reading tar file integrator.xml
11:15:11:WU00:FS01:0x21:Reading tar file state.xml
11:15:11:WU00:FS01:0x21:Reading tar file system.xml
11:15:11:WU00:FS01:0x21:Digital signatures verified
11:15:11:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:15:11:WU00:FS01:0x21:Version 0.0.18
11:15:12:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
11:16:10:WU00:FS01:Starting
11:16:10:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1149 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
11:16:10:WU00:FS01:Started FahCore on PID 5040
11:16:10:WU00:FS01:Core PID:5044
11:16:10:WU00:FS01:FahCore 0x21 started
11:16:11:WU00:FS01:0x21:*********************** Log Started 2018-05-21T11:16:10Z ***********************
11:16:11:WU00:FS01:0x21:Project: 11713 (Run 5, Clone 269, Gen 18)
11:16:11:WU00:FS01:0x21:Unit: 0x0000001b8ca304e75adf74183284c129
11:16:11:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:16:11:WU00:FS01:0x21:Machine: 1
11:16:11:WU00:FS01:0x21:Reading tar file core.xml
11:16:11:WU00:FS01:0x21:Reading tar file integrator.xml
11:16:11:WU00:FS01:0x21:Reading tar file state.xml
11:16:11:WU00:FS01:0x21:Reading tar file system.xml
11:16:11:WU00:FS01:0x21:Digital signatures verified
11:16:11:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:16:11:WU00:FS01:0x21:Version 0.0.18
11:16:12:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
11:17:10:WU00:FS01:Starting
11:17:10:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1149 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
11:17:10:WU00:FS01:Started FahCore on PID 5059
11:17:10:WU00:FS01:Core PID:5063
11:17:10:WU00:FS01:FahCore 0x21 started
11:17:11:WU00:FS01:0x21:*********************** Log Started 2018-05-21T11:17:10Z ***********************
11:17:11:WU00:FS01:0x21:Project: 11713 (Run 5, Clone 269, Gen 18)
11:17:11:WU00:FS01:0x21:Unit: 0x0000001b8ca304e75adf74183284c129
11:17:11:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:17:11:WU00:FS01:0x21:Machine: 1
11:17:11:WU00:FS01:0x21:Reading tar file core.xml
11:17:11:WU00:FS01:0x21:Reading tar file integrator.xml
11:17:11:WU00:FS01:0x21:Reading tar file state.xml
11:17:11:WU00:FS01:0x21:Reading tar file system.xml
11:17:11:WU00:FS01:0x21:Digital signatures verified
11:17:11:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:17:11:WU00:FS01:0x21:Version 0.0.18
11:17:12:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
11:18:10:WU00:FS01:Starting
11:18:10:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1149 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
11:18:10:WU00:FS01:Started FahCore on PID 5066
11:18:10:WU00:FS01:Core PID:5070
11:18:10:WU00:FS01:FahCore 0x21 started
11:18:11:WU00:FS01:0x21:*********************** Log Started 2018-05-21T11:18:10Z ***********************
11:18:11:WU00:FS01:0x21:Project: 11713 (Run 5, Clone 269, Gen 18)
11:18:11:WU00:FS01:0x21:Unit: 0x0000001b8ca304e75adf74183284c129
11:18:11:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:18:11:WU00:FS01:0x21:Machine: 1
11:18:11:WU00:FS01:0x21:Reading tar file core.xml
11:18:11:WU00:FS01:0x21:Reading tar file integrator.xml
11:18:11:WU00:FS01:0x21:Reading tar file state.xml
11:18:11:WU00:FS01:0x21:Reading tar file system.xml
11:18:11:WU00:FS01:0x21:Digital signatures verified
11:18:11:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:18:11:WU00:FS01:0x21:Version 0.0.18
11:18:12:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
11:19:10:WU00:FS01:Starting
11:19:10:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1149 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
11:19:10:WU00:FS01:Started FahCore on PID 5078
11:19:10:WU00:FS01:Core PID:5082
11:19:10:WU00:FS01:FahCore 0x21 started
11:19:11:WU00:FS01:0x21:*********************** Log Started 2018-05-21T11:19:10Z ***********************
11:19:11:WU00:FS01:0x21:Project: 11713 (Run 5, Clone 269, Gen 18)
11:19:11:WU00:FS01:0x21:Unit: 0x0000001b8ca304e75adf74183284c129
11:19:11:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:19:11:WU00:FS01:0x21:Machine: 1
11:19:11:WU00:FS01:0x21:Reading tar file core.xml
11:19:11:WU00:FS01:0x21:Reading tar file integrator.xml
11:19:11:WU00:FS01:0x21:Reading tar file state.xml
11:19:11:WU00:FS01:0x21:Reading tar file system.xml
11:19:11:WU00:FS01:0x21:Digital signatures verified
11:19:11:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:19:11:WU00:FS01:0x21:Version 0.0.18
11:19:12:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
ad infinitum, even across a reboot and a driver reinstall.

Linux Mint 17.3 and a single GTX 1080 FE
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Project: 11713 (Run 5, Clone 269, Gen 18)

Post by bruce »

Please show the first couple of pages of fah.log.

Which version of FAHClient are you running?
rwh202
Posts: 425
Joined: Mon Nov 15, 2010 8:51 pm
Hardware configuration: 8x GTX 1080
3x GTX 1080 Ti
3x GTX 1060
Various other bits and pieces
Location: South Coast, UK

Re: Project: 11713 (Run 5, Clone 269, Gen 18)

Post by rwh202 »

Running 7.4.4 and here's the head of the log:

Code: Select all

*********************** Log Started 2018-05-23T07:06:32Z ***********************
07:06:32:************************* Folding@home Client *************************
07:06:32:    Website: http://folding.stanford.edu/
07:06:32:  Copyright: (c) 2009-2014 Stanford University
07:06:32:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:06:32:       Args: --child --lifeline 1153 /etc/fahclient/config.xml --run-as
07:06:32:             fahclient --pid-file=/var/run/fahclient.pid --daemon
07:06:32:     Config: /etc/fahclient/config.xml
07:06:32:******************************** Build ********************************
07:06:32:    Version: 7.4.4
07:06:32:       Date: Mar 4 2014
07:06:32:       Time: 12:02:38
07:06:32:    SVN Rev: 4130
07:06:32:     Branch: fah/trunk/client
07:06:32:   Compiler: GNU 4.4.7
07:06:32:    Options: -std=gnu++98 -O3 -funroll-loops -mfpmath=sse -ffast-math
07:06:32:             -fno-unsafe-math-optimizations -msse2
07:06:32:   Platform: linux2 3.2.0-1-amd64
07:06:32:       Bits: 64
07:06:32:       Mode: Release
07:06:32:******************************* System ********************************
07:06:32:        CPU: Intel(R) Pentium(R) Gold G5400 CPU @ 3.70GHz
07:06:32:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 10
07:06:32:       CPUs: 4
07:06:32:     Memory: 7.74GiB
07:06:32:Free Memory: 7.37GiB
07:06:32:    Threads: POSIX_THREADS
07:06:32: OS Version: 4.4
07:06:32:Has Battery: false
07:06:32: On Battery: false
07:06:32: UTC Offset: 1
07:06:32:        PID: 1155
07:06:32:        CWD: /var/lib/fahclient
07:06:32:         OS: Linux 4.4.0-124-generic x86_64
07:06:32:    OS Arch: AMD64
07:06:32:       GPUs: 2
07:06:32:      GPU 0: NVIDIA:7 GP104 [GeForce GTX 1080] 8873
07:06:32:      GPU 1: UNSUPPORTED: NV3 [PCI]
07:06:32:       CUDA: 6.1
07:06:32:CUDA Driver: 9000
07:06:32:***********************************************************************
07:06:32:<config>
07:06:32:  <!-- Client Control -->
07:06:32:  <fold-anon v='true'/>
07:06:32:
07:06:32:  <!-- HTTP Server -->
07:06:32:  <allow v='127.0.0.1 192.168.0.0/24'/>
07:06:32:
07:06:32:  <!-- Network -->
07:06:32:  <proxy v=':8080'/>
07:06:32:
07:06:32:  <!-- Remote Command Server -->
07:06:32:  <password v='******'/>
07:06:32:
07:06:32:  <!-- Slot Control -->
07:06:32:  <power v='full'/>
07:06:32:
07:06:32:  <!-- User Information -->
07:06:32:  <passkey v='********************************'/>
07:06:32:  <team v='224497'/>
07:06:32:  <user v='TML_ALL_1Fb7XpTZN9gefApL87L51CY34aT7DZKCUu'/>
07:06:32:
07:06:32:  <!-- Work Unit Control -->
07:06:32:  <next-unit-percentage v='100'/>
07:06:32:
07:06:32:  <!-- Folding Slots -->
07:06:32:  <slot id='1' type='GPU'>
07:06:32:    <paused v='true'/>
07:06:32:  </slot>
07:06:32:</config>
Thanks, Rob
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Project: 11713 (Run 5, Clone 269, Gen 18)

Post by bruce »

Project: 11713 (Run 5, Clone 269, Gen 18) was reassigned and failed again. Normally after being retried on 3 systems, the trajectory will be suspended as a bad WU. We'll see what happens.
Post Reply