171.67.108.200 - Internal Server Error

[billford]

Been getting this for the last 15 minutes or so:

Code: Select all

20:09:14:WU01:FS00:Connecting to 171.67.108.200:8080
20:09:15:WARNING:WU01:FS00:Failed to get assignment from '171.67.108.200:8080': 10001: Server responded: HTTP_INTERNAL_SERVER_ERROR
20:09:15:WU01:FS00:Connecting to 171.67.108.204:80
20:09:15:WARNING:WU01:FS00:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
20:09:15:ERROR:WU01:FS00:Exception: Could not get an assignment

Config:

Code: Select all

08:07:10:************************* Folding@home Client *************************
08:07:10:    Website: http://folding.stanford.edu/
08:07:10:  Copyright: (c) 2009-2014 Stanford University
08:07:10:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
08:07:10:       Args: 
08:07:10:     Config: /Library/Application Support/FAHClient/config.xml
08:07:10:******************************** Build ********************************
08:07:10:    Version: 7.4.4
08:07:10:       Date: Mar 4 2014
08:07:10:       Time: 20:27:54
08:07:10:    SVN Rev: 4130
08:07:10:     Branch: fah/trunk/client
08:07:10:   Compiler: GNU 4.2.1 (Apple Inc. build 5666) (dot 3)
08:07:10:    Options: -std=gnu++98 -O3 -funroll-loops -mfpmath=sse -ffast-math
08:07:10:             -fno-unsafe-math-optimizations -msse3 -arch x86_64
08:07:10:             -mmacosx-version-min=10.6
08:07:10:   Platform: darwin 10.8.0
08:07:10:       Bits: 64
08:07:10:       Mode: Release
08:07:10:******************************* System ********************************
08:07:10:        CPU: Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz
08:07:10:     CPU ID: GenuineIntel Family 6 Model 58 Stepping 9
08:07:10:       CPUs: 4
08:07:10:     Memory: 8.00GiB
08:07:10:Free Memory: 3.83GiB
08:07:10:    Threads: POSIX_THREADS
08:07:10: OS Version: 10.8
08:07:10:Has Battery: true
08:07:10: On Battery: false
08:07:10: UTC Offset: 1
08:07:10:        PID: 3325
08:07:10:        CWD: /Library/Application Support/FAHClient
08:07:10:         OS: Darwin 12.6.0 x86_64
08:07:10:    OS Arch: AMD64
08:07:10:       GPUs: 1
08:07:10:      GPU 0: NVIDIA:3 GK104 [GeForce GTX 675MX]
08:07:10:       CUDA: Not detected
08:07:10:***********************************************************************
08:07:10:<config>
08:07:10:  <!-- Folding Slot Configuration -->
08:07:10:  <client-type v='advanced'/>
08:07:10:
08:07:10:  <!-- HTTP Server -->
08:07:10:  <allow v='127.0.0.1  192.168.1.0/24'/>
08:07:10:
08:07:10:  <!-- Network -->
08:07:10:  <proxy v=':8080'/>
08:07:10:
08:07:10:  <!-- Remote Command Server -->
08:07:10:  <command-allow-no-pass v='127.0.0.1 192.168.1.0/24'/>
08:07:10:  <password v=''/>
08:07:10:
08:07:10:  <!-- Slot Control -->
08:07:10:  <power v='medium'/>
08:07:10:
08:07:10:  <!-- User Information -->
08:07:10:  <passkey v='********************************'/>
08:07:10:  <user v='[redacted]'/>
08:07:10:
08:07:10:  <!-- Work Unit Control -->
08:07:10:  <next-unit-percentage v='100'/>
08:07:10:
08:07:10:  <!-- Folding Slots -->
08:07:10:  <slot id='0' type='CPU'/>
08:07:10:</config>

[billford]

Just to note- putting the IP address into a browser works OK (ports 80 and 8080), displays the FAH logo and "Assignment Server Version 3.4.25".

[fangfufu]

I have the same problem. It has been going on for a while now.

Code: Select all

*********************** Log Started 2015-05-06T21:31:04Z ***********************
21:31:04:************************* Folding@home Client *************************
21:31:04:    Website: http://folding.stanford.edu/
21:31:04:  Copyright: (c) 2009-2014 Stanford University
21:31:04:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
21:31:04:       Args: --child --lifeline 7996 /etc/fahclient/config.xml --run-as
21:31:04:             fahclient --pid-file=/var/run/fahclient.pid --daemon
21:31:04:     Config: /etc/fahclient/config.xml
21:31:04:******************************** Build ********************************
21:31:04:    Version: 7.4.4
21:31:04:       Date: Mar 4 2014
21:31:04:       Time: 12:02:38
21:31:04:    SVN Rev: 4130
21:31:04:     Branch: fah/trunk/client
21:31:04:   Compiler: GNU 4.4.7
21:31:04:    Options: -std=gnu++98 -O3 -funroll-loops -mfpmath=sse -ffast-math
21:31:04:             -fno-unsafe-math-optimizations -msse2
21:31:04:   Platform: linux2 3.2.0-1-amd64
21:31:04:       Bits: 64
21:31:04:       Mode: Release
21:31:04:******************************* System ********************************
21:31:04:        CPU: Intel(R) Core(TM) i5-4200M CPU @ 2.50GHz
21:31:04:     CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
21:31:04:       CPUs: 4
21:31:04:     Memory: 15.61GiB
21:31:04:Free Memory: 10.63GiB
21:31:04:    Threads: POSIX_THREADS
21:31:04: OS Version: 3.16
21:31:04:Has Battery: true
21:31:04: On Battery: false
21:31:04: UTC Offset: 1
21:31:04:        PID: 7998
21:31:04:        CWD: /var/lib/fahclient
21:31:04:         OS: Linux 3.16.0-4-amd64 x86_64
21:31:04:    OS Arch: AMD64
21:31:04:       GPUs: 1
21:31:04:      GPU 0: NVIDIA:3 GK208 [GeForce GT 730M]
21:31:04:       CUDA: 3.5
21:31:04:CUDA Driver: 6050
21:31:04:***********************************************************************
21:31:04:<config>
21:31:04:  <!-- Folding Core -->
21:31:04:  <checkpoint v='30'/>
21:31:04:
21:31:04:  <!-- Network -->
21:31:04:  <proxy v='192.168.80.5:3128'/>
21:31:04:
21:31:04:  <!-- Slot Control -->
21:31:04:  <power v='full'/>
21:31:04:
21:31:04:  <!-- User Information -->
21:31:04:  <passkey v='********************************'/>
21:31:04:  <team v='50959'/>
21:31:04:  <user v='fangfufu'/>
21:31:04:
21:31:04:  <!-- Folding Slots -->
21:31:04:  <slot id='0' type='CPU'>
21:31:04:    <cpus v='1'/>
21:31:04:  </slot>
21:31:04:</config>
21:31:04:Switching to user fahclient
21:31:04:Trying to access database...
21:31:04:Successfully acquired database lock
21:31:04:Enabled folding slot 00: READY cpu:1
21:31:05:WU00:FS00:Connecting to 171.67.108.200:8080
21:31:06:WARNING:WU00:FS00:Failed to get assignment from '171.67.108.200:8080': 10001: Server responded: HTTP_INTERNAL_SERVER_ERROR
21:31:06:WU00:FS00:Connecting to 171.67.108.204:80
21:31:08:WARNING:WU00:FS00:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
21:31:08:ERROR:WU00:FS00:Exception: Could not get an assignment
21:31:08:WU00:FS00:Connecting to 171.67.108.200:8080
21:31:09:WARNING:WU00:FS00:Failed to get assignment from '171.67.108.200:8080': 10001: Server responded: HTTP_INTERNAL_SERVER_ERROR
21:31:09:WU00:FS00:Connecting to 171.67.108.204:80
21:31:10:WARNING:WU00:FS00:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
21:31:10:ERROR:WU00:FS00:Exception: Could not get an assignment
21:32:08:WU00:FS00:Connecting to 171.67.108.200:8080
21:32:09:WARNING:WU00:FS00:Failed to get assignment from '171.67.108.200:8080': 10001: Server responded: HTTP_INTERNAL_SERVER_ERROR
21:32:09:WU00:FS00:Connecting to 171.67.108.204:80
21:32:09:WARNING:WU00:FS00:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
21:32:09:ERROR:WU00:FS00:Exception: Could not get an assignment

[billford]

The iMac's CPU is probably glad of the break- first time it's been below 70ºC for weeks

[fangfufu]

It feels so weird, my CPU fan sometimes actually stop now. It used to spin constantly at a low rate. My room is so quiet. It is unnatural.

[DutchForce]

I've just checked my logs and also noticed this message, but I got (GPU) assignments from the second AS.

Folding system with 2x GTX780 Ti's (client-type = advanced)
No problem with AS 171.67.108.200:

Code: Select all

******************************* Date: 2015-05-06 *******************************
18:25:28:WU01:FS00:Connecting to 171.67.108.200:80
18:25:29:WU01:FS00:Assigned to work server 171.64.65.58
18:25:29:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.58
18:25:29:WU01:FS00:Connecting to 171.64.65.58:8080
18:25:30:WU01:FS00:Downloading 884.76KiB

Problem with AS 171.67.108.200:

Code: Select all

******************************* Date: 2015-05-06 *******************************
19:46:41:WU02:FS01:Connecting to 171.67.108.200:80
19:46:43:WARNING:WU02:FS01:Failed to get assignment from '171.67.108.200:80': 10001: Server responded: HTTP_INTERNAL_SERVER_ERROR
19:46:43:WU02:FS01:Connecting to 171.67.108.204:80
19:46:43:WU02:FS01:Assigned to work server 171.64.65.93
19:46:43:WU02:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:GK110 [GeForce GTX 780 Ti] from 171.64.65.93
19:46:43:WU02:FS01:Connecting to 171.64.65.93:8080
19:46:44:WU02:FS01:Downloading 3.62MiB

Folding system with 3x GTX660 Ti's (client-type = advanced):

Code: Select all

******************************* Date: 2015-05-06 *******************************
21:26:43:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.200:80': 10001: Server responded: HTTP_INTERNAL_SERVER_ERROR
21:26:43:WU01:FS01:Connecting to 171.67.108.204:80
21:26:44:WU01:FS01:Assigned to work server 171.64.65.84
21:26:44:WU01:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GK104 [GeForce GTX 660 Ti] from 171.64.65.84
21:26:44:WU01:FS01:Connecting to 171.64.65.84:8080
21:26:45:WU01:FS01:Downloading 3.54MiB

[billford]

It feels so weird, my CPU fan sometimes actually stop now. It used to spin constantly at a low rate. My room is so quiet. It is unnatural.

I'll probably feel the same in about 20 minutes when one of the Linux (smp) clients fails to get a WU

[billford]

but I got (GPU) assignments from the second AS.

I'll confirm that here (or not) in about 15 minutes...

[fangfufu]

I have decided to change the settings to fold with 2 cores, at least it gets in a work unit. My 24 hour average is already very low, I need to keep folding. I normally fold with only on 1 core - otherwise the system sometimes feel slow.

[billford]

All mine are on cpu:3 at the moment- the Linux machines because the GPUs want the 4th core, and the iMac because the weather is getting warmer, and so was the processor on cpu:4.

I don't notice any slowing down, except for one program (Handbrake) which will grab any clock cycle it can lay its hands on.

[billford]

Both GTX 980's working OK:

Code: Select all

22:16:17:WU00:FS01:Connecting to 171.67.108.200:80
.
.
22:16:18:WARNING:WU00:FS01:Failed to get assignment from '171.67.108.200:80': 10001: Server responded: HTTP_INTERNAL_SERVER_ERROR
22:16:18:WU00:FS01:Connecting to 171.67.108.204:80
22:16:18:WU00:FS01:Assigned to work server 171.64.65.56
22:16:18:WU00:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GM204 [GeForce GTX 980] from 171.64.65.56
22:16:18:WU00:FS01:Connecting to 171.64.65.56:8080
22:16:19:WU00:FS01:Downloading 887.21KiB
.
.
22:16:21:WU00:FS01:Download complete
22:16:21:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9411 run:871 clone:0 gen:93 core:0x17 unit:0x0000006dab40413854d27c6a96919df8
22:16:21:WU00:FS01:Starting
22:16:21:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 00 -suffix 01 -version 704 -lifeline 1564 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
22:16:21:WU00:FS01:Started FahCore on PID 23948
22:16:21:WU00:FS01:Core PID:23952
22:16:21:WU00:FS01:FahCore 0x17 started
22:16:21:WU00:FS01:0x17:*********************** Log Started 2015-05-06T22:16:21Z ***********************
22:16:21:WU00:FS01:0x17:Project: 9411 (Run 871, Clone 0, Gen 93)
22:16:21:WU00:FS01:0x17:Unit: 0x0000006dab40413854d27c6a96919df8
22:16:21:WU00:FS01:0x17:CPU: 0x00000000000000000000000000000000
22:16:21:WU00:FS01:0x17:Machine: 1
22:16:21:WU00:FS01:0x17:Reading tar file state.xml
22:16:21:WU00:FS01:0x17:Reading tar file system.xml
22:16:21:WU00:FS01:0x17:Reading tar file integrator.xml
22:16:21:WU00:FS01:0x17:Reading tar file core.xml
22:16:21:WU00:FS01:0x17:Digital signatures verified
22:16:26:WU01:FS01:Upload 14.22%
22:16:27:WU00:FS01:0x17:Completed 0 out of 16000000 steps (0%)

Internal error on first AS still there.

Linux smp failing:

Code: Select all

22:22:39:WU01:FS00:Connecting to 171.67.108.200:8080
22:22:40:WARNING:WU01:FS00:Failed to get assignment from '171.67.108.200:8080': 10001: Server responded: HTTP_INTERNAL_SERVER_ERROR
22:22:40:WU01:FS00:Connecting to 171.67.108.204:80
22:22:40:WARNING:WU01:FS00:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
22:22:40:ERROR:WU01:FS00:Exception: Could not get an assignment

[billford]

Now there's odd... the iMac was set to "Medium" with cpu:4 so it was running with cpu:3. I changed it to "Full" on cpu:2 and it promptly picked up a WU.

Tried the same on the Linux client (normally set to "Full" on cpu:3) and it didn't work

Maybe of help to find the problem?


Put the iMac back to cpu:3, and I'm off to bed. Hopefully it will be fixed before 02:00 UTC when the smp clients finish...

[ChristianVirtual]

Same here with my virtual CPU:3 on VMware ... Idle ...with the same error messages from assignment servers

[jfisher@lightfleet]

Apparently the bottom-of-log messages only report the AS (assignment server) IPs, which isn't enough for admins to work with. They want to know what WS (work server) a slot last got work from (and maybe more). It might be helpful if a future version of the client would bottom-line the info that an admin wants to see. Then we who are mostly clueless would tend to post that info instead of the AS warnings.

[billford]

They want to know what WS (work server) a slot last got work from (and maybe more).

Then why hasn't someone come on here and requested that information?

I thought the new AS software was supposed to provide better logging information to avoid all these problems?