Troubleshooting Server Connectivity Issues (Do This First)

Moderators: Site Moderators, PandeGroup

Troubleshooting Server Connectivity Issues (Do This First)

Postby PantherX » Sun Oct 10, 2010 5:49 am

A F@h Server can be classified into one of three categories:
1) AS -> Assignment Server
2) WS -> Work Server
3) CS -> Collection Server


Whenever there is a communication error, an IP address will be mentioned in the FAHlog. However, not all IP addresses are mentioned by default in the FAHlog:
IP address of AS -> By default
IP address of WS -> Will only be specified in V6 if the -verbosity 9 flag is being used; always shown in V7.
IP address of CS -> Will only be specified in V6 if the -verbosity 9 flag is being used; always shown in V7.


Here is an extract of FAHlog which doesn't use -verbosity 9 flag:
Code: Select all
[17:08:53] Folding@home Core Shutdown: FINISHED_UNIT
[17:08:55] CoreStatus = 64 (100)
[17:08:55] Sending work to server
[17:08:55] Project: X (Run X, Clone X, Gen X)


[17:08:55] + Attempting to send results [October 1 17:08:55 UTC]
[17:12:43] + Results successfully sent
[17:12:43] Thank you for your contribution to Folding@Home.
[17:12:43] + Number of Units Completed: X


Now here is an extract of the FAHlog which uses -verbosity 9 flag:
Code: Select all
[23:41:09] Folding@home Core Shutdown: FINISHED_UNIT
[23:41:13] CoreStatus = 64 (100)
[23:41:13] Unit 6 finished with 99 percent of time to deadline remaining.
[23:41:13] Updated performance fraction: 0.968683
[23:41:13] Sending work to server
[23:41:13] Project: X (Run X, Clone X, Gen X)


[23:41:13] + Attempting to send results [October 8 23:41:13 UTC]
[23:41:13] - Reading file work/wuresults_06.dat from core
[23:41:13]   (Read 3098115 bytes from disk)
[23:41:13] Connecting to http://X.X.X.X:8080/
[00:02:20] Posted data.
[00:02:20] Initial: 0000; - Uploaded at ~2 kB/s
[00:02:20] - Averaged speed for that direction ~9 kB/s
[00:02:20] + Results successfully sent
[00:02:20] Thank you for your contribution to Folding@Home.
[00:02:20] + Number of Units Completed: X


Now here is an extract of the V7 log:
Code: Select all
04:22:28:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
04:22:29:WU00:FS00:Sending unit results: id:00 state:SEND error:OK project:X run:X clone:X gen:X core:0xa4 unit:X
04:22:29:WU00:FS00:Uploading 1.14MiB to 171.67.108.58
04:22:35:WU00:FS00:Upload 21.99%
04:22:41:WU00:FS00:Upload 43.97%
04:22:48:WU00:FS00:Upload 65.96%
04:22:54:WU00:FS00:Upload 87.94%
04:22:58:WU00:FS00:Upload complete
04:22:58:WU00:FS00:Server responded WORK_ACK (400)


Hence it is highly recommended to use -verbosity 9 flag with all V6 F@h Clients so that troubleshooting it will be easy.
Last edited by PantherX on Wed Oct 13, 2010 4:19 am, edited 1 time in total.
User avatar
PantherX
Site Moderator
 
Posts: 6614
Joined: Wed Dec 23, 2009 9:33 am

Troubleshooting The Assignment Server (AS)

Postby PantherX » Sun Oct 10, 2010 5:50 am

  • Assignment Server (AS)
Before any downloading occurs, your F@h Client must be able to connect to the Assignment Server. A F@h Client which has been installed for the first time on a system, will request a User ID from the AS. Please note that the AS doesn't issue WUs. They keep track of the status of the Work Servers and assigns you to a WS that can give you an appropriate WU, depending on your F@h Clients configuration and available WUs. Failure to connect to the AS can will be indicated by these messages in the FAHlog:
1) [03:28:32] + Could not connect to Assignment Server
2) [03:28:40] + Could not connect to Assignment Server 2


When this happens, most of the time, there is an issue is on the F@h Donor's side. The following should be checked:
1) Internet Connection -> Make sure that you are online
2) Firewall -> Make sure that you have configured the Firewall to allow F@h Client to connect to the internet
3) Router -> Make sure that it configured correctly. F@h Clients use port 80 and 8080 only.
4) Installed Software(s) -> Make sure that any internet monitoring software is configured to allow the F@h Client to connect to the internet. For example, Anti-Spyware software which may be included with your Firewall, Anti-Virus, or other Anti-Malware software


Once you have checked the above, please try opening these sites in your browser according to the F@h Client that you installed:
1) F@h CPU Clients (Classic, SMP2) -> http://assign.stanford.edu:8080/
2) F@h CPU Clients (Classic, SMP2) -> http://assign2.stanford.edu:80/
3) F@h GPU Clients (GPU2, GPU3) -> http://assign-gpu.stanford.edu/
4) F@h GPU Clients (GPU2, GPU3) -> http://assign-gpu.stanford.edu:8080/
5) F@h GPU Clients (V7.4.0+) -> http://assign-gpu2.stanford.edu:8080/
6) F@h Folding App -> http://assign5.stanford.edu/
7) F@h Folding App -> http://assign5.stanford.edu:8080/


Depending on the browser, the correct/expected response should be:
1) An OK message being displayed (For old WS).
2) A blank page is displayed (with new WS Code).
3) A prompt to download a file (you can ignore it).
4) A 401 HTTP UNAUTHORIZED / message being displayed

Last Updated: 04 July 2014
Last edited by PantherX on Wed Oct 13, 2010 3:33 am, edited 1 time in total.
User avatar
PantherX
Site Moderator
 
Posts: 6614
Joined: Wed Dec 23, 2009 9:33 am

Troubleshooting The Work Server (WS)

Postby PantherX » Sun Oct 10, 2010 5:50 am

  • Work Server (AS)
Once the F@h Client sends a WU request to the Assignment Server, it forwards the request to the appropriate Work Server. The WS handles:
1) F@h Client's request to download a WU
2) F@h Client's request to upload the wuresult

Before starting a new thread on the troublesome Server, you should:
1) First check the Server Status page for that Server. If you find that:
A) The NET LOAD is >200 for that Server, it is possible that it is heavily loaded and may issue Error 503 in the FAHlog. (Please read Note 1)
B) The CPU LOAD is >3 for that Server, it is heavily loaded and may issue Error 503 in the FAHlog. (Please read Note 2)

Note 1: Please remember that the NET LOAD Value varies from Server to Server. If Server A has a NET LOAD of 300, it may function without any problem while Server B may have issues if the NET LOAD is 250.

Note 2: Occasionally the CPU LOAD can be high without causing any problems.


2) Search this forum for the a recent thread containing the IP address in its title. If found, please post the relevant section of the FAHlog. In this case, the Pande Group may already be working on the issue (during daylight or evening hours) so please be patient during this time.

3) To ensure that the Server is problematic, please open the IP address in your browser and depending on the browser, the correct/expected response should be:
A) An OK message being displayed.
B) A prompt to download a file (you can ignore it)
For example, in the FAHlog, you see this:
[01:45:27] Connecting to http://171.67.108.21:8080/
[01:45:27] - Couldn't send HTTP request to server
[01:45:27] (Got status X)
[01:45:27] + Could not connect to Work Server
[01:45:27] - Attempt #1 to get work failed, and no other work to do.Waiting before retry.
So you should attempt to open the IP address in your browser. In this case, it is http://171.67.108.21:8080/ and if it fails, try http://171.67.108.21:80/ to confirm your issue.


Server Listing
To view all F@h Server status, please visit this page. Do note that it can be really tiresome to hunt through them so there are some special sites that generates Server Status reports according to F@h Client type only. The sites are:
1) All F@h Classic Clients -> http://fah-web.stanford.edu/localinfo/c ... assic.html
2) All F@h SMP2 Clients -> http://fah-web.stanford.edu/localinfo/contact.SMP.html
3) All F@h GPU Clients -> http://fah-web.stanford.edu/localinfo/contact.GPU.html
4) All F@h PS3 Clients -> http://fah-web.stanford.edu/localinfo/contact.PS3.html

[Note: (12/7/12) The above links are no longer actively updated, the sorting ability of the current server status page replaces them]

Downloading Problem
The AS should automatically be redirected you to another Server shortly. Make sure that you have NOT specified WUs Without Deadlines. If you're being redirected to 0.0.0.0 or you get a message saying Failed to get assignment from x.x.x.x, that means that all of the Servers which have WUs for your configuration have run out of work to assign. Modifying the selection criteria may help, but you may also need to wait until more WUs are available. The selection criteria that you can modify are:
1) The size of WUs you the Client is configured to accept:
A) Small
B) Normal
C) Big (The Big setting will also accept either Normal or Small)

2) Using -advmethods flag (You will be assigned WUs that are in late-beta stage which increases your probability of EUEing. If there are none of those, you'll get the standard F@h WUs which are generally more stable, but the AS may have fewer choices.)

Note 3: If you're having trouble getting GPU assignments, be sure to tell us which type of GPU you have:
1) ATI
2) G80 (Nvidia, Non-Fermi)
3) Fermi



Uploading Problem
If the WS is not functioning as expected, the F@h Client is designed to deal with this:
1) After it finishes the WU, it will attempt to upload the result to the WS
2) If the initial attempts to upload the result to the WS fails, it will attempt to upload to the Collection Server (details below)
3) If it still fails to upload the result, it will keep it in the work folder and will request a new WU to fold. The autosend feature will attempt to upload the result periodically.

If the upload attempts have continued to fail for some time, please add the -verbosity 9 flag to your F@h Client and post the section of FAHlog that shows the errors. It is preferred to show the initial connection when the WU finished processing, a sample FAHlog:
Code: Select all
[11:10:25] Folding@home Core Shutdown: FINISHED_UNIT
[11:10:29] CoreStatus = 64 (100)
[11:10:29] Unit 8 finished with 99 percent of time to deadline remaining.
[11:10:29] Updated performance fraction: 0.975414
[11:10:29] Sending work to server
[11:10:29] Project: X (Run X, Clone X, Gen X)

[11:10:29] + Attempting to send results [October 11 11:10:29 UTC]
[11:10:29] - Reading file work/wuresults_08.dat from core
[11:10:29]   (Read 2781020 bytes from disk)
[11:10:29] Connecting to http://X.X.X.X:8080/
[11:25:29] - Couldn't send HTTP request to server
[11:25:29] + Could not connect to Work Server (results)
[11:25:29]     (X.X.X.X:8080)
[11:25:29] + Retrying using alternative port
[11:25:29] Connecting to http://X.X.X.X:80/
[11:36:51] - Couldn't send HTTP request to server
[11:36:51]   (Got status X)
[11:36:51] + Could not connect to Work Server (results)
[11:36:51]     (X.X.X.X:80)
[11:36:51] - Error: Could not transmit unit 08 (completed October 11) to work server.
[11:36:51] - 1 failed uploads of this unit.

[11:36:51] + Attempting to send results [October 11 11:36:51 UTC]
[11:36:51] - Reading file work/wuresults_08.dat from core
[11:36:51]   (Read 2781020 bytes from disk)
[11:36:51] Connecting to http://X.X.X.X:8080/
[11:51:54] - Couldn't send HTTP request to server
[11:51:54]   (Got status X)
[11:51:54] + Could not connect to Work Server (results)
[11:51:54]     (X.X.X.X:8080)
[11:51:54] + Retrying using alternative port
[11:51:54] Connecting to http://X.X.X.X:80/
[12:07:05] - Couldn't send HTTP request to server
[12:07:05] + Could not connect to Work Server (results)
[12:07:05]     (X.X.X.X:80)
[12:07:05] - Error: Could not transmit unit 08 (completed October 11) to work server.
[12:07:05] - 2 failed uploads of this unit.

[12:07:05] + Attempting to send results [October 11 12:07:05 UTC]
[12:07:05] - Reading file work/wuresults_08.dat from core
[12:07:05]   (Read 2781020 bytes from disk)
[12:07:05] Connecting to http://X.X.X.X:8080/
[12:11:02] - Couldn't send HTTP request to server
[12:11:02] + Could not connect to Work Server (results)
[12:11:02]     (X.X.X.X:8080)
[12:11:02] + Retrying using alternative port
[12:11:02] Connecting to http://X.X.X.X:80/
[12:11:07] - Couldn't send HTTP request to server
[12:11:07]   (Got status X)
[12:11:07] + Could not connect to Work Server (results)
[12:11:07]     (X.X.X.X:80)
[12:11:07]   Could not transmit unit 08 to Collection server; keeping in queue.
[12:11:07] + Sent 0 of 1 completed units to the server

Note 4: Please remember that if the WU was an SMP one, your bonus points will decrease accordingly until the result has been successfully uploaded to the WS.

Note 5: If the server appears to be functioning normally and you are using a Windows F@h Client v4/v5, be sure that Use IE Settings is set to NO.
Last edited by PantherX on Wed Oct 13, 2010 4:22 am, edited 1 time in total.
User avatar
PantherX
Site Moderator
 
Posts: 6614
Joined: Wed Dec 23, 2009 9:33 am

Details Of Collection Server (CS)

Postby PantherX » Sun Oct 10, 2010 5:54 am

This feature isn't fully implemented hence you should not report the Collection Server. Instead, you have to report the Work Server which is mentioned before the CS.The following CS should not be reported:
http://171.65.103.100
http://171.67.108.17
http://171.67.108.25
http://171.67.108.26

  • Collection Server (CS)
Each Work Server is assigned a Collection Server. The role of the CS is to receive wuresults if the Work Server is overloaded, down, offline, etc. The CS can get overloaded quite easily and nothing can be done about that. Of course, if it's not operating properly, someone should report it. However, the main focus of the error report should be on getting the WS repaired so it can accept the uploads thus nobody has to revert to the CS.

Here is an extract of the FAHlog showing a successful upload to the CS:
Code: Select all
[08:49:00] + Attempting to send results [October 2 08:49:00 UTC]
[08:49:00] - Reading file work/wuresults_09.dat from core
[08:49:00]   (Read 2772549 bytes from disk)
[08:49:00] Connecting to http://X.X.X.X:8080/
[08:49:02] - Couldn't send HTTP request to server
[08:49:02] + Could not connect to Work Server (results)
[08:49:02]     (X.X.X.X:8080)
[08:49:02] + Retrying using alternative port
[08:49:02] Connecting to http://X.X.X.X:80/
[08:49:08] - Couldn't send HTTP request to server
[08:49:08] + Could not connect to Work Server (results)
[08:49:08]     (X.X.X.X:80)
[08:49:08] - Error: Could not transmit unit 09 (completed October 2) to work server.
[08:49:08] - 2 failed uploads of this unit.

[08:49:08] + Attempting to send results [October 2 08:49:08 UTC]
[08:49:08] - Reading file work/wuresults_09.dat from core
[08:49:08]   (Read 2772549 bytes from disk)
[08:49:08] Connecting to http://X.X.X.X:8080/
[08:49:48] Posted data.
[08:49:48] Initial: 0000; - Uploaded at ~67 kB/s
[08:49:48] - Averaged speed for that direction ~154 kB/s
[08:49:48] + Results successfully sent
[08:49:48] Thank you for your contribution to Folding@Home.
[08:49:48] + Number of Units Completed: X

[08:49:48]   Successfully sent unit 09 to Collection server.
[08:49:49] + Sent 1 of 1 completed units to the server


This is an extract of the FAHlog showing an unsuccessful upload to the CS:
Code: Select all
[15:56:43] + Attempting to send results [October 2 15:56:43 UTC]
[15:56:43] - Reading file work/wuresults_06.dat from core
[15:56:43]   (Read 2774411 bytes from disk)
[15:56:43] Connecting to http://X.X.X.X:8080/
[15:56:44] - Couldn't send HTTP request to server
[15:56:44] + Could not connect to Work Server (results)
[15:56:44]     (X.X.X.X:8080)
[15:56:44] + Retrying using alternative port
[15:56:44] Connecting to http://X.X.X.X:80/
[15:56:45] - Couldn't send HTTP request to server
[15:56:45]   (Got status X)
[15:56:45] + Could not connect to Work Server (results)
[15:56:45]     (X.X.X.X:80)
[15:56:45] - Error: Could not transmit unit 06 (completed October 2) to work server.
[15:56:45] - 3 failed uploads of this unit.


[15:56:45] + Attempting to send results [October 2 15:56:45 UTC]
[15:56:45] - Reading file work/wuresults_06.dat from core
[15:56:45]   (Read 2774411 bytes from disk)
[15:56:45] Connecting to http://X.X.X.X:8080/
[15:57:04] Posted data.
[15:57:04] Initial: 0000; - Uploaded at ~142 kB/s
[15:57:04] - Averaged speed for that direction ~79 kB/s
[15:57:04] - Server does not have record of this unit. Will try again later.
[15:57:04]   Could not transmit unit 06 to Collection server; keeping in queue.
[15:57:04] + Sent 0 of 1 completed units to the server

If you're getting the message "Server does not have record of this unit" don't worry about it. Each WU ultimately has to be sent to its WS. If that WS is busy, down, etc the F@h Client will attempt to upload the wuresult to the corresponding CS. Normally the CS has a list of all/most of the WUs that it can accept. If that list was incomplete when the WS became unavailable, the remaining WUs will need to wait for the WS to be on-line again. Normally that is only a very small percentage of the outstanding WUs.
Last edited by PantherX on Wed Oct 13, 2010 4:21 am, edited 1 time in total.
User avatar
PantherX
Site Moderator
 
Posts: 6614
Joined: Wed Dec 23, 2009 9:33 am

Descriptions Of Columns In Server Status Page

Postby PantherX » Sun Oct 10, 2010 5:58 am

When viewing the Server Status Page, you will come across some column heading which you may not know. In order to make it easier for donors to understand, below you will find a brief description of the most columns and they have been colour coded for easy reference. You may want to hover over the icon Image to get additional details on the Server Status Page:
Green -> For New F@h Donors
Black -> For Expert F@h Donors
Gery -> Purpose is Guessed/Unknown/Unconfirmed


1) Server IP -> IP address of F@H server

2) Name -> Name of the F@H Server

3) Who -> Pande Group Member responsible for that Server

4) Client -> The job description of the Server which can be:
A) AS (classic) -> It is an Assignment Server for Classic Clients (Port 8080)
B) AS (classic p80) -> It is an Assignment Server for Classic Clients (Port 80)
C) AS (GPU) -> It is the Primary Assignment Server for GPU Clients (ATI/AMD/Nvidia)
D) AS (GPU2) -> It is the Backup Assignment Server for GPU Clients (ATI/AMD/Nvidia)
E) AS (PS3) -> It is an Assignment Server for PS3
F) AS (PS3 B) -> It is an Assignment Server for PS3
G) CS X -> Collection Server X
H) classic -> Work Server that distributes Classic WUs
I) SMP -> Work Server that distributes SMP2 WUs
J) PS3 -> Work Server that distributes PS3 WUs
K) GPU -> Work Server that distributes GPU WUs (ATI/AMD/Nvidia)

5) Status -> Current state of the server which is:
A) Full -> Server is in full operation. Everything is okay
B) Accept -> Server is only accepting WUs, not assigning them
C) Fail -> Server can't be reached by the stats system
D) Standby -> Server is not serving F@h WUs
E) No Response -> Server isn't responding


6) Connect -> Current result of the server status code to try to connect to the F@H server which are:
A) Accepting -> Server handles incoming connections as expected
B) Reject -> Server is online but isn't handling incoming connections as expected (yellow colour)n (unplanned)
C) Not Accept -> Server is on standby and will not accept incoming connections (probably planned but temporary)
D) Offline -> Server is offline for planned maintenance


7) CPU Load -> Load on the server (number of processes). If the CPU LOAD is >3 for that Server, it is heavily loaded and may issue Error 503 in the FAHlog. (Please read Note 2 in Troubleshooting The Work Server (WS)) LOAD <1 is common. For LOAD>3, the server might be analyzing the previously returned results and if the server owner has set them to nice (background-priority), that's not a problem. It can also mean something is really wrong.

8) Net Load -> Number of current connections the server is handling. The NET LOAD is >200 for that Server, it is possible that it is heavily loaded and may issue Error 503 in the FAHlog. (Please read Note 1 in Troubleshooting The Work Server (WS))

9) Fin Wait ->

10) DL -> Days left on at least one of the partitions

11) Diff Time ->

12) WUs Avail -> Number of WUs that are available for the server to give out (this number can increase or decrease, depending on the current rates of assignments and finished WU returns).

13) WUs To Go ->

14) WUs Wait -> Shows how many WUs have been assigned by the data server. This includes WUs which are beyond the deadline, so it may give an exaggerated count.

15) % Ass -> % of last 500 assignments on port 8080

16) % Ass 80 -> % of last 500 assignments on port 80

17) % Ass G -> % of last 500 GPU assignments (on port 8080)

18) % Ass G2 -> % of last 500 GPU assignments by GPU AS2

19) % Ass PS -> % of last 500 PS3 assignments since the last update (on port 8080)

20) WUs Rcv -> Shows how many WUs have been received since the last time the returned WUs results were updated into the stats. It is the relative number of WUs being received on the each server (if all is fine) or which servers are not being inputted into the stats database if there is some problem.

21) WUs E -> Number of WUs that have expired (crossed the Final Deadline) and will be reissued

22) In Stats ->

23) CS -> See VijayPande's post below.

24) NMJ ->

25) 80 -> 0 means that 80 isn't being redirected. 1 means that 80 is being redirected. Red cell colour means that it is unexpected.

26) OS_Weight_Program_Port -> The breakdown is as follows:
- OS -> Operating Systems that this server can assign work to
A) W -> Windows
B) L -> Linux
C) X -> OSX
- Weight -> "Weight" of the server (how frequently the AS will assign clients to that particular server). Higher numbers means more frequently. Zero means never.
- Program -> Type of projects currently handled by the server.
A) F -> F@H projects for everyone
B) A -> Advmethods projects. Possible to move to F, but might be stopped
C) B -> Beta projects, for testing purposes only. Possible to move to A, then to F, but might be stopped.
D) I -> Internal projects, for testing purposes only. Possible to move to B, then to A, then to F but they might be only experimental WU that will never be released.
- Port -> Tells if the Server is on AS port 80 or 8080

27) Others
- GB Tot -> Total disk space
- GB AV -> Available disk space
- T ->
- St ->
- Min Ver -> Minimum client version to get work from that server
- Min Packet -> Minimum Packet size of WUs for deciding small/medium/big assignments
- Max Packet -> Maximum Packet size of WUs for deciding small/medium/big assignments
- Memory -> Minimum memory required, as reported by the F@h client, to be eligible to get work from that server
- SMP Cores -> Minimum number of cores for SMP WU assignment
- Min SMP ->
-GP Type -> Indicates the WUs available for a particular GPU Type. It is broken into:
A) 1 -> WUs for ATI/AMD GPUs
B) 2 -> WUs for Nvidia GPUs


VijayPande wrote:We've been sprucing up serversstat by adding new columns, deleting old ones which we didn't use, and speeding up the update code (by about 2x). The main addition has been better collection server (CS) info. In the past, FAH had just one CS, so the question was "is this server set up correctly for *the* CS", but now with multiple-CS's, there are 2 questions: "Does the CS know about this server and is this server set up for the right CS." The serverstat has been updated to give that info. You'll see there are a few issues and my team is working on them now that they're more obvious (ETA: Tuesday). We're also working to setup a new CS for SMP WU's only.


The following two rows can be of some use:
Totals -> Any Status of the Server is acceptable
Totals (full machines) -> The Status of the Server should be Full


I wish to thanks the following users who have contributed in this thread: (alphabetically)
7im
anandhanju
bruce
folding_hoomer
toTOW
uncle fuzzy


Last Updated: 8 February 2014
User avatar
PantherX
Site Moderator
 
Posts: 6614
Joined: Wed Dec 23, 2009 9:33 am

Re: Troubleshooting Server Connectivity Issues (Do This Firs

Postby bruce » Fri Mar 27, 2015 1:44 am

If you get a message "Empty work server assignment" from an assignment server: DO NOT report a problem with that server. It is searching for a Work Server that has WUs which can be assigned to you and finding that there are none. The problem can only be solved by figuring out which Work Server(s) should be supplying WUs.

Look back in your log and find the IP addresses to/from which you have previously received/returned WUs. Examine the status of those servers and (if necessary) report the problem associated with THAT server. Normally, there are several WSs which have WUs for you and the AS chooses one of them. When the last server in that list crashes or runs out of WUs, the AS will report "Empty work server assignment."
bruce
 
Posts: 21272
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.


Return to Issues with a specific server

Who is online

Users browsing this forum: No registered users and 1 guest

cron