Failed to get assignment from '171.67.108.200'...

Moderators: Site Moderators, FAHC Science Team

Post Reply
TheBlademaster01
Posts: 39
Joined: Sat Jun 05, 2010 11:10 am

Failed to get assignment from '171.67.108.200'...

Post by TheBlademaster01 »

Hi, I'm trying to get some folding done on my system but I get empty WS reports on the CPU side. My GPU seems to be failing units early on as well.

Here's the log

Code: Select all

*
22:58:35:        CPU: Genuine Intel(R) CPU @ 2.30GHz
22:58:35:     CPU ID: GenuineIntel Family 6 Model 45 Stepping 2
22:58:35:       CPUs: 32
22:58:35:     Memory: 27.44GiB
22:58:35:Free Memory: 23.66GiB
22:58:35:    Threads: POSIX_THREADS
22:58:35: OS Version: 3.2
22:58:35:Has Battery: false
22:58:35: On Battery: false
22:58:35: UTC Offset: 2
22:58:35:        PID: 10060
22:58:35:        CWD: /home/tbm01/fahv7/v7
22:58:35:         OS: Linux 3.2.0-51-generic x86_64
22:58:35:    OS Arch: AMD64
22:58:35:       GPUs: 1
22:58:35:      GPU 0: NVIDIA:3 GK110 [GeForce GTX 780 Ti]
22:58:35:       CUDA: 3.5
22:58:35:CUDA Driver: 6000
22:58:35:***********************************************************************
22:58:35:<config>
22:58:35:  <!-- Folding Slot Configuration -->
22:58:35:  <client-type v='advanced'/>
22:58:35:  <cpus v='30'/>
22:58:35:  <max-packet-size v='large'/>
22:58:35:
22:58:35:  <!-- Slot Control -->
22:58:35:  <power v='full'/>
22:58:35:
22:58:35:  <!-- User Information -->
22:58:35:  <passkey v='********************************'/>
22:58:35:  <team v='37726'/>
22:58:35:  <user v='TheBlademaster01'/>
22:58:35:
22:58:35:  <!-- Folding Slots -->
22:58:35:  <slot id='0' type='CPU'/>
22:58:35:  <slot id='1' type='GPU'/>
22:58:35:</config>
22:58:35:Trying to access database...
22:58:35:Successfully acquired database lock
22:58:35:FS00:Set client configured
22:58:35:Enabled folding slot 00: READY cpu:30
22:58:35:Enabled folding slot 01: READY gpu:0:GK110 [GeForce GTX 780 Ti]
22:58:35:WU00:FS00:Connecting to 171.67.108.200:8080
22:58:35:WU01:FS01:Connecting to 171.67.108.200:8080
22:58:36:WU00:FS00:Connecting to 171.67.108.200:8080
22:58:37:WU01:FS01:Connecting to 171.67.108.201:80
22:58:38:WARNING:WU00:FS00:Failed to get assignment from '171.67.108.200:8080': Empty work server assignment
22:58:38:WU00:FS00:Connecting to 171.64.65.121:80
22:58:38:WU01:FS01:Assigned to work server 171.64.65.93
22:58:38:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.93
22:58:38:WU01:FS01:Connecting to 171.64.65.93:8080
22:58:39:WARNING:WU00:FS00:Failed to get assignment from '171.64.65.121:80': Empty work server assignment
22:58:39:ERROR:WU00:FS00:Exception: Could not get an assignment
22:58:39:WU00:FS00:Connecting to 171.67.108.200:8080
22:58:39:WU01:FS01:Downloading 3.12MiB
22:58:40:WARNING:WU00:FS00:Failed to get assignment from '171.67.108.200:8080': Empty work server assignment
22:58:40:WU00:FS00:Connecting to 171.64.65.121:80
22:58:41:WARNING:WU00:FS00:Failed to get assignment from '171.64.65.121:80': Empty work server assignment
22:58:41:ERROR:WU00:FS00:Exception: Could not get an assignment
22:58:45:WU01:FS01:Download 70.12%
22:58:48:WU01:FS01:Download complete
22:58:48:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9101 run:283 clone:0 gen:11 core:0x17 unit:0x0000000c0a3b1e81533f4a361cc3c99e
22:58:48:WU01:FS01:Starting
22:58:48:WU01:FS01:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 01 -suffix 01 -version 704 -lifeline 10060 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
22:58:48:WU01:FS01:Started FahCore on PID 10189
22:58:48:WU01:FS01:Core PID:10193
22:58:48:WU01:FS01:FahCore 0x17 started
22:58:48:WU01:FS01:0x17:*********************** Log Started 2014-04-20T22:58:48Z ***********************
22:58:48:WU01:FS01:0x17:Project: 9101 (Run 283, Clone 0, Gen 11)
22:58:48:WU01:FS01:0x17:Unit: 0x0000000c0a3b1e81533f4a361cc3c99e
22:58:48:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
22:58:48:WU01:FS01:0x17:Machine: 1
22:58:48:WU01:FS01:0x17:Reading tar file state.xml
22:58:48:WU01:FS01:0x17:Reading tar file system.xml
22:58:49:WU01:FS01:0x17:Reading tar file integrator.xml
22:58:49:WU01:FS01:0x17:Reading tar file core.xml
22:58:49:WU01:FS01:0x17:Digital signatures verified
Top
bollix47
Posts: 2942
Joined: Sun Dec 02, 2007 5:04 am
Location: Canada

Re: Failed to get assignment from '171.67.108.200'...

Post by bollix47 »

Part of your empty work server problem may be this line:

22:58:35: <max-packet-size v='large'/>

The value should be 'big' if that's what you intended. I know of no "large" value. If you continue to get empty work server after changing the value to big then I would remove that option completely.

Do you have a log showing a core 17 failure? There is always a delay before core_17 starts showing progress but the delay should be fairly short for a 780ti ... mine is under a minute but my cpu is faster so your delay may not be quite as short.

Also, regarding the gpu, are you using your distro's proprietary drivers? Which version?
Image
Top
davidcoton
Posts: 1102
Joined: Wed Nov 05, 2008 3:19 pm
Location: Cambridge, UK

Re: Failed to get assignment from '171.67.108.200'...

Post by davidcoton »

What GPU driver are you using? See here (and possibly more of that thread) for some thoughts related to another 780Ti on Linux.

David
Image
Top
TheBlademaster01
Posts: 39
Joined: Sat Jun 05, 2010 11:10 am

Re: Failed to get assignment from '171.67.108.200'...

Post by TheBlademaster01 »

Okay, thanks all. I replaced large with big and the CPU client downloaded Project 8101 (R8, C6, G323). I also found that while booting into my OS, X wouldn't launch, so I purged the nVidia 331.67 drivers I installed yesterday and installed 331.49 that I ran without issues on my other Linux install.

I'll report back if the GPU manages to fold without errors.

Many thanks for the helpful feedback ;)
Top
TheBlademaster01
Posts: 39
Joined: Sat Jun 05, 2010 11:10 am

Re: Failed to get assignment from '171.67.108.200'...

Post by TheBlademaster01 »

Okay, well it seems the GPU still fails units

Code: Select all

tbm01@Pernod:~/fahv7/v7$ ./FAHClient 
10:40:21:INFO(1):Read GPUs.txt
10:40:21:************************* Folding@home Client *************************
10:40:21:    Website: http://folding.stanford.edu/
10:40:21:  Copyright: (c) 2009-2014 Stanford University
10:40:21:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
10:40:21:       Args: 
10:40:21:     Config: /home/tbm01/fahv7/v7/config.xml
10:40:21:******************************** Build ********************************
10:40:21:    Version: 7.4.4
10:40:21:       Date: Mar 4 2014
10:40:21:       Time: 12:02:38
10:40:21:    SVN Rev: 4130
10:40:21:     Branch: fah/trunk/client
10:40:21:   Compiler: GNU 4.4.7
10:40:21:    Options: -std=gnu++98 -O3 -funroll-loops -mfpmath=sse -ffast-math
10:40:21:             -fno-unsafe-math-optimizations -msse2
10:40:21:   Platform: linux2 3.2.0-1-amd64
10:40:21:       Bits: 64
10:40:21:       Mode: Release
10:40:21:******************************* System ********************************
10:40:21:        CPU: Genuine Intel(R) CPU @ 2.30GHz
10:40:21:     CPU ID: GenuineIntel Family 6 Model 45 Stepping 2
10:40:21:       CPUs: 32
10:40:21:     Memory: 27.44GiB
10:40:21:Free Memory: 25.92GiB
10:40:21:    Threads: POSIX_THREADS
10:40:21: OS Version: 3.2
10:40:21:Has Battery: false
10:40:21: On Battery: false
10:40:21: UTC Offset: 2
10:40:21:        PID: 4953
10:40:21:        CWD: /home/tbm01/fahv7/v7
10:40:21:         OS: Linux 3.2.0-51-generic x86_64
10:40:21:    OS Arch: AMD64
10:40:21:       GPUs: 1
10:40:21:      GPU 0: NVIDIA:3 GK110 [GeForce GTX 780 Ti]
10:40:21:       CUDA: 3.5
10:40:21:CUDA Driver: 6000
10:40:21:***********************************************************************
10:40:21:<config>
10:40:21:  <!-- Folding Core -->
10:40:21:  <checkpoint v='3'/>
10:40:21:
10:40:21:  <!-- Folding Slot Configuration -->
10:40:21:  <client-type v='bigbeta'/>
10:40:21:  <cpus v='30'/>
10:40:21:  <max-packet-size v='big'/>
10:40:21:
10:40:21:  <!-- Slot Control -->
10:40:21:  <power v='full'/>
10:40:21:
10:40:21:  <!-- User Information -->
10:40:21:  <passkey v='********************************'/>
10:40:21:  <team v='37726'/>
10:40:21:  <user v='TheBlademaster01'/>
10:40:21:
10:40:21:  <!-- Folding Slots -->
10:40:21:  <slot id='0' type='CPU'/>
10:40:21:  <slot id='1' type='GPU'/>
10:40:21:</config>
10:40:21:Trying to access database...
10:40:21:Successfully acquired database lock
10:40:21:Enabled folding slot 00: READY cpu:30
10:40:21:Enabled folding slot 01: READY gpu:0:GK110 [GeForce GTX 780 Ti]
10:40:21:WU00:FS00:Starting
10:40:21:WU00:FS00:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/beta/Core_a5.fah/FahCore_a5 -dir 00 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -np 30
10:40:21:WU00:FS00:Started FahCore on PID 4962
10:40:21:WU00:FS00:Core PID:4966
10:40:21:WU00:FS00:FahCore 0xa5 started
10:40:21:WU01:FS01:Starting
10:40:21:WU01:FS01:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 01 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -gpu 0 -gpu-vendor nvidia
10:40:21:WU01:FS01:Started FahCore on PID 4967
10:40:21:WU01:FS01:Core PID:4971
10:40:21:WU01:FS01:FahCore 0x17 started
10:40:22:WU00:FS00:0xa5:
10:40:22:WU00:FS00:0xa5:*------------------------------*
10:40:22:WU00:FS00:0xa5:Folding@Home Gromacs SMP Core
10:40:22:WU00:FS00:0xa5:Version 2.27 (Thu Feb 10 09:46:40 PST 2011)
10:40:22:WU00:FS00:0xa5:
10:40:22:WU00:FS00:0xa5:Preparing to commence simulation
10:40:22:WU00:FS00:0xa5:- Looking at optimizations...
10:40:22:WU00:FS00:0xa5:- Files status OK
10:40:22:WU01:FS01:0x17:*********************** Log Started 2014-04-21T10:40:21Z ***********************
10:40:22:WU01:FS01:0x17:Project: 9408 (Run 706, Clone 0, Gen 0)
10:40:22:WU01:FS01:0x17:Unit: 0x000000000a3b1e5c5342e1db98401a99
10:40:22:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
10:40:22:WU01:FS01:0x17:Machine: 1
10:40:22:WU01:FS01:0x17:Digital signatures verified
10:40:22:WU01:FS01:0x17:  Found a checkpoint file
10:40:24:WU00:FS00:0xa5:- Expanded 30304776 -> 33158020 (decompressed 109.4 percent)
10:40:24:WU00:FS00:0xa5:Called DecompressByteArray: compressed_data_size=30304776 data_size=33158020, decompressed_data_size=33158020 diff=0
10:40:25:WU00:FS00:0xa5:- Digital signature verified
10:40:25:WU00:FS00:0xa5:
10:40:25:WU00:FS00:0xa5:Project: 8101 (Run 8, Clone 6, Gen 323)
10:40:25:WU00:FS00:0xa5:
10:40:25:WU00:FS00:0xa5:Assembly optimizations on if available.
10:40:25:WU00:FS00:0xa5:Entering M.D.
10:40:31:WU00:FS00:0xa5:Using Gromacs checkpoints
10:40:33:WU00:FS00:0xa5:Mapping NT from 30 to 30 
Reading file ./work/00/wudata_01.tpr, VERSION 4.5.5-dev-20120903-d64b9e3 (single precision)
Reading file ./work/00/wudata_01.tpr, VERSION 4.5.5-dev-20120903-d64b9e3 (single precision)
10:40:48:WU00:FS00:0xa5:Resuming from checkpoint
10:41:00:WU00:FS00:0xa5:Verified 00/wudata_01.log
10:41:00:WU00:FS00:0xa5:Verified 00/wudata_01.trr
10:41:01:WU00:FS00:0xa5:Verified 00/wudata_01.xtc
10:41:01:WU00:FS00:0xa5:Verified 00/wudata_01.edr
10:41:01:WU00:FS00:0xa5:Completed 1715 out of 250000 steps  (0%)
10:43:02:WU01:FS01:0x17:Completed 200000 out of 5000000 steps (4%)
10:48:02:WU01:FS01:0x17:Completed 250000 out of 5000000 steps (5%)
10:48:14:WU01:FS01:0x17:ERROR:Guru Meditation #dec55273fb11015.105c7c2559009bbe (626688.830924) '01/01/positions.xtc'
10:48:14:WU01:FS01:0x17:WARNING:Unexpected exit() call
10:48:14:WU01:FS01:0x17:WARNING:Unexpected exit from science code
10:48:14:WU01:FS01:0x17:Saving result file logfile_01.txt
10:48:14:WU01:FS01:0x17:Saving result file checkpt.crc
10:48:14:WU01:FS01:0x17:Saving result file log.txt
10:48:14:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
10:48:15:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
10:48:15:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9408 run:706 clone:0 gen:0 core:0x17 unit:0x000000000a3b1e5c5342e1db98401a99
10:48:15:WU01:FS01:Uploading 3.91KiB to 171.64.65.56
10:48:15:WU01:FS01:Connecting to 171.64.65.56:8080
10:48:16:WU02:FS01:Connecting to 171.67.108.201:80
10:48:17:WU02:FS01:Assigned to work server 171.64.65.56
10:48:17:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.56
10:48:17:WU02:FS01:Connecting to 171.64.65.56:8080
10:48:21:WU00:FS00:0xa5:Completed 2500 out of 250000 steps  (1%)
10:48:24:WU01:FS01:Upload complete
10:48:24:WU01:FS01:Server responded WORK_ACK (400)
10:48:24:WU01:FS01:Cleaning up
10:48:24:WU02:FS01:Downloading 4.33MiB
10:48:30:WU02:FS01:Download 43.34%
10:48:35:WU02:FS01:Download complete
10:48:35:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:9408 run:724 clone:0 gen:0 core:0x17 unit:0x000000000a3b1e5c5342e272464cae5e
10:48:35:WU02:FS01:Starting
10:48:35:WU02:FS01:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 02 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -gpu 0 -gpu-vendor nvidia
10:48:35:WU02:FS01:Started FahCore on PID 8092
10:48:35:WU02:FS01:Core PID:8096
10:48:35:WU02:FS01:FahCore 0x17 started
10:48:36:WU02:FS01:0x17:*********************** Log Started 2014-04-21T10:48:35Z ***********************
10:48:36:WU02:FS01:0x17:Project: 9408 (Run 724, Clone 0, Gen 0)
10:48:36:WU02:FS01:0x17:Unit: 0x000000000a3b1e5c5342e272464cae5e
10:48:36:WU02:FS01:0x17:CPU: 0x00000000000000000000000000000000
10:48:36:WU02:FS01:0x17:Machine: 1
10:48:36:WU02:FS01:0x17:Reading tar file system.xml
10:48:36:WU02:FS01:0x17:Reading tar file integrator.xml
10:48:36:WU02:FS01:0x17:Reading tar file state.xml
10:48:37:WU02:FS01:0x17:Reading tar file core.xml
10:48:37:WU02:FS01:0x17:Digital signatures verified

It's a GByte Windforce at factory settings.
Top
bollix47
Posts: 2942
Joined: Sun Dec 02, 2007 5:04 am
Location: Canada

Re: Failed to get assignment from '171.67.108.200'...

Post by bollix47 »

The work unit that failed in your log may just be a bad one. There is one other return for this WU and it was a failure too.

You did receive a partial credit:

Hi TheBlademaster01 (team 37726),
Your WU (P9408 R706 C0 G0) was added to the stats database on 2014-04-21 04:04:06 for 650 points of credit.

The fact that the WU did progress a few percentages before it failed suggests your drivers are working so let's see what it does on a few more.

Have you checked your GPU temperatures? I try to keep mine no higher than the low 80's. If you need to adjust the fan settings there is a guide.
Image
Top
TheBlademaster01
Posts: 39
Joined: Sat Jun 05, 2010 11:10 am

Re: Failed to get assignment from '171.67.108.200'...

Post by TheBlademaster01 »

Hmm, yes seems like it. It has been folding just fine.

I don't really know the temperatures of the card, but I know that it has been 24/7 stable in folding before and some mining applications.

I'll check out your link later, thanks :)

Code: Select all

10:48:36:WU02:FS01:0x17:Project: 9408 (Run 724, Clone 0, Gen 0)
10:48:36:WU02:FS01:0x17:Unit: 0x000000000a3b1e5c5342e272464cae5e
10:48:36:WU02:FS01:0x17:CPU: 0x00000000000000000000000000000000
10:48:36:WU02:FS01:0x17:Machine: 1
10:48:36:WU02:FS01:0x17:Reading tar file system.xml
10:48:36:WU02:FS01:0x17:Reading tar file integrator.xml
10:48:36:WU02:FS01:0x17:Reading tar file state.xml
10:48:37:WU02:FS01:0x17:Reading tar file core.xml
10:48:37:WU02:FS01:0x17:Digital signatures verified
10:51:21:WU02:FS01:0x17:Completed 0 out of 5000000 steps (0%)
10:56:23:WU02:FS01:0x17:Completed 50000 out of 5000000 steps (1%)
11:01:25:WU02:FS01:0x17:Completed 100000 out of 5000000 steps (2%)
11:06:27:WU02:FS01:0x17:Completed 150000 out of 5000000 steps (3%)
11:11:29:WU02:FS01:0x17:Completed 200000 out of 5000000 steps (4%)
11:11:57:WU00:FS00:0xa5:Completed 5000 out of 250000 steps  (2%)
11:16:30:WU02:FS01:0x17:Completed 250000 out of 5000000 steps (5%)
11:21:32:WU02:FS01:0x17:Completed 300000 out of 5000000 steps (6%)
11:26:34:WU02:FS01:0x17:Completed 350000 out of 5000000 steps (7%)
11:31:36:WU02:FS01:0x17:Completed 400000 out of 5000000 steps (8%)
11:35:31:WU00:FS00:0xa5:Completed 7500 out of 250000 steps  (3%)
11:36:38:WU02:FS01:0x17:Completed 450000 out of 5000000 steps (9%)
11:41:40:WU02:FS01:0x17:Completed 500000 out of 5000000 steps (10%)
11:46:42:WU02:FS01:0x17:Completed 550000 out of 5000000 steps (11%)
11:51:43:WU02:FS01:0x17:Completed 600000 out of 5000000 steps (12%)
11:56:43:WU02:FS01:0x17:Completed 650000 out of 5000000 steps (13%)
11:59:09:WU00:FS00:0xa5:Completed 10000 out of 250000 steps  (4%)
12:01:44:WU02:FS01:0x17:Completed 700000 out of 5000000 steps (14%)
12:06:44:WU02:FS01:0x17:Completed 750000 out of 5000000 steps (15%)
12:11:44:WU02:FS01:0x17:Completed 800000 out of 5000000 steps (16%)
12:16:46:WU02:FS01:0x17:Completed 850000 out of 5000000 steps (17%)
Top
TheBlademaster01
Posts: 39
Joined: Sat Jun 05, 2010 11:10 am

Re: Failed to get assignment from '171.67.108.200'...

Post by TheBlademaster01 »

Sorry, but on the matter of the GPU. Is it normal that about all Project 9101 units fail on my GPU when I've completed the others?

Code: Select all

16:04:28:WU02:FS01:0x17:Project: 9408 (Run 754, Clone 0, Gen 2)
16:04:28:WU02:FS01:0x17:Unit: 0x000000020a3b1e5c5342e36e136b6fc2
16:04:28:WU02:FS01:0x17:CPU: 0x00000000000000000000000000000000
16:04:28:WU02:FS01:0x17:Machine: 1
16:04:28:WU02:FS01:0x17:Reading tar file state.xml
16:04:29:WU02:FS01:0x17:Reading tar file system.xml
16:04:29:WU02:FS01:0x17:Reading tar file integrator.xml
16:04:29:WU02:FS01:0x17:Reading tar file core.xml
16:04:29:WU02:FS01:0x17:Digital signatures verified
16:04:34:WU01:FS01:Upload 17.60%
16:04:42:WU01:FS01:Upload 36.21%
16:04:48:WU01:FS01:Upload 55.82%
16:04:55:WU01:FS01:Upload 72.92%
16:05:01:WU01:FS01:Upload 90.02%
16:05:17:WU01:FS01:Upload complete
16:05:18:WU01:FS01:Server responded WORK_ACK (400)
16:05:18:WU01:FS01:Final credit estimate, 47768.00 points
16:05:18:WU01:FS01:Cleaning up
16:07:13:WU02:FS01:0x17:Completed 0 out of 5000000 steps (0%)
16:12:11:WU02:FS01:0x17:Completed 50000 out of 5000000 steps (1%)
16:12:57:WU00:FS00:0xa5:Completed 190000 out of 250000 steps  (76%)
16:17:11:WU02:FS01:0x17:Completed 100000 out of 5000000 steps (2%)
16:22:13:WU02:FS01:0x17:Completed 150000 out of 5000000 steps (3%)
16:27:15:WU02:FS01:0x17:Completed 200000 out of 5000000 steps (4%)
16:32:17:WU02:FS01:0x17:Completed 250000 out of 5000000 steps (5%)
16:36:30:WU00:FS00:0xa5:Completed 192500 out of 250000 steps  (77%)
16:37:19:WU02:FS01:0x17:Completed 300000 out of 5000000 steps (6%)
16:42:20:WU02:FS01:0x17:Completed 350000 out of 5000000 steps (7%)
******************************* Date: 2014-04-22 *******************************
16:47:22:WU02:FS01:0x17:Completed 400000 out of 5000000 steps (8%)
16:52:23:WU02:FS01:0x17:Completed 450000 out of 5000000 steps (9%)
16:57:25:WU02:FS01:0x17:Completed 500000 out of 5000000 steps (10%)
16:59:58:WU00:FS00:0xa5:Completed 195000 out of 250000 steps  (78%)
17:02:27:WU02:FS01:0x17:Completed 550000 out of 5000000 steps (11%)
17:07:29:WU02:FS01:0x17:Completed 600000 out of 5000000 steps (12%)
17:12:30:WU02:FS01:0x17:Completed 650000 out of 5000000 steps (13%)
17:17:33:WU02:FS01:0x17:Completed 700000 out of 5000000 steps (14%)
17:22:35:WU02:FS01:0x17:Completed 750000 out of 5000000 steps (15%)
17:23:30:WU00:FS00:0xa5:Completed 197500 out of 250000 steps  (79%)
17:27:37:WU02:FS01:0x17:Completed 800000 out of 5000000 steps (16%)
17:32:40:WU02:FS01:0x17:Completed 850000 out of 5000000 steps (17%)
17:37:42:WU02:FS01:0x17:Completed 900000 out of 5000000 steps (18%)
17:42:44:WU02:FS01:0x17:Completed 950000 out of 5000000 steps (19%)
17:47:54:WU02:FS01:0x17:Completed 1000000 out of 5000000 steps (20%)
17:48:47:WU00:FS00:0xa5:Completed 200000 out of 250000 steps  (80%)
17:52:58:WU02:FS01:0x17:Completed 1050000 out of 5000000 steps (21%)
17:58:08:WU02:FS01:0x17:Completed 1100000 out of 5000000 steps (22%)
18:03:09:WU02:FS01:0x17:Completed 1150000 out of 5000000 steps (23%)
18:08:10:WU02:FS01:0x17:Completed 1200000 out of 5000000 steps (24%)
18:13:18:WU00:FS00:0xa5:Completed 202500 out of 250000 steps  (81%)
18:13:29:WU02:FS01:0x17:Completed 1250000 out of 5000000 steps (25%)
18:18:34:WU02:FS01:0x17:Completed 1300000 out of 5000000 steps (26%)
18:23:40:WU02:FS01:0x17:Completed 1350000 out of 5000000 steps (27%)
18:28:45:WU02:FS01:0x17:Completed 1400000 out of 5000000 steps (28%)
18:33:45:WU02:FS01:0x17:Completed 1450000 out of 5000000 steps (29%)
18:37:22:WU00:FS00:0xa5:Completed 205000 out of 250000 steps  (82%)
18:38:46:WU02:FS01:0x17:Completed 1500000 out of 5000000 steps (30%)
18:43:45:WU02:FS01:0x17:Completed 1550000 out of 5000000 steps (31%)
18:48:45:WU02:FS01:0x17:Completed 1600000 out of 5000000 steps (32%)
18:53:46:WU02:FS01:0x17:Completed 1650000 out of 5000000 steps (33%)
18:58:46:WU02:FS01:0x17:Completed 1700000 out of 5000000 steps (34%)
19:01:31:WU00:FS00:0xa5:Completed 207500 out of 250000 steps  (83%)
19:03:47:WU02:FS01:0x17:Completed 1750000 out of 5000000 steps (35%)
19:08:46:WU02:FS01:0x17:Completed 1800000 out of 5000000 steps (36%)
19:13:46:WU02:FS01:0x17:Completed 1850000 out of 5000000 steps (37%)
19:18:47:WU02:FS01:0x17:Completed 1900000 out of 5000000 steps (38%)
19:23:46:WU02:FS01:0x17:Completed 1950000 out of 5000000 steps (39%)
19:25:41:WU00:FS00:0xa5:Completed 210000 out of 250000 steps  (84%)
19:28:46:WU02:FS01:0x17:Completed 2000000 out of 5000000 steps (40%)
19:33:47:WU02:FS01:0x17:Completed 2050000 out of 5000000 steps (41%)
19:38:47:WU02:FS01:0x17:Completed 2100000 out of 5000000 steps (42%)
19:43:48:WU02:FS01:0x17:Completed 2150000 out of 5000000 steps (43%)
19:48:49:WU02:FS01:0x17:Completed 2200000 out of 5000000 steps (44%)
19:50:07:WU00:FS00:0xa5:Completed 212500 out of 250000 steps  (85%)
19:53:52:WU02:FS01:0x17:Completed 2250000 out of 5000000 steps (45%)
19:58:57:WU02:FS01:0x17:Completed 2300000 out of 5000000 steps (46%)
20:04:02:WU02:FS01:0x17:Completed 2350000 out of 5000000 steps (47%)
20:09:04:WU02:FS01:0x17:Completed 2400000 out of 5000000 steps (48%)
20:14:08:WU02:FS01:0x17:Completed 2450000 out of 5000000 steps (49%)
20:14:36:WU00:FS00:0xa5:Completed 215000 out of 250000 steps  (86%)
20:19:10:WU02:FS01:0x17:Completed 2500000 out of 5000000 steps (50%)
20:24:06:WU02:FS01:0x17:Completed 2550000 out of 5000000 steps (51%)
20:29:07:WU02:FS01:0x17:Completed 2600000 out of 5000000 steps (52%)
20:34:08:WU02:FS01:0x17:Completed 2650000 out of 5000000 steps (53%)
20:38:56:WU00:FS00:0xa5:Completed 217500 out of 250000 steps  (87%)
20:39:09:WU02:FS01:0x17:Completed 2700000 out of 5000000 steps (54%)
20:44:10:WU02:FS01:0x17:Completed 2750000 out of 5000000 steps (55%)
20:49:11:WU02:FS01:0x17:Completed 2800000 out of 5000000 steps (56%)
20:54:11:WU02:FS01:0x17:Completed 2850000 out of 5000000 steps (57%)
20:59:16:WU02:FS01:0x17:Completed 2900000 out of 5000000 steps (58%)
21:03:10:WU00:FS00:0xa5:Completed 220000 out of 250000 steps  (88%)
21:04:21:WU02:FS01:0x17:Completed 2950000 out of 5000000 steps (59%)
21:09:27:WU02:FS01:0x17:Completed 3000000 out of 5000000 steps (60%)
21:14:31:WU02:FS01:0x17:Completed 3050000 out of 5000000 steps (61%)
21:19:32:WU02:FS01:0x17:Completed 3100000 out of 5000000 steps (62%)
21:24:37:WU02:FS01:0x17:Completed 3150000 out of 5000000 steps (63%)
21:27:12:WU00:FS00:0xa5:Completed 222500 out of 250000 steps  (89%)
21:29:42:WU02:FS01:0x17:Completed 3200000 out of 5000000 steps (64%)
21:34:47:WU02:FS01:0x17:Completed 3250000 out of 5000000 steps (65%)
21:39:52:WU02:FS01:0x17:Completed 3300000 out of 5000000 steps (66%)
21:44:57:WU02:FS01:0x17:Completed 3350000 out of 5000000 steps (67%)
21:49:58:WU02:FS01:0x17:Completed 3400000 out of 5000000 steps (68%)
21:52:15:WU00:FS00:0xa5:Completed 225000 out of 250000 steps  (90%)
21:55:04:WU02:FS01:0x17:Completed 3450000 out of 5000000 steps (69%)
22:00:09:WU02:FS01:0x17:Completed 3500000 out of 5000000 steps (70%)
22:05:15:WU02:FS01:0x17:Completed 3550000 out of 5000000 steps (71%)
22:10:20:WU02:FS01:0x17:Completed 3600000 out of 5000000 steps (72%)
22:15:26:WU02:FS01:0x17:Completed 3650000 out of 5000000 steps (73%)
22:16:17:WU00:FS00:0xa5:Completed 227500 out of 250000 steps  (91%)
22:20:31:WU02:FS01:0x17:Completed 3700000 out of 5000000 steps (74%)
22:25:37:WU02:FS01:0x17:Completed 3750000 out of 5000000 steps (75%)
22:30:42:WU02:FS01:0x17:Completed 3800000 out of 5000000 steps (76%)
22:35:43:WU02:FS01:0x17:Completed 3850000 out of 5000000 steps (77%)
22:40:26:WU00:FS00:0xa5:Completed 230000 out of 250000 steps  (92%)
22:40:45:WU02:FS01:0x17:Completed 3900000 out of 5000000 steps (78%)
22:45:46:WU02:FS01:0x17:Completed 3950000 out of 5000000 steps (79%)
******************************* Date: 2014-04-22 *******************************
22:50:51:WU02:FS01:0x17:Completed 4000000 out of 5000000 steps (80%)
22:55:57:WU02:FS01:0x17:Completed 4050000 out of 5000000 steps (81%)
23:01:00:WU02:FS01:0x17:Completed 4100000 out of 5000000 steps (82%)
23:04:32:WU00:FS00:0xa5:Completed 232500 out of 250000 steps  (93%)
23:06:06:WU02:FS01:0x17:Completed 4150000 out of 5000000 steps (83%)
23:11:08:WU02:FS01:0x17:Completed 4200000 out of 5000000 steps (84%)
23:16:09:WU02:FS01:0x17:Completed 4250000 out of 5000000 steps (85%)
23:21:12:WU02:FS01:0x17:Completed 4300000 out of 5000000 steps (86%)
23:26:17:WU02:FS01:0x17:Completed 4350000 out of 5000000 steps (87%)
23:28:30:WU00:FS00:0xa5:Completed 235000 out of 250000 steps  (94%)
23:31:23:WU02:FS01:0x17:Completed 4400000 out of 5000000 steps (88%)
23:36:27:WU02:FS01:0x17:Completed 4450000 out of 5000000 steps (89%)
23:41:28:WU02:FS01:0x17:Completed 4500000 out of 5000000 steps (90%)
23:46:29:WU02:FS01:0x17:Completed 4550000 out of 5000000 steps (91%)
23:51:32:WU02:FS01:0x17:Completed 4600000 out of 5000000 steps (92%)
23:52:36:WU00:FS00:0xa5:Completed 237500 out of 250000 steps  (95%)
23:56:37:WU02:FS01:0x17:Completed 4650000 out of 5000000 steps (93%)
00:01:43:WU02:FS01:0x17:Completed 4700000 out of 5000000 steps (94%)
00:06:47:WU02:FS01:0x17:Completed 4750000 out of 5000000 steps (95%)
00:11:52:WU02:FS01:0x17:Completed 4800000 out of 5000000 steps (96%)
00:16:31:WU00:FS00:0xa5:Completed 240000 out of 250000 steps  (96%)
00:16:58:WU02:FS01:0x17:Completed 4850000 out of 5000000 steps (97%)
00:22:03:WU02:FS01:0x17:Completed 4900000 out of 5000000 steps (98%)
00:27:08:WU02:FS01:0x17:Completed 4950000 out of 5000000 steps (99%)
00:27:10:WU01:FS01:Connecting to 171.67.108.201:80
00:27:12:WU01:FS01:Assigned to work server 171.64.65.93
00:27:12:WU01:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.93
00:27:12:WU01:FS01:Connecting to 171.64.65.93:8080
00:27:13:WU01:FS01:Downloading 2.88MiB
00:27:19:WU01:FS01:Download 80.35%
00:27:19:WU01:FS01:Download complete
00:27:20:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9101 run:632 clone:0 gen:47 core:0x17 unit:0x000000300a3b1e81533f699697dd255e
00:32:14:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
00:32:26:WU02:FS01:0x17:Saving result file logfile_01.txt
00:32:26:WU02:FS01:0x17:Saving result file checkpointState.xml
00:32:30:WU02:FS01:0x17:Saving result file checkpt.crc
00:32:30:WU02:FS01:0x17:Saving result file log.txt
00:32:30:WU02:FS01:0x17:Saving result file positions.xtc
00:32:38:WU02:FS01:0x17:Folding@home Core Shutdown: FINISHED_UNIT
00:32:39:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
00:32:39:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:9408 run:754 clone:0 gen:2 core:0x17 unit:0x000000020a3b1e5c5342e36e136b6fc2
00:32:39:WU02:FS01:Uploading 23.30MiB to 171.64.65.56
00:32:39:WU02:FS01:Connecting to 171.64.65.56:8080
00:32:39:WU01:FS01:Starting
00:32:39:WU01:FS01:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 01 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -gpu 0 -gpu-vendor nvidia
00:32:39:WU01:FS01:Started FahCore on PID 31862
00:32:39:WU01:FS01:Core PID:31866
00:32:39:WU01:FS01:FahCore 0x17 started
00:32:39:WU01:FS01:0x17:*********************** Log Started 2014-04-23T00:32:39Z ***********************
00:32:39:WU01:FS01:0x17:Project: 9101 (Run 632, Clone 0, Gen 47)
00:32:39:WU01:FS01:0x17:Unit: 0x000000300a3b1e81533f699697dd255e
00:32:39:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
00:32:39:WU01:FS01:0x17:Machine: 1
00:32:39:WU01:FS01:0x17:Reading tar file state.xml
00:32:40:WU01:FS01:0x17:Reading tar file system.xml
00:32:40:WU01:FS01:0x17:Reading tar file integrator.xml
00:32:40:WU01:FS01:0x17:Reading tar file core.xml
00:32:40:WU01:FS01:0x17:Digital signatures verified
00:32:45:WU02:FS01:Upload 9.12%
00:32:51:WU02:FS01:Upload 17.70%
00:32:57:WU02:FS01:Upload 27.63%
00:33:03:WU02:FS01:Upload 35.68%
00:33:09:WU02:FS01:Upload 43.72%
00:33:15:WU02:FS01:Upload 52.57%
00:33:21:WU02:FS01:Upload 60.62%
00:33:27:WU02:FS01:Upload 70.28%
00:33:33:WU02:FS01:Upload 78.86%
00:33:39:WU02:FS01:Upload 86.91%
00:33:45:WU02:FS01:Upload 94.96%
00:34:00:WU01:FS01:0x17:Completed 0 out of 2500000 steps (0%)
00:34:02:WU02:FS01:Upload complete
00:34:02:WU02:FS01:Server responded WORK_ACK (400)
00:34:02:WU02:FS01:Final credit estimate, 71047.00 points
00:34:02:WU02:FS01:Cleaning up
00:40:15:WU01:FS01:0x17:ERROR:exception: The periodic box size has decreased to less than twice the nonbonded cutoff.
00:40:15:WU01:FS01:0x17:Saving result file logfile_01.txt
00:40:15:WU01:FS01:0x17:Saving result file log.txt
00:40:15:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
00:40:15:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
00:40:15:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9101 run:632 clone:0 gen:47 core:0x17 unit:0x000000300a3b1e81533f699697dd255e
00:40:15:WU01:FS01:Uploading 2.34KiB to 171.64.65.93
00:40:15:WU01:FS01:Connecting to 171.64.65.93:8080
00:40:16:WU02:FS01:Connecting to 171.67.108.201:80
00:40:16:WU01:FS01:Upload complete
00:40:16:WU01:FS01:Server responded WORK_ACK (400)
00:40:16:WU01:FS01:Cleaning up
00:40:17:WU02:FS01:Assigned to work server 171.64.65.93
00:40:17:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.93
00:40:17:WU02:FS01:Connecting to 171.64.65.93:8080
00:40:17:WU02:FS01:Downloading 2.79MiB
00:40:23:WU02:FS01:Download 73.80%
00:40:24:WU02:FS01:Download complete
00:40:24:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:9101 run:609 clone:0 gen:35 core:0x17 unit:0x000000250a3b1e81533f67982a52f9d7
00:40:24:WU02:FS01:Starting
00:40:24:WU02:FS01:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 02 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -gpu 0 -gpu-vendor nvidia
00:40:24:WU02:FS01:Started FahCore on PID 2610
00:40:24:WU02:FS01:Core PID:2614
00:40:24:WU02:FS01:FahCore 0x17 started
00:40:24:WU02:FS01:0x17:*********************** Log Started 2014-04-23T00:40:24Z ***********************
00:40:24:WU02:FS01:0x17:Project: 9101 (Run 609, Clone 0, Gen 35)
00:40:24:WU02:FS01:0x17:Unit: 0x000000250a3b1e81533f67982a52f9d7
00:40:24:WU02:FS01:0x17:CPU: 0x00000000000000000000000000000000
00:40:24:WU02:FS01:0x17:Machine: 1
00:40:24:WU02:FS01:0x17:Reading tar file state.xml
00:40:25:WU02:FS01:0x17:Reading tar file system.xml
00:40:25:WU02:FS01:0x17:Reading tar file integrator.xml
00:40:25:WU02:FS01:0x17:Reading tar file core.xml
00:40:25:WU02:FS01:0x17:Digital signatures verified
00:40:59:WU00:FS00:0xa5:Completed 242500 out of 250000 steps  (97%)
00:41:49:WU02:FS01:0x17:Completed 0 out of 2500000 steps (0%)
00:43:37:WU02:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
00:51:09:WU02:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
00:51:17:WU02:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
00:52:59:WU02:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
00:55:36:WU02:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
00:55:43:WU02:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
00:57:25:WU02:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
01:05:09:WU00:FS00:0xa5:Completed 245000 out of 250000 steps  (98%)
01:05:40:WU02:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
01:05:48:WU02:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
01:05:48:WU02:FS01:0x17:Max number of retries reached. Aborting.
01:05:48:WU02:FS01:0x17:ERROR:exception: Max Retries Reached
01:05:48:WU02:FS01:0x17:Saving result file logfile_01.txt
01:05:48:WU02:FS01:0x17:Saving result file badStateCheckpoint_1022402095
01:05:50:WU02:FS01:0x17:Saving result file badStateCheckpoint_80095718
01:05:52:WU02:FS01:0x17:Saving result file badStateCheckpoint_907331616
01:05:55:WU02:FS01:0x17:Saving result file badStateForceGroup0_1022402095Core.xml
01:05:58:WU02:FS01:0x17:Saving result file badStateForceGroup0_1022402095Ref.xml
01:06:00:WU02:FS01:0x17:Saving result file badStateForceGroup0_80095718Core.xml
01:06:02:WU02:FS01:0x17:Saving result file badStateForceGroup0_80095718Ref.xml
01:06:05:WU02:FS01:0x17:Saving result file badStateForceGroup0_907331616Core.xml
01:06:08:WU02:FS01:0x17:Saving result file badStateForceGroup0_907331616Ref.xml
01:06:10:WU02:FS01:0x17:Saving result file badStateForceGroup1_1022402095Core.xml
01:06:14:WU02:FS01:0x17:Saving result file badStateForceGroup1_1022402095Ref.xml
01:06:16:WU02:FS01:0x17:Saving result file badStateForceGroup1_80095718Core.xml
01:06:20:WU02:FS01:0x17:Saving result file badStateForceGroup1_80095718Ref.xml
01:06:23:WU02:FS01:0x17:Saving result file badStateForceGroup1_907331616Core.xml
01:06:26:WU02:FS01:0x17:Saving result file badStateForceGroup1_907331616Ref.xml
01:06:29:WU02:FS01:0x17:Saving result file badStateForceGroup2_1022402095Core.xml
01:06:31:WU02:FS01:0x17:Saving result file badStateForceGroup2_1022402095Ref.xml
01:06:34:WU02:FS01:0x17:Saving result file badStateForceGroup2_80095718Core.xml
01:06:38:WU02:FS01:0x17:Saving result file badStateForceGroup2_80095718Ref.xml
01:06:40:WU02:FS01:0x17:Saving result file badStateForceGroup2_907331616Core.xml
01:06:43:WU02:FS01:0x17:Saving result file badStateForceGroup2_907331616Ref.xml
01:06:46:WU02:FS01:0x17:Saving result file log.txt
01:06:46:WU02:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
01:06:46:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
01:06:46:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:9101 run:609 clone:0 gen:35 core:0x17 unit:0x000000250a3b1e81533f67982a52f9d7
01:06:46:WU02:FS01:Uploading 6.71MiB to 171.64.65.93
01:06:46:WU02:FS01:Connecting to 171.64.65.93:8080
01:06:48:WU01:FS01:Connecting to 171.67.108.201:80
01:06:49:WU01:FS01:Assigned to work server 171.64.65.93
01:06:49:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.93
01:06:49:WU01:FS01:Connecting to 171.64.65.93:8080
01:06:50:WU01:FS01:Downloading 2.99MiB
01:06:52:WU02:FS01:Upload 27.01%
01:06:56:WU01:FS01:Download 23.02%
01:06:58:WU02:FS01:Upload 54.94%
01:07:02:WU01:FS01:Download 48.14%
01:07:04:WU02:FS01:Upload 88.46%
01:07:08:WU01:FS01:Download 62.79%
01:07:08:WU02:FS01:Upload complete
01:07:08:WU02:FS01:Server responded WORK_ACK (400)
01:07:08:WU02:FS01:Cleaning up
01:07:14:WU01:FS01:Download 100.00%
01:07:14:WU01:FS01:Download complete
01:07:14:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9101 run:436 clone:0 gen:35 core:0x17 unit:0x000000290a3b1e81533f582ec59c5bba
01:07:14:WU01:FS01:Starting
01:07:14:WU01:FS01:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 01 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -gpu 0 -gpu-vendor nvidia
01:07:14:WU01:FS01:Started FahCore on PID 13563
01:07:14:WU01:FS01:Core PID:13567
01:07:14:WU01:FS01:FahCore 0x17 started
01:07:14:WU01:FS01:0x17:*********************** Log Started 2014-04-23T01:07:14Z ***********************
01:07:14:WU01:FS01:0x17:Project: 9101 (Run 436, Clone 0, Gen 35)
01:07:14:WU01:FS01:0x17:Unit: 0x000000290a3b1e81533f582ec59c5bba
01:07:14:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
01:07:14:WU01:FS01:0x17:Machine: 1
01:07:14:WU01:FS01:0x17:Reading tar file state.xml
01:07:14:WU01:FS01:0x17:Reading tar file system.xml
01:07:15:WU01:FS01:0x17:Reading tar file integrator.xml
01:07:15:WU01:FS01:0x17:Reading tar file core.xml
01:07:15:WU01:FS01:0x17:Digital signatures verified
01:08:41:WU01:FS01:0x17:Completed 0 out of 2500000 steps (0%)
01:12:24:WU01:FS01:0x17:ERROR:exception: The periodic box size has decreased to less than twice the nonbonded cutoff.
01:12:24:WU01:FS01:0x17:Saving result file logfile_01.txt
01:12:24:WU01:FS01:0x17:Saving result file log.txt
01:12:24:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
01:12:25:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
01:12:25:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9101 run:436 clone:0 gen:35 core:0x17 unit:0x000000290a3b1e81533f582ec59c5bba
01:12:25:WU01:FS01:Uploading 2.34KiB to 171.64.65.93
01:12:25:WU01:FS01:Connecting to 171.64.65.93:8080
01:12:25:WU02:FS01:Connecting to 171.67.108.201:80
01:12:25:WU01:FS01:Upload complete
01:12:25:WU01:FS01:Server responded WORK_ACK (400)
01:12:25:WU01:FS01:Cleaning up
01:12:26:WU02:FS01:Assigned to work server 171.64.65.93
01:12:26:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.93
01:12:26:WU02:FS01:Connecting to 171.64.65.93:8080
01:12:27:WU02:FS01:Downloading 3.01MiB
01:12:33:WU02:FS01:Download 74.75%
01:12:35:WU02:FS01:Download complete
01:12:35:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:9101 run:658 clone:0 gen:36 core:0x17 unit:0x0000002a0a3b1e81533f6bd2bf30e0e8
01:12:35:WU02:FS01:Starting
01:12:35:WU02:FS01:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 02 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -gpu 0 -gpu-vendor nvidia
01:12:35:WU02:FS01:Started FahCore on PID 15696
01:12:35:WU02:FS01:Core PID:15700
01:12:35:WU02:FS01:FahCore 0x17 started
01:12:36:WU02:FS01:0x17:*********************** Log Started 2014-04-23T01:12:35Z ***********************
01:12:36:WU02:FS01:0x17:Project: 9101 (Run 658, Clone 0, Gen 36)
01:12:36:WU02:FS01:0x17:Unit: 0x0000002a0a3b1e81533f6bd2bf30e0e8
01:12:36:WU02:FS01:0x17:CPU: 0x00000000000000000000000000000000
01:12:36:WU02:FS01:0x17:Machine: 1
01:12:36:WU02:FS01:0x17:Reading tar file state.xml
01:12:36:WU02:FS01:0x17:Reading tar file system.xml
01:12:37:WU02:FS01:0x17:Reading tar file integrator.xml
01:12:37:WU02:FS01:0x17:Reading tar file core.xml
01:12:37:WU02:FS01:0x17:Digital signatures verified
01:14:05:WU02:FS01:0x17:Completed 0 out of 2500000 steps (0%)
01:19:43:WU02:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
01:24:28:WU02:FS01:0x17:ERROR:exception: The periodic box size has decreased to less than twice the nonbonded cutoff.
01:24:28:WU02:FS01:0x17:Saving result file logfile_01.txt
01:24:28:WU02:FS01:0x17:Saving result file log.txt
01:24:28:WU02:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
01:24:28:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
01:24:28:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:9101 run:658 clone:0 gen:36 core:0x17 unit:0x0000002a0a3b1e81533f6bd2bf30e0e8
01:24:28:WU02:FS01:Uploading 2.35KiB to 171.64.65.93
01:24:28:WU02:FS01:Connecting to 171.64.65.93:8080
01:24:28:WU01:FS01:Connecting to 171.67.108.201:80
01:24:29:WU02:FS01:Upload complete
01:24:29:WU02:FS01:Server responded WORK_ACK (400)
01:24:29:WU02:FS01:Cleaning up
01:24:29:WU01:FS01:Assigned to work server 171.64.65.93
01:24:29:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.93
01:24:29:WU01:FS01:Connecting to 171.64.65.93:8080
01:24:30:WU01:FS01:Downloading 3.08MiB
01:24:36:WU01:FS01:Download 76.99%
01:24:38:WU01:FS01:Download complete
01:24:39:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9101 run:182 clone:0 gen:42 core:0x17 unit:0x0000002a0a3b1e81533f4126081f64f9
01:24:39:WU01:FS01:Starting
01:24:39:WU01:FS01:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 01 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -gpu 0 -gpu-vendor nvidia
01:24:39:WU01:FS01:Started FahCore on PID 20495
01:24:39:WU01:FS01:Core PID:20499
01:24:39:WU01:FS01:FahCore 0x17 started
01:24:39:WU01:FS01:0x17:*********************** Log Started 2014-04-23T01:24:39Z ***********************
01:24:39:WU01:FS01:0x17:Project: 9101 (Run 182, Clone 0, Gen 42)
01:24:39:WU01:FS01:0x17:Unit: 0x0000002a0a3b1e81533f4126081f64f9
01:24:39:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
01:24:39:WU01:FS01:0x17:Machine: 1
01:24:39:WU01:FS01:0x17:Reading tar file state.xml
01:24:39:WU01:FS01:0x17:Reading tar file system.xml
01:24:40:WU01:FS01:0x17:Reading tar file integrator.xml
01:24:40:WU01:FS01:0x17:Reading tar file core.xml
01:24:40:WU01:FS01:0x17:Digital signatures verified
01:26:11:WU01:FS01:0x17:Completed 0 out of 2500000 steps (0%)
01:28:48:WU01:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
01:29:24:WU00:FS00:0xa5:Completed 247500 out of 250000 steps  (99%)
01:29:25:WU02:FS00:Connecting to 171.67.108.200:8080
01:29:26:WU02:FS00:Assigned to work server 128.143.231.201
01:29:26:WU02:FS00:Requesting new work unit for slot 00: RUNNING cpu:30 from 128.143.231.201
01:29:26:WU02:FS00:Connecting to 128.143.231.201:8080
01:29:33:WU02:FS00:Downloading 28.85MiB
01:29:39:WU02:FS00:Download 16.25%
01:29:45:WU02:FS00:Download 34.45%
01:29:51:WU02:FS00:Download 52.65%
01:29:57:WU02:FS00:Download 70.85%
01:30:03:WU02:FS00:Download 86.23%
01:30:09:WU02:FS00:Download 95.98%
01:30:11:WU02:FS00:Download complete
01:30:11:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:8104 run:0 clone:36 gen:437 core:0xa5 unit:0x0000027c088988e1515201d008406c6e
01:37:59:WU01:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
01:38:07:WU01:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
01:45:03:WU01:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
01:47:51:WU01:FS01:0x17:ERROR:exception: The periodic box size has decreased to less than twice the nonbonded cutoff.
01:47:51:WU01:FS01:0x17:Saving result file logfile_01.txt
01:47:51:WU01:FS01:0x17:Saving result file badStateCheckpoint_1489502184
01:47:53:WU01:FS01:0x17:Saving result file badStateForceGroup0_1489502184Core.xml
01:47:56:WU01:FS01:0x17:Saving result file badStateForceGroup0_1489502184Ref.xml
01:47:59:WU01:FS01:0x17:Saving result file badStateForceGroup1_1489502184Core.xml
01:48:03:WU01:FS01:0x17:Saving result file badStateForceGroup1_1489502184Ref.xml
01:48:06:WU01:FS01:0x17:Saving result file badStateForceGroup2_1489502184Core.xml
01:48:10:WU01:FS01:0x17:Saving result file badStateForceGroup2_1489502184Ref.xml
01:48:13:WU01:FS01:0x17:Saving result file log.txt
01:48:14:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
01:48:14:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
01:48:14:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9101 run:182 clone:0 gen:42 core:0x17 unit:0x0000002a0a3b1e81533f4126081f64f9
01:48:14:WU01:FS01:Uploading 2.47MiB to 171.64.65.93
01:48:14:WU01:FS01:Connecting to 171.64.65.93:8080
01:48:15:WU03:FS01:Connecting to 171.67.108.201:80
01:48:16:WU03:FS01:Assigned to work server 171.64.65.93
01:48:16:WU03:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.93
01:48:16:WU03:FS01:Connecting to 171.64.65.93:8080
01:48:17:WU03:FS01:Downloading 2.96MiB
01:48:20:WU01:FS01:Upload 78.43%
01:48:23:WU03:FS01:Download 14.80%
01:48:23:WU01:FS01:Upload complete
01:48:23:WU01:FS01:Server responded WORK_ACK (400)
01:48:23:WU01:FS01:Cleaning up
01:48:30:WU03:FS01:Download 100.00%
01:48:30:WU03:FS01:Download complete
01:48:30:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:9101 run:35 clone:0 gen:48 core:0x17 unit:0x000000320a3b1e81533f3403303c5b72
01:48:30:WU03:FS01:Starting
01:48:30:WU03:FS01:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 03 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -gpu 0 -gpu-vendor nvidia
01:48:30:WU03:FS01:Started FahCore on PID 30063
01:48:30:WU03:FS01:Core PID:30067
01:48:30:WU03:FS01:FahCore 0x17 started
01:48:30:WU03:FS01:0x17:*********************** Log Started 2014-04-23T01:48:30Z ***********************
01:48:30:WU03:FS01:0x17:Project: 9101 (Run 35, Clone 0, Gen 48)
01:48:30:WU03:FS01:0x17:Unit: 0x000000320a3b1e81533f3403303c5b72
01:48:30:WU03:FS01:0x17:CPU: 0x00000000000000000000000000000000
01:48:30:WU03:FS01:0x17:Machine: 1
01:48:30:WU03:FS01:0x17:Reading tar file state.xml
01:48:30:WU03:FS01:0x17:Reading tar file system.xml
01:48:31:WU03:FS01:0x17:Reading tar file integrator.xml
01:48:31:WU03:FS01:0x17:Reading tar file core.xml
01:48:31:WU03:FS01:0x17:Digital signatures verified
01:49:54:WU03:FS01:0x17:Completed 0 out of 2500000 steps (0%)
01:51:47:WU03:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
01:53:24:WU00:FS00:0xa5:Completed 250000 out of 250000 steps  (100%)
01:53:38:WU00:FS00:0xa5:DynamicWrapper: Finished Work Unit: sleep=10000
01:53:48:WU00:FS00:0xa5:
01:53:48:WU00:FS00:0xa5:Finished Work Unit:
01:53:48:WU00:FS00:0xa5:- Reading up to 64340496 from "00/wudata_01.trr": Read 64340496
01:53:49:WU00:FS00:0xa5:trr file hash check passed.
01:53:49:WU00:FS00:0xa5:- Reading up to 31675540 from "00/wudata_01.xtc": Read 31675540
01:53:49:WU00:FS00:0xa5:xtc file hash check passed.
01:53:49:WU00:FS00:0xa5:edr file hash check passed.
01:53:49:WU00:FS00:0xa5:logfile size: 237565
01:53:49:WU00:FS00:0xa5:Leaving Run
01:53:52:WU00:FS00:0xa5:- Writing 96414477 bytes of core data to disk...
01:54:12:WU00:FS00:0xa5:Done: 96413965 -> 91635852 (compressed to 5.9 percent)
01:54:12:WU00:FS00:0xa5:  ... Done.
01:54:13:WU03:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
01:54:17:WU03:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
01:56:50:WU00:FS00:0xa5:- Shutting down core
01:56:50:WU00:FS00:0xa5:
01:56:50:WU00:FS00:0xa5:Folding@home Core Shutdown: FINISHED_UNIT
01:57:10:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
01:57:10:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:8101 run:8 clone:6 gen:323 core:0xa5 unit:0x000001c5088988e14f296b98661443f1
01:57:10:WU00:FS00:Uploading 87.39MiB to 128.143.231.201
01:57:10:WU00:FS00:Connecting to 128.143.231.201:8080
01:57:10:WU02:FS00:Starting
01:57:10:WU02:FS00:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/beta/Core_a5.fah/FahCore_a5 -dir 02 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -np 30
01:57:10:WU02:FS00:Started FahCore on PID 1082
01:57:10:WU02:FS00:Core PID:1086
01:57:10:WU02:FS00:FahCore 0xa5 started
01:57:10:WU02:FS00:0xa5:
01:57:10:WU02:FS00:0xa5:*------------------------------*
01:57:10:WU02:FS00:0xa5:Folding@Home Gromacs SMP Core
01:57:10:WU02:FS00:0xa5:Version 2.27 (Thu Feb 10 09:46:40 PST 2011)
01:57:10:WU02:FS00:0xa5:
01:57:10:WU02:FS00:0xa5:Preparing to commence simulation
01:57:10:WU02:FS00:0xa5:- Looking at optimizations...
01:57:10:WU02:FS00:0xa5:- Created dyn
01:57:10:WU02:FS00:0xa5:- Files status OK
01:57:13:WU02:FS00:0xa5:- Expanded 30248806 -> 33024540 (decompressed 109.1 percent)
01:57:13:WU02:FS00:0xa5:Called DecompressByteArray: compressed_data_size=30248806 data_size=33024540, decompressed_data_size=33024540 diff=0
01:57:13:WU02:FS00:0xa5:- Digital signature verified
01:57:13:WU02:FS00:0xa5:
01:57:13:WU02:FS00:0xa5:Project: 8104 (Run 0, Clone 36, Gen 437)
01:57:13:WU02:FS00:0xa5:
01:57:13:WU02:FS00:0xa5:Assembly optimizations on if available.
01:57:13:WU02:FS00:0xa5:Entering M.D.
01:57:16:WU00:FS00:Upload 2.43%
01:57:20:WU02:FS00:0xa5:Mapping NT from 30 to 30 
01:57:22:WU02:FS00:0xa5:Completed 0 out of 250000 steps  (0%)
01:57:22:WU00:FS00:Upload 5.08%
01:57:28:WU00:FS00:Upload 7.58%
Reading file ./work/02/wudata_01.tpr, VERSION 4.5.5-dev-20120903-d64b9e3 (single precision)
Reading file ./work/02/wudata_01.tpr, VERSION 4.5.5-dev-20120903-d64b9e3 (single precision)
01:57:34:WU00:FS00:Upload 9.73%
01:57:40:WU00:FS00:Upload 12.01%
01:57:46:WU00:FS00:Upload 14.30%
01:57:52:WU00:FS00:Upload 16.52%
01:57:58:WU00:FS00:Upload 18.88%
01:58:04:WU00:FS00:Upload 21.03%
01:58:10:WU00:FS00:Upload 23.46%
01:58:16:WU00:FS00:Upload 25.96%
01:58:23:WU00:FS00:Upload 28.25%
01:58:29:WU00:FS00:Upload 31.04%
01:58:35:WU00:FS00:Upload 33.18%
01:58:41:WU00:FS00:Upload 35.47%
01:58:47:WU00:FS00:Upload 37.69%
01:58:53:WU00:FS00:Upload 39.91%
01:58:59:WU00:FS00:Upload 42.41%
01:59:05:WU00:FS00:Upload 44.27%
01:59:11:WU00:FS00:Upload 46.99%
01:59:17:WU00:FS00:Upload 49.13%
01:59:23:WU00:FS00:Upload 51.35%
01:59:29:WU00:FS00:Upload 53.78%
01:59:36:WU00:FS00:Upload 56.28%
01:59:42:WU00:FS00:Upload 58.50%
01:59:48:WU00:FS00:Upload 61.15%
01:59:54:WU00:FS00:Upload 62.94%
02:00:00:WU00:FS00:Upload 65.80%
02:00:06:WU00:FS00:Upload 68.01%
02:00:12:WU00:FS00:Upload 70.16%
02:00:18:WU00:FS00:Upload 72.45%
02:00:18:WU03:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
02:00:25:WU00:FS00:Upload 74.95%
02:00:31:WU00:FS00:Upload 77.60%
02:00:37:WU00:FS00:Upload 79.81%
02:00:43:WU00:FS00:Upload 81.96%
02:00:49:WU00:FS00:Upload 84.46%
02:00:55:WU00:FS00:Upload 86.82%
02:01:01:WU00:FS00:Upload 89.11%
02:01:08:WU00:FS00:Upload 91.26%
02:01:14:WU00:FS00:Upload 93.90%
02:01:20:WU00:FS00:Upload 96.41%
02:01:26:WU00:FS00:Upload 98.41%
02:01:44:WU00:FS00:Upload complete
02:01:44:WU00:FS00:Server responded WORK_ACK (400)
02:01:44:WU00:FS00:Final credit estimate, 246621.00 points
02:01:44:WU00:FS00:Cleaning up
02:09:06:WU03:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
02:09:14:WU03:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
02:11:00:WU02:FS00:0xa5:Completed 2500 out of 250000 steps  (1%)
02:11:01:WU03:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
02:15:14:WU03:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
02:15:22:WU03:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
02:15:22:WU03:FS01:0x17:Max number of retries reached. Aborting.
02:15:22:WU03:FS01:0x17:ERROR:exception: Max Retries Reached
02:15:22:WU03:FS01:0x17:Saving result file logfile_01.txt
02:15:22:WU03:FS01:0x17:Saving result file badStateCheckpoint_1418553412
02:15:24:WU03:FS01:0x17:Saving result file badStateCheckpoint_1732536296
02:15:27:WU03:FS01:0x17:Saving result file badStateCheckpoint_1905116342
02:15:29:WU03:FS01:0x17:Saving result file badStateForceGroup0_1418553412Core.xml
02:15:33:WU03:FS01:0x17:Saving result file badStateForceGroup0_1418553412Ref.xml
02:15:35:WU03:FS01:0x17:Saving result file badStateForceGroup0_1732536296Core.xml
02:15:38:WU03:FS01:0x17:Saving result file badStateForceGroup0_1732536296Ref.xml
02:15:41:WU03:FS01:0x17:Saving result file badStateForceGroup0_1905116342Core.xml
02:15:44:WU03:FS01:0x17:Saving result file badStateForceGroup0_1905116342Ref.xml
02:15:47:WU03:FS01:0x17:Saving result file badStateForceGroup1_1418553412Core.xml
02:15:50:WU03:FS01:0x17:Saving result file badStateForceGroup1_1418553412Ref.xml
02:15:53:WU03:FS01:0x17:Saving result file badStateForceGroup1_1732536296Core.xml
02:15:57:WU03:FS01:0x17:Saving result file badStateForceGroup1_1732536296Ref.xml
02:16:00:WU03:FS01:0x17:Saving result file badStateForceGroup1_1905116342Core.xml
02:16:03:WU03:FS01:0x17:Saving result file badStateForceGroup1_1905116342Ref.xml
02:16:06:WU03:FS01:0x17:Saving result file badStateForceGroup2_1418553412Core.xml
02:16:09:WU03:FS01:0x17:Saving result file badStateForceGroup2_1418553412Ref.xml
02:16:12:WU03:FS01:0x17:Saving result file badStateForceGroup2_1732536296Core.xml
02:16:15:WU03:FS01:0x17:Saving result file badStateForceGroup2_1732536296Ref.xml
02:16:18:WU03:FS01:0x17:Saving result file badStateForceGroup2_1905116342Core.xml
02:16:21:WU03:FS01:0x17:Saving result file badStateForceGroup2_1905116342Ref.xml
02:16:24:WU03:FS01:0x17:Saving result file log.txt
02:16:24:WU03:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
02:16:25:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
02:16:25:WU03:FS01:Sending unit results: id:03 state:SEND error:FAULTY project:9101 run:35 clone:0 gen:48 core:0x17 unit:0x000000320a3b1e81533f3403303c5b72
02:16:25:WU03:FS01:Uploading 7.13MiB to 171.64.65.93
02:16:25:WU03:FS01:Connecting to 171.64.65.93:8080
02:16:25:WU00:FS01:Connecting to 171.67.108.201:80
02:16:26:WU00:FS01:Assigned to work server 171.64.65.93
02:16:26:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.93
02:16:26:WU00:FS01:Connecting to 171.64.65.93:8080
02:16:27:WU00:FS01:Downloading 3.32MiB
02:16:31:WU03:FS01:Upload 29.82%
02:16:35:WU00:FS01:Download 9.40%
02:16:37:WU03:FS01:Upload 59.64%
02:16:41:WU00:FS01:Download 13.17%
02:16:43:WU03:FS01:Upload 85.07%
02:16:47:WU03:FS01:Upload complete
02:16:47:WU03:FS01:Server responded WORK_ACK (400)
02:16:47:WU03:FS01:Cleaning up
02:16:47:WU00:FS01:Download 16.93%
02:16:53:WU00:FS01:Download 58.31%
02:16:55:WU00:FS01:Download complete
02:16:55:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9101 run:933 clone:0 gen:25 core:0x17 unit:0x000000190a3b1e81533f847b5423c63f
02:16:55:WU00:FS01:Starting
02:16:55:WU00:FS01:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 00 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -gpu 0 -gpu-vendor nvidia
02:16:55:WU00:FS01:Started FahCore on PID 9176
02:16:55:WU00:FS01:Core PID:9180
02:16:55:WU00:FS01:FahCore 0x17 started
02:16:56:WU00:FS01:0x17:*********************** Log Started 2014-04-23T02:16:55Z ***********************
02:16:56:WU00:FS01:0x17:Project: 9101 (Run 933, Clone 0, Gen 25)
02:16:56:WU00:FS01:0x17:Unit: 0x000000190a3b1e81533f847b5423c63f
02:16:56:WU00:FS01:0x17:CPU: 0x00000000000000000000000000000000
02:16:56:WU00:FS01:0x17:Machine: 1
02:16:56:WU00:FS01:0x17:Reading tar file state.xml
02:16:56:WU00:FS01:0x17:Reading tar file system.xml
02:16:57:WU00:FS01:0x17:Reading tar file integrator.xml
02:16:57:WU00:FS01:0x17:Reading tar file core.xml
02:16:57:WU00:FS01:0x17:Digital signatures verified
02:18:36:WU00:FS01:0x17:Completed 0 out of 2500000 steps (0%)
02:20:41:WU00:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
02:22:40:WU00:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
02:24:54:WU02:FS00:0xa5:Completed 5000 out of 250000 steps  (2%)
02:31:57:WU00:FS01:0x17:Completed 75000 out of 2500000 steps (3%)
02:38:39:WU02:FS00:0xa5:Completed 7500 out of 250000 steps  (3%)
02:41:53:WU00:FS01:0x17:Completed 100000 out of 2500000 steps (4%)
02:42:03:WU00:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
02:44:02:WU00:FS01:0x17:Completed 75000 out of 2500000 steps (3%)
02:46:01:WU00:FS01:0x17:Completed 100000 out of 2500000 steps (4%)
02:48:10:WU00:FS01:0x17:Completed 125000 out of 2500000 steps (5%)
02:50:09:WU00:FS01:0x17:Completed 150000 out of 2500000 steps (6%)
02:52:18:WU00:FS01:0x17:Completed 175000 out of 2500000 steps (7%)
02:52:31:WU02:FS00:0xa5:Completed 10000 out of 250000 steps  (4%)
02:54:17:WU00:FS01:0x17:Completed 200000 out of 2500000 steps (8%)
02:56:25:WU00:FS01:0x17:Completed 225000 out of 2500000 steps (9%)
02:58:25:WU00:FS01:0x17:Completed 250000 out of 2500000 steps (10%)
03:00:34:WU00:FS01:0x17:Completed 275000 out of 2500000 steps (11%)
03:02:32:WU00:FS01:0x17:Completed 300000 out of 2500000 steps (12%)
03:04:41:WU00:FS01:0x17:Completed 325000 out of 2500000 steps (13%)
03:06:29:WU02:FS00:0xa5:Completed 12500 out of 250000 steps  (5%)
03:06:40:WU00:FS01:0x17:Completed 350000 out of 2500000 steps (14%)
03:08:49:WU00:FS01:0x17:Completed 375000 out of 2500000 steps (15%)
03:10:48:WU00:FS01:0x17:Completed 400000 out of 2500000 steps (16%)
03:12:58:WU00:FS01:0x17:Completed 425000 out of 2500000 steps (17%)
03:14:57:WU00:FS01:0x17:Completed 450000 out of 2500000 steps (18%)
03:20:32:WU02:FS00:0xa5:Completed 15000 out of 250000 steps  (6%)
03:24:26:WU00:FS01:0x17:Completed 475000 out of 2500000 steps (19%)
03:34:23:WU00:FS01:0x17:Completed 500000 out of 2500000 steps (20%)
03:34:33:WU00:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
03:34:38:WU02:FS00:0xa5:Completed 17500 out of 250000 steps  (7%)
03:38:26:WU00:FS01:0x17:Completed 475000 out of 2500000 steps (19%)
03:48:23:WU00:FS01:0x17:Completed 500000 out of 2500000 steps (20%)
03:48:32:WU00:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
03:48:32:WU00:FS01:0x17:Max number of retries reached. Aborting.
03:48:32:WU00:FS01:0x17:ERROR:exception: Max Retries Reached
03:48:32:WU00:FS01:0x17:Saving result file logfile_01.txt
03:48:32:WU00:FS01:0x17:Saving result file badStateCheckpoint_1109040189
03:48:34:WU00:FS01:0x17:Saving result file badStateCheckpoint_2103561287
03:48:35:WU02:FS00:0xa5:Completed 20000 out of 250000 steps  (8%)
03:48:37:WU00:FS01:0x17:Saving result file badStateCheckpoint_2133866875
03:48:40:WU00:FS01:0x17:Saving result file badStateForceGroup0_1109040189Core.xml
03:48:43:WU00:FS01:0x17:Saving result file badStateForceGroup0_1109040189Ref.xml
03:48:47:WU00:FS01:0x17:Saving result file badStateForceGroup0_2103561287Core.xml
03:48:50:WU00:FS01:0x17:Saving result file badStateForceGroup0_2103561287Ref.xml
03:48:53:WU00:FS01:0x17:Saving result file badStateForceGroup0_2133866875Core.xml
03:48:56:WU00:FS01:0x17:Saving result file badStateForceGroup0_2133866875Ref.xml
03:49:00:WU00:FS01:0x17:Saving result file badStateForceGroup1_1109040189Core.xml
03:49:03:WU00:FS01:0x17:Saving result file badStateForceGroup1_1109040189Ref.xml
03:49:07:WU00:FS01:0x17:Saving result file badStateForceGroup1_2103561287Core.xml
03:49:11:WU00:FS01:0x17:Saving result file badStateForceGroup1_2103561287Ref.xml
03:49:15:WU00:FS01:0x17:Saving result file badStateForceGroup1_2133866875Core.xml
03:49:19:WU00:FS01:0x17:Saving result file badStateForceGroup1_2133866875Ref.xml
03:49:22:WU00:FS01:0x17:Saving result file badStateForceGroup2_1109040189Core.xml
03:49:26:WU00:FS01:0x17:Saving result file badStateForceGroup2_1109040189Ref.xml
03:49:29:WU00:FS01:0x17:Saving result file badStateForceGroup2_2103561287Core.xml
03:49:32:WU00:FS01:0x17:Saving result file badStateForceGroup2_2103561287Ref.xml
03:49:35:WU00:FS01:0x17:Saving result file badStateForceGroup2_2133866875Core.xml
03:49:39:WU00:FS01:0x17:Saving result file badStateForceGroup2_2133866875Ref.xml
03:49:42:WU00:FS01:0x17:Saving result file log.txt
03:49:43:WU00:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
03:49:43:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:49:43:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9101 run:933 clone:0 gen:25 core:0x17 unit:0x000000190a3b1e81533f847b5423c63f
03:49:43:WU00:FS01:Uploading 8.13MiB to 171.64.65.93
03:49:43:WU00:FS01:Connecting to 171.64.65.93:8080
03:49:44:WU01:FS01:Connecting to 171.67.108.201:80
03:49:45:WU01:FS01:Assigned to work server 171.64.65.93
03:49:45:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.93
03:49:45:WU01:FS01:Connecting to 171.64.65.93:8080
03:49:46:WU01:FS01:Downloading 3.08MiB
03:49:49:WU00:FS01:Upload 22.30%
03:49:52:WU01:FS01:Download 22.32%
03:49:55:WU00:FS01:Upload 48.45%
03:49:58:WU01:FS01:Download 46.68%
03:50:01:WU00:FS01:Upload 71.52%
03:50:04:WU01:FS01:Download 60.88%
03:50:07:WU00:FS01:Upload 99.21%
03:50:09:WU00:FS01:Upload complete
03:50:09:WU00:FS01:Server responded WORK_ACK (400)
03:50:09:WU00:FS01:Cleaning up
03:50:10:WU01:FS01:Download 79.15%
03:50:13:WU01:FS01:Download complete
03:50:13:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9101 run:20 clone:1 gen:24 core:0x17 unit:0x000000180a3b1e81533f32a93965999b
03:50:13:WU01:FS01:Starting
03:50:13:WU01:FS01:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 01 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -gpu 0 -gpu-vendor nvidia
03:50:13:WU01:FS01:Started FahCore on PID 14136
03:50:13:WU01:FS01:Core PID:14140
03:50:13:WU01:FS01:FahCore 0x17 started
03:50:14:WU01:FS01:0x17:*********************** Log Started 2014-04-23T03:50:13Z ***********************
03:50:14:WU01:FS01:0x17:Project: 9101 (Run 20, Clone 1, Gen 24)
03:50:14:WU01:FS01:0x17:Unit: 0x000000180a3b1e81533f32a93965999b
03:50:14:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
03:50:14:WU01:FS01:0x17:Machine: 1
03:50:14:WU01:FS01:0x17:Reading tar file state.xml
03:50:14:WU01:FS01:0x17:Reading tar file system.xml
03:50:15:WU01:FS01:0x17:Reading tar file integrator.xml
03:50:15:WU01:FS01:0x17:Reading tar file core.xml
03:50:15:WU01:FS01:0x17:Digital signatures verified
03:51:47:WU01:FS01:0x17:Completed 0 out of 2500000 steps (0%)
03:53:44:WU01:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
04:00:56:WU01:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
04:01:04:WU01:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
04:02:31:WU02:FS00:0xa5:Completed 22500 out of 250000 steps  (9%)
04:04:27:WU01:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
04:13:37:WU01:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
04:13:45:WU01:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
04:16:42:WU02:FS00:0xa5:Completed 25000 out of 250000 steps  (10%)
04:18:56:WU01:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
04:28:06:WU01:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
04:28:14:WU01:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
04:28:14:WU01:FS01:0x17:Max number of retries reached. Aborting.
04:28:14:WU01:FS01:0x17:ERROR:exception: Max Retries Reached
04:28:14:WU01:FS01:0x17:Saving result file logfile_01.txt
04:28:14:WU01:FS01:0x17:Saving result file badStateCheckpoint_1076910645
04:28:17:WU01:FS01:0x17:Saving result file badStateCheckpoint_1377403916
04:28:19:WU01:FS01:0x17:Saving result file badStateCheckpoint_856068276
04:28:22:WU01:FS01:0x17:Saving result file badStateForceGroup0_1076910645Core.xml
04:28:25:WU01:FS01:0x17:Saving result file badStateForceGroup0_1076910645Ref.xml
04:28:27:WU01:FS01:0x17:Saving result file badStateForceGroup0_1377403916Core.xml
04:28:30:WU01:FS01:0x17:Saving result file badStateForceGroup0_1377403916Ref.xml
04:28:33:WU01:FS01:0x17:Saving result file badStateForceGroup0_856068276Core.xml
04:28:36:WU01:FS01:0x17:Saving result file badStateForceGroup0_856068276Ref.xml
04:28:39:WU01:FS01:0x17:Saving result file badStateForceGroup1_1076910645Core.xml
04:28:42:WU01:FS01:0x17:Saving result file badStateForceGroup1_1076910645Ref.xml
04:28:45:WU01:FS01:0x17:Saving result file badStateForceGroup1_1377403916Core.xml
04:28:49:WU01:FS01:0x17:Saving result file badStateForceGroup1_1377403916Ref.xml
04:28:51:WU01:FS01:0x17:Saving result file badStateForceGroup1_856068276Core.xml
04:28:54:WU01:FS01:0x17:Saving result file badStateForceGroup1_856068276Ref.xml
04:28:55:WU01:FS01:0x17:Saving result file badStateForceGroup2_1076910645Core.xml
04:28:57:WU01:FS01:0x17:Saving result file badStateForceGroup2_1076910645Ref.xml
04:28:59:WU01:FS01:0x17:Saving result file badStateForceGroup2_1377403916Core.xml
04:29:01:WU01:FS01:0x17:Saving result file badStateForceGroup2_1377403916Ref.xml
04:29:03:WU01:FS01:0x17:Saving result file badStateForceGroup2_856068276Core.xml
04:29:05:WU01:FS01:0x17:Saving result file badStateForceGroup2_856068276Ref.xml
04:29:07:WU01:FS01:0x17:Saving result file log.txt
04:29:07:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
04:29:08:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
04:29:08:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9101 run:20 clone:1 gen:24 core:0x17 unit:0x000000180a3b1e81533f32a93965999b
04:29:08:WU01:FS01:Uploading 7.39MiB to 171.64.65.93
04:29:08:WU01:FS01:Connecting to 171.64.65.93:8080
04:29:08:WU00:FS01:Connecting to 171.67.108.201:80
04:29:09:WU00:FS01:Assigned to work server 171.64.65.93
04:29:09:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.93
04:29:09:WU00:FS01:Connecting to 171.64.65.93:8080
04:29:10:WU00:FS01:Downloading 3.11MiB
04:29:14:WU01:FS01:Upload 30.43%
04:29:16:WU00:FS01:Download 14.08%
04:29:20:WU01:FS01:Upload 54.94%
04:29:22:WU00:FS01:Download 36.21%
04:29:26:WU01:FS01:Upload 83.68%
04:29:28:WU00:FS01:Download 52.30%
04:29:31:WU01:FS01:Upload complete
04:29:31:WU01:FS01:Server responded WORK_ACK (400)
04:29:31:WU01:FS01:Cleaning up
04:29:34:WU00:FS01:Download 82.47%
04:29:37:WU00:FS01:Download complete
04:29:37:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9101 run:1 clone:0 gen:55 core:0x17 unit:0x000000390a3b1e81533f30f5cf5f55dc
04:29:37:WU00:FS01:Starting
04:29:37:WU00:FS01:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 00 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -gpu 0 -gpu-vendor nvidia
04:29:37:WU00:FS01:Started FahCore on PID 29794
04:29:37:WU00:FS01:Core PID:29798
04:29:37:WU00:FS01:FahCore 0x17 started
04:29:37:WU00:FS01:0x17:*********************** Log Started 2014-04-23T04:29:37Z ***********************
04:29:37:WU00:FS01:0x17:Project: 9101 (Run 1, Clone 0, Gen 55)
04:29:37:WU00:FS01:0x17:Unit: 0x000000390a3b1e81533f30f5cf5f55dc
04:29:37:WU00:FS01:0x17:CPU: 0x00000000000000000000000000000000
04:29:37:WU00:FS01:0x17:Machine: 1
04:29:37:WU00:FS01:0x17:Reading tar file state.xml
04:29:38:WU00:FS01:0x17:Reading tar file system.xml
04:29:38:WU00:FS01:0x17:Reading tar file integrator.xml
04:29:38:WU00:FS01:0x17:Reading tar file core.xml
04:29:38:WU00:FS01:0x17:Digital signatures verified
04:30:45:WU02:FS00:0xa5:Completed 27500 out of 250000 steps  (11%)
04:31:12:WU00:FS01:0x17:Completed 0 out of 2500000 steps (0%)
04:33:10:WU00:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
04:39:27:WU00:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
04:39:35:WU00:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
04:44:36:WU02:FS00:0xa5:Completed 30000 out of 250000 steps  (12%)
04:44:40:WU00:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
******************************* Date: 2014-04-23 *******************************
04:53:55:WU00:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
04:54:03:WU00:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
04:55:55:WU00:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
04:57:46:WU00:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
04:58:27:WU02:FS00:0xa5:Completed 32500 out of 250000 steps  (13%)
05:00:18:WU00:FS01:0x17:Completed 75000 out of 2500000 steps (3%)
05:09:33:WU00:FS01:0x17:Completed 100000 out of 2500000 steps (4%)
05:09:42:WU00:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
05:09:42:WU00:FS01:0x17:Max number of retries reached. Aborting.
05:09:42:WU00:FS01:0x17:ERROR:exception: Max Retries Reached
05:09:42:WU00:FS01:0x17:Saving result file logfile_01.txt
05:09:42:WU00:FS01:0x17:Saving result file badStateCheckpoint_1108170130
05:09:45:WU00:FS01:0x17:Saving result file badStateCheckpoint_776945735
05:09:48:WU00:FS01:0x17:Saving result file badStateCheckpoint_812386184
05:09:50:WU00:FS01:0x17:Saving result file badStateForceGroup0_1108170130Core.xml
05:09:54:WU00:FS01:0x17:Saving result file badStateForceGroup0_1108170130Ref.xml
05:09:57:WU00:FS01:0x17:Saving result file badStateForceGroup0_776945735Core.xml
05:10:00:WU00:FS01:0x17:Saving result file badStateForceGroup0_776945735Ref.xml
05:10:02:WU00:FS01:0x17:Saving result file badStateForceGroup0_812386184Core.xml
05:10:05:WU00:FS01:0x17:Saving result file badStateForceGroup0_812386184Ref.xml
05:10:08:WU00:FS01:0x17:Saving result file badStateForceGroup1_1108170130Core.xml
05:10:12:WU00:FS01:0x17:Saving result file badStateForceGroup1_1108170130Ref.xml
05:10:14:WU00:FS01:0x17:Saving result file badStateForceGroup1_776945735Core.xml
05:10:18:WU00:FS01:0x17:Saving result file badStateForceGroup1_776945735Ref.xml
05:10:21:WU00:FS01:0x17:Saving result file badStateForceGroup1_812386184Core.xml
05:10:24:WU00:FS01:0x17:Saving result file badStateForceGroup1_812386184Ref.xml
05:10:26:WU00:FS01:0x17:Saving result file badStateForceGroup2_1108170130Core.xml
05:10:28:WU00:FS01:0x17:Saving result file badStateForceGroup2_1108170130Ref.xml
05:10:30:WU00:FS01:0x17:Saving result file badStateForceGroup2_776945735Core.xml
05:10:32:WU00:FS01:0x17:Saving result file badStateForceGroup2_776945735Ref.xml
05:10:34:WU00:FS01:0x17:Saving result file badStateForceGroup2_812386184Core.xml
05:10:36:WU00:FS01:0x17:Saving result file badStateForceGroup2_812386184Ref.xml
05:10:38:WU00:FS01:0x17:Saving result file log.txt
05:10:39:WU00:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
05:10:39:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
05:10:39:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9101 run:1 clone:0 gen:55 core:0x17 unit:0x000000390a3b1e81533f30f5cf5f55dc
05:10:39:WU00:FS01:Uploading 7.48MiB to 171.64.65.93
05:10:39:WU00:FS01:Connecting to 171.64.65.93:8080
05:10:41:WU01:FS01:Connecting to 171.67.108.201:80
05:10:42:WU01:FS01:Assigned to work server 171.64.65.93
05:10:42:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.93
05:10:42:WU01:FS01:Connecting to 171.64.65.93:8080
05:10:43:WU01:FS01:Downloading 2.95MiB
05:10:45:WU00:FS01:Upload 24.23%
05:10:49:WU01:FS01:Download 14.83%
05:10:51:WU00:FS01:Upload 52.63%
05:10:55:WU01:FS01:Download 31.77%
05:10:57:WU00:FS01:Upload 80.19%
05:11:01:WU01:FS01:Download 55.07%
05:11:03:WU00:FS01:Upload complete
05:11:03:WU00:FS01:Server responded WORK_ACK (400)
05:11:03:WU00:FS01:Cleaning up
05:11:06:WU01:FS01:Download complete
05:11:06:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9101 run:982 clone:0 gen:22 core:0x17 unit:0x000000160a3b1e81533f88f6b5eb27a3
05:11:06:WU01:FS01:Starting
05:11:06:WU01:FS01:Running FahCore: /home/tbm01/fahv7/v7/FAHCoreWrapper /home/tbm01/fahv7/v7/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17 -dir 01 -suffix 01 -version 704 -lifeline 4953 -checkpoint 3 -gpu 0 -gpu-vendor nvidia
05:11:06:WU01:FS01:Started FahCore on PID 14203
05:11:06:WU01:FS01:Core PID:14207
05:11:06:WU01:FS01:FahCore 0x17 started
05:11:06:WU01:FS01:0x17:*********************** Log Started 2014-04-23T05:11:06Z ***********************
05:11:06:WU01:FS01:0x17:Project: 9101 (Run 982, Clone 0, Gen 22)
05:11:06:WU01:FS01:0x17:Unit: 0x000000160a3b1e81533f88f6b5eb27a3
05:11:06:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
05:11:06:WU01:FS01:0x17:Machine: 1
05:11:06:WU01:FS01:0x17:Reading tar file state.xml
05:11:07:WU01:FS01:0x17:Reading tar file system.xml
05:11:07:WU01:FS01:0x17:Reading tar file integrator.xml
05:11:07:WU01:FS01:0x17:Reading tar file core.xml
05:11:07:WU01:FS01:0x17:Digital signatures verified
05:12:22:WU02:FS00:0xa5:Completed 35000 out of 250000 steps  (14%)
05:12:34:WU01:FS01:0x17:Completed 0 out of 2500000 steps (0%)
05:14:26:WU01:FS01:0x17:Completed 25000 out of 2500000 steps (1%)
05:16:11:WU01:FS01:0x17:Completed 50000 out of 2500000 steps (2%)
05:21:56:WU01:FS01:0x17:Completed 75000 out of 2500000 steps (3%)
05:26:27:WU02:FS00:0xa5:Completed 37500 out of 250000 steps  (15%)
05:30:43:WU01:FS01:0x17:Completed 100000 out of 2500000 steps (4%)
05:30:51:WU01:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
05:32:36:WU01:FS01:0x17:Completed 75000 out of 2500000 steps (3%)
05:38:56:WU01:FS01:0x17:Completed 100000 out of 2500000 steps (4%)
05:39:04:WU01:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
05:40:28:WU02:FS00:0xa5:Completed 40000 out of 250000 steps  (16%)
05:40:49:WU01:FS01:0x17:Completed 75000 out of 2500000 steps (3%)
05:49:22:WU01:FS01:0x17:Completed 100000 out of 2500000 steps (4%)
05:49:29:WU01:FS01:0x17:Bad State detected... attempting to resume from last good checkpoint
05:49:29:WU01:FS01:0x17:Max number of retries reached. Aborting.
05:49:29:WU01:FS01:0x17:ERROR:exception: Max Retries Reached
05:49:29:WU01:FS01:0x17:Saving result file logfile_01.txt
05:49:29:WU01:FS01:0x17:Saving result file badStateCheckpoint_1390493414
05:49:31:WU01:FS01:0x17:Saving result file badStateCheckpoint_260750969
05:49:33:WU01:FS01:0x17:Saving result file badStateCheckpoint_334315952
05:49:36:WU01:FS01:0x17:Saving result file badStateForceGroup0_1390493414Core.xml
05:49:39:WU01:FS01:0x17:Saving result file badStateForceGroup0_1390493414Ref.xml
05:49:42:WU01:FS01:0x17:Saving result file badStateForceGroup0_260750969Core.xml
05:49:44:WU01:FS01:0x17:Saving result file badStateForceGroup0_260750969Ref.xml
05:49:47:WU01:FS01:0x17:Saving result file badStateForceGroup0_334315952Core.xml
05:49:49:WU01:FS01:0x17:Saving result file badStateForceGroup0_334315952Ref.xml
05:49:52:WU01:FS01:0x17:Saving result file badStateForceGroup1_1390493414Core.xml
05:49:56:WU01:FS01:0x17:Saving result file badStateForceGroup1_1390493414Ref.xml
05:49:59:WU01:FS01:0x17:Saving result file badStateForceGroup1_260750969Core.xml
05:50:02:WU01:FS01:0x17:Saving result file badStateForceGroup1_260750969Ref.xml
05:50:05:WU01:FS01:0x17:Saving result file badStateForceGroup1_334315952Core.xml
05:50:08:WU01:FS01:0x17:Saving result file badStateForceGroup1_334315952Ref.xml
05:50:11:WU01:FS01:0x17:Saving result file badStateForceGroup2_1390493414Core.xml
05:50:14:WU01:FS01:0x17:Saving result file badStateForceGroup2_1390493414Ref.xml
05:50:17:WU01:FS01:0x17:Saving result file badStateForceGroup2_260750969Core.xml
05:50:19:WU01:FS01:0x17:Saving result file badStateForceGroup2_260750969Ref.xml
05:50:22:WU01:FS01:0x17:Saving result file badStateForceGroup2_334315952Core.xml
05:50:25:WU01:FS01:0x17:Saving result file badStateForceGroup2_334315952Ref.xml
05:50:28:WU01:FS01:0x17:Saving result file log.txt
05:50:28:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
05:50:29:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
05:50:29:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9101 run:982 clone:0 gen:22 core:0x17 unit:0x000000160a3b1e81533f88f6b5eb27a3
05:50:29:WU01:FS01:Uploading 7.17MiB to 171.64.65.93
05:50:29:WU01:FS01:Connecting to 171.64.65.93:8080
05:50:35:WU01:FS01:Upload 25.29%
05:50:41:WU01:FS01:Upload 53.19%
05:50:47:WU01:FS01:Upload 80.22%
05:50:52:WU01:FS01:Upload complete
05:50:52:WU01:FS01:Server responded WORK_ACK (400)
05:50:52:WU01:FS01:Cleaning up
05:54:23:WU02:FS00:0xa5:Completed 42500 out of 250000 steps  (17%)
Top
P5-133XL
Posts: 2948
Joined: Sun Dec 02, 2007 4:36 am
Hardware configuration: Machine #1:

Intel Q9450; 2x2GB=8GB Ram; Gigabyte GA-X48-DS4 Motherboard; PC Power and Cooling Q750 PS; 2x GTX 460; Windows Server 2008 X64 (SP1).

Machine #2:

Intel Q6600; 2x2GB=4GB Ram; Gigabyte GA-X48-DS4 Motherboard; PC Power and Cooling Q750 PS; 2x GTX 460 video card; Windows 7 X64.

Machine 3:

Dell Dimension 8400, 3.2GHz P4 4x512GB Ram, Video card GTX 460, Windows 7 X32

I am currently folding just on the 5x GTX 460's for aprox. 70K PPD
Location: Salem. OR USA

Re: Failed to get assignment from '171.67.108.200'...

Post by P5-133XL »

No it is not normal for you to fail all p9101's but not other Core_17 projects.
Image
Top
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Failed to get assignment from '171.67.108.200'...

Post by bruce »

TheBlademaster01 wrote:Sorry, but on the matter of the GPU. Is it normal that about all Project 9101 units fail on my GPU when I've completed the others?

Code: Select all

00:32:39:WU01:FS01:0x17:*********************** Log Started 2014-04-23T00:32:39Z ***********************
00:32:39:WU01:FS01:0x17:Project: 9101 (Run 632, Clone 0, Gen 47)
. . .
00:40:15:WU01:FS01:0x17:ERROR:exception: The periodic box size has decreased to less than twice the nonbonded cutoff.
00:40:15:WU01:FS01:0x17:Saving result file logfile_01.txt
00:40:15:WU01:FS01:0x17:Saving result file log.txt
00:40:15:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
. . .
00:40:16:WU01:FS01:Server responded WORK_ACK (400)
That WU failed and you earned zero points. It was reassigned and the next assignee completed it successfully. It's highly likely that your GPU is unstable and unable to calculate that WU although others can.
Top
TheBlademaster01
Posts: 39
Joined: Sat Jun 05, 2010 11:10 am

Re: Failed to get assignment from '171.67.108.200'...

Post by TheBlademaster01 »

Okay. Could it be a conflict with Linux? My GPU is pretty much at factory settings.

In any case, I gave up. I figured I am hurting the project more than helping it by folding on this system.
Top
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Failed to get assignment from '171.67.108.200'...

Post by bruce »

The default Linux drivers don't fold. You have to find the proprietary drivers and install them.
Top
TheBlademaster01
Posts: 39
Joined: Sat Jun 05, 2010 11:10 am

Re: Failed to get assignment from '171.67.108.200'...

Post by TheBlademaster01 »

This was done with proprietary drivers. I had FW 331.49 installed. Which can also be seen by the completed projects 9408 and 9406. Only 9101 fails consistently.
Top
PantherX
Site Moderator
Posts: 7020
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:
Contact PantherX

Re: Failed to get assignment from '171.67.108.200'...

Post by PantherX »

A quick search (search.php?keywords=Ubuntu&t=26163&sf=msgonly) shows that Ubuntu was tested with GTX 780 Ti but the driver version for at least one tester was 304.xx while you are using 331.49. Maybe you can change the driver and see if that resolves your issue or not.
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Top
Post Reply

Return to “Issues with a specific server”