Project:9011 run:145 clone:2 gen:75

davidcoton · Post by **davidcoton** » Fri Jul 24, 2015 11:19 pm

*********************** Log Started 2015-07-24T22:56:11Z ***********************
22:56:11:************************* Folding@home Client *************************
22:56:11:      Website: http://folding.stanford.edu/
22:56:11:    Copyright: (c) 2009-2014 Stanford University
22:56:11:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:56:11:         Args: 
22:56:11:       Config: C:/Users/David/AppData/Roaming/FAHClient/config.xml
22:56:11:******************************** Build ********************************
22:56:11:      Version: 7.4.4
22:56:11:         Date: Mar 4 2014
22:56:11:         Time: 20:26:54
22:56:11:      SVN Rev: 4130
22:56:11:       Branch: fah/trunk/client
22:56:11:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
22:56:11:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
22:56:11:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
22:56:11:     Platform: win32 XP
22:56:11:         Bits: 32
22:56:11:         Mode: Release
22:56:11:******************************* System ********************************
22:56:11:          CPU: AMD Athlon(tm) II X4 640 Processor
22:56:11:       CPU ID: AuthenticAMD Family 16 Model 5 Stepping 3
22:56:11:         CPUs: 4
22:56:11:       Memory: 3.12GiB
22:56:11:  Free Memory: 1.74GiB
22:56:11:      Threads: WINDOWS_THREADS
22:56:11:   OS Version: 6.0
22:56:11:  Has Battery: false
22:56:11:   On Battery: false
22:56:11:   UTC Offset: 1
22:56:11:          PID: 5900
22:56:11:          CWD: C:/Users/David/AppData/Roaming/FAHClient
22:56:11:           OS: Windows Vista (TM) Home Premium Service Pack 2
22:56:11:      OS Arch: X86
22:56:11:         GPUs: 1
22:56:11:        GPU 0: NVIDIA:5 GM204 [GeForce GTX 980]
22:56:11:         CUDA: 5.2
22:56:11:  CUDA Driver: 7050
22:56:11:Win32 Service: false
22:56:11:***********************************************************************
22:56:11:<config>
22:56:11:  <!-- Folding Core -->
22:56:11:  <checkpoint v='5'/>
22:56:11:
22:56:11:  <!-- HTTP Server -->
22:56:11:  <allow v='127.0.0.1 192.168.1.0/24'/>
22:56:11:  <deny v='0.0.0.0/0'/>
22:56:11:  <http-addresses v='127.0.0.1:7396 david-ubuntu:7396'/>
22:56:11:
22:56:11:  <!-- Network -->
22:56:11:  <proxy v=':8080'/>
22:56:11:
22:56:11:  <!-- Remote Command Server -->
22:56:11:  <password v='*******'/>
22:56:11:
22:56:11:  <!-- Slot Control -->
22:56:11:  <power v='full'/>
22:56:11:
22:56:11:  <!-- User Information -->
22:56:11:  <passkey v='********************************'/>
22:56:11:  <user v='davidcoton'/>
22:56:11:
22:56:11:  <!-- Web Server -->
22:56:11:  <web-allow v='127.0.0.1 168.192.1.0/24'/>
22:56:11:
22:56:11:  <!-- Folding Slots -->
22:56:11:  <slot id='0' type='CPU'>
22:56:11:    <client-type v='advanced'/>
22:56:11:    <cpus v='3'/>
22:56:11:    <paused v='true'/>
22:56:11:  </slot>
22:56:11:  <slot id='1' type='GPU'>
22:56:11:    <client-type v='advanced'/>
22:56:11:  </slot>
22:56:11:</config>
22:56:11:Trying to access database...
22:56:16:Successfully acquired database lock
22:56:16:Enabled folding slot 00: PAUSED cpu:3 (by user)
22:56:16:Enabled folding slot 01: READY gpu:0:GM204 [GeForce GTX 980]
22:56:17:ERROR:Exception: Could not bind socket to david-ubuntu:7396: The requested address is not valid in its context.
22:56:19:WU01:FS01:Starting
...
22:59:24:WU00:FS00:Started FahCore on PID 7944
22:59:27:WU00:FS00:Core PID:8100
22:59:27:WU00:FS00:FahCore 0xa4 started
22:59:28:WU00:FS00:0xa4:
22:59:28:WU00:FS00:0xa4:*------------------------------*
22:59:28:WU00:FS00:0xa4:Folding@Home Gromacs GB Core
22:59:28:WU00:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
22:59:28:WU00:FS00:0xa4:
22:59:28:WU00:FS00:0xa4:Preparing to commence simulation
22:59:28:WU00:FS00:0xa4:- Ensuring status. Please wait.
22:59:30:Removing old file 'configs/config-20150601-230747.xml'
22:59:30:Saving configuration to config.xml
22:59:30:<config>
22:59:30:  <!-- Folding Core -->
22:59:30:  <checkpoint v='5'/>
22:59:30:
22:59:30:  <!-- HTTP Server -->
22:59:30:  <allow v='127.0.0.1 192.168.1.0/24'/>
22:59:30:  <deny v='0.0.0.0/0'/>
22:59:30:  <http-addresses v='127.0.0.1:7396 david-ubuntu:7396'/>
22:59:30:
22:59:30:  <!-- Network -->
22:59:30:  <proxy v=':8080'/>
22:59:30:
22:59:30:  <!-- Remote Command Server -->
22:59:30:  <password v='*******'/>
22:59:30:
22:59:30:  <!-- Slot Control -->
22:59:30:  <power v='full'/>
22:59:30:
22:59:30:  <!-- User Information -->
22:59:30:  <passkey v='********************************'/>
22:59:30:  <user v='davidcoton'/>
22:59:30:
22:59:30:  <!-- Web Server -->
22:59:30:  <web-allow v='127.0.0.1 168.192.1.0/24'/>
22:59:30:
22:59:30:  <!-- Folding Slots -->
22:59:30:  <slot id='0' type='CPU'>
22:59:30:    <client-type v='advanced'/>
22:59:30:    <cpus v='3'/>
22:59:30:  </slot>
22:59:30:  <slot id='1' type='GPU'>
22:59:30:    <client-type v='advanced'/>
22:59:30:  </slot>
22:59:30:</config>
22:59:35:ERROR:FS00:
22:59:35:ERROR:FS00:-------------------------------------------------------
22:59:35:ERROR:FS00:Program Folding@home, VERSION 4.5.4
22:59:35:ERROR:FS00:Source code file: gromacs-4.5.4\src\gmxlib\smalloc.c, line: 171
22:59:35:ERROR:FS00:
22:59:35:ERROR:FS00:Fatal error:
22:59:35:ERROR:FS00:Not enough memory. Failed to calloc ld elements of size ld for 
22:59:35:ERROR:FS00:(called from file (null), line 12)
22:59:35:ERROR:FS00:For more information and tips for troubleshooting, please check the GROMACS
22:59:35:ERROR:FS00:website at http://www.gromacs.org/Documentation/Errors
22:59:35:ERROR:FS00:-------------------------------------------------------
22:59:35:ERROR:FS00:
22:59:35:ERROR:FS00:Thanx for Using GROMACS - Have a Nice Day
22:59:35:ERROR:FS00:
22:59:35:ERROR:FS00:-------------------------------------------------------
22:59:35:ERROR:FS00:Program Folding@home, VERSION 4.5.4
22:59:35:ERROR:FS00:Source code file: gromacs-4.5.4\src\gmxlib\smalloc.c, line: 171
22:59:35:ERROR:FS00:
22:59:35:ERROR:FS00:Fatal error:
22:59:35:ERROR:FS00:Not enough memory. Failed to calloc ld elements of size ld for 
22:59:35:ERROR:FS00:(called from file (null), line 1)
22:59:35:ERROR:FS00:For more information and tips for troubleshooting, please check the GROMACS
22:59:35:ERROR:FS00:website at http://www.gromacs.org/Documentation/Errors
22:59:35:ERROR:FS00:-------------------------------------------------------
22:59:35:ERROR:FS00:
22:59:35:ERROR:FS00:Thanx for Using GROMACS - Have a Nice Day
22:59:37:WU00:FS00:0xa4:- Looking at optimizations...
22:59:37:WU00:FS00:0xa4:- Working with standard loops on this execution.
22:59:37:WU00:FS00:0xa4:- Previous termination of core was improper.
22:59:37:WU00:FS00:0xa4:- Going to use standard loops.
22:59:37:WU00:FS00:0xa4:- Files status OK
22:59:40:WU00:FS00:0xa4:- Expanded 304829 -> 729088 (decompressed 239.1 percent)
22:59:40:WU00:FS00:0xa4:Called DecompressByteArray: compressed_data_size=304829 data_size=729088, decompressed_data_size=729088 diff=0
22:59:40:WU00:FS00:0xa4:- Digital signature verified
22:59:40:WU00:FS00:0xa4:
22:59:40:WU00:FS00:0xa4:Project: 9011 (Run 145, Clone 2, Gen 75)
22:59:40:WU00:FS00:0xa4:
22:59:41:WU00:FS00:0xa4:Entering M.D.
22:59:47:WU00:FS00:0xa4:Mapping NT from 3 to 3 
23:00:26:WARNING:WU00:FS00:FahCore returned an unknown error code which probably indicates that it crashed
23:00:26:WARNING:WU00:FS00:FahCore returned: UNKNOWN_ENUM (-1073740777 = 0xc0000417)
23:00:28:WU00:FS00:Starting
23:00:28:WU00:FS00:Running FahCore: "C:\Program Files\FAHClient/FAHCoreWrapper.exe" C:/Users/David/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/x86/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 704 -lifeline 5900 -checkpoint 5 -np 3
23:00:28:WU00:FS00:Started FahCore on PID 7008
23:00:28:WU00:FS00:Core PID:7932
23:00:28:WU00:FS00:FahCore 0xa4 started
23:00:28:WU00:FS00:0xa4:
23:00:28:WU00:FS00:0xa4:*------------------------------*
23:00:28:WU00:FS00:0xa4:Folding@Home Gromacs GB Core
23:00:28:WU00:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
23:00:28:WU00:FS00:0xa4:
23:00:28:WU00:FS00:0xa4:Preparing to commence simulation
23:00:28:WU00:FS00:0xa4:- Ensuring status. Please wait.
23:00:29:WU01:FS01:0x18:Completed 1925000 out of 2500000 steps (77%)
23:00:37:WU00:FS00:0xa4:- Looking at optimizations...
23:00:37:WU00:FS00:0xa4:- Working with standard loops on this execution.
23:00:37:WU00:FS00:0xa4:- Previous termination of core was improper.
23:00:37:WU00:FS00:0xa4:- Going to use standard loops.
23:00:37:WU00:FS00:0xa4:- Files status OK
23:00:39:WU00:FS00:0xa4:- Expanded 304829 -> 729088 (decompressed 239.1 percent)
23:00:39:WU00:FS00:0xa4:Called DecompressByteArray: compressed_data_size=304829 data_size=729088, decompressed_data_size=729088 diff=0
23:00:39:WU00:FS00:0xa4:- Digital signature verified
23:00:39:WU00:FS00:0xa4:
23:00:39:WU00:FS00:0xa4:Project: 9011 (Run 145, Clone 2, Gen 75)
23:00:39:WU00:FS00:0xa4:
23:00:39:WU00:FS00:0xa4:Entering M.D.
23:00:45:WU00:FS00:0xa4:Mapping NT from 3 to 3 
23:00:47:WARNING:WU00:FS00:FahCore returned an unknown error code which probably indicates that it crashed
23:00:47:WARNING:WU00:FS00:FahCore returned: UNKNOWN_ENUM (-1073740777 = 0xc0000417)
23:00:47:WARNING:WU00:FS00:Too many errors, failing
23:00:48:WU00:FS00:Sending unit results: id:00 state:SEND error:FAILED project:9011 run:145 clone:2 gen:75 core:0xa4 unit:0x0000005fab40417c554e934113e639ba
23:00:49:WU00:FS00:Connecting to 171.64.65.124:8080
23:00:49:WU02:FS00:Connecting to 171.67.108.200:8080
23:00:51:WU00:FS00:Server responded PLEASE_WAIT (464)
23:00:51:WARNING:WU00:FS00:Failed to send results, will try again later
23:00:52:WU00:FS00:Sending unit results: id:00 state:SEND error:FAILED project:9011 run:145 clone:2 gen:75 core:0xa4 unit:0x0000005fab40417c554e934113e639ba
23:00:53:WU00:FS00:Connecting to 171.64.65.124:8080
23:00:53:WU02:FS00:Assigned to work server 155.247.166.220
23:00:53:WU02:FS00:Requesting new work unit for slot 00: READY cpu:3 from 155.247.166.220
23:00:53:WU02:FS00:Connecting to 155.247.166.220:8080
23:00:57:FS00:Paused
23:00:57:WU02:FS00:Downloading 199.83KiB
23:00:57:WU00:FS00:Server responded PLEASE_WAIT (464)
23:00:57:WARNING:WU00:FS00:Failed to send results, will try again later
23:00:57:WU02:FS00:Download complete
23:00:59:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:6386 run:36 clone:1 gen:146 core:0xa4 unit:0x000000970002894c5417558f2c1b8bb0
23:01:19:FS00:Unpaused
23:01:19:WU02:FS00:Starting
23:01:19:WU02:FS00:Running FahCore: "C:\Program Files\FAHClient/FAHCoreWrapper.exe" C:/Users/David/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/x86/Core_a4.fah/FahCore_a4.exe -dir 02 -suffix 01 -version 704 -lifeline 5900 -checkpoint 5 -np 3
23:01:19:WU02:FS00:Started FahCore on PID 7288
23:01:22:WU02:FS00:Core PID:7876
23:01:22:WU02:FS00:FahCore 0xa4 started
23:01:22:WU02:FS00:0xa4:
23:01:22:WU02:FS00:0xa4:*------------------------------*
23:01:22:WU02:FS00:0xa4:Folding@Home Gromacs GB Core
23:01:22:WU02:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
23:01:22:WU02:FS00:0xa4:
23:01:22:WU02:FS00:0xa4:Preparing to commence simulation
23:01:22:WU02:FS00:0xa4:- Looking at optimizations...
23:01:22:WU02:FS00:0xa4:- Created dyn
23:01:22:WU02:FS00:0xa4:- Files status OK
23:01:22:WU02:FS00:0xa4:- Expanded 204113 -> 430180 (decompressed 210.7 percent)
23:01:22:WU02:FS00:0xa4:Called DecompressByteArray: compressed_data_size=204113 data_size=430180, decompressed_data_size=430180 diff=0
23:01:22:WU02:FS00:0xa4:- Digital signature verified
23:01:22:WU02:FS00:0xa4:
23:01:22:WU02:FS00:0xa4:Project: 6386 (Run 36, Clone 1, Gen 146)
23:01:22:WU02:FS00:0xa4:
23:01:23:WU02:FS00:0xa4:Assembly optimizations on if available.
23:01:23:WU02:FS00:0xa4:Entering M.D.
23:01:29:WU02:FS00:0xa4:Mapping NT from 3 to 3 
23:01:34:WU02:FS00:0xa4:Completed 0 out of 2500000 steps  (0%)
23:01:52:WU00:FS00:Sending unit results: id:00 state:SEND error:FAILED project:9011 run:145 clone:2 gen:75 core:0xa4 unit:0x0000005fab40417c554e934113e639ba
23:01:53:WU00:FS00:Connecting to 171.64.65.124:8080
23:01:54:WU00:FS00:Server responded PLEASE_WAIT (464)
23:01:54:WARNING:WU00:FS00:Failed to send results, will try again later
23:02:09:WU01:FS01:0x18:Completed 1950000 out of 2500000 steps (78%)
23:03:29:WU00:FS00:Sending unit results: id:00 state:SEND error:FAILED project:9011 run:145 clone:2 gen:75 core:0xa4 unit:0x0000005fab40417c554e934113e639ba
23:03:30:WU00:FS00:Connecting to 171.64.65.124:8080
23:03:31:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
23:03:31:WU00:FS00:Connecting to 171.64.65.124:80
23:03:32:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to connect to 171.64.65.124:80: No connection could be made because the target machine actively refused it.
23:03:32:WU00:FS00:Trying to send results to collection server
23:03:33:WU00:FS00:Connecting to 171.65.103.160:8080
23:03:33:WU00:FS00:Server responded WORK_QUIT (404)
23:03:33:WARNING:WU00:FS00:Server did not like results, dumping
23:03:34:WU00:FS00:Cleaning up
23:03:34:ERROR:WU00:FS00:Exception: Failed to remove directory './work/00': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\00\wudata_01.tpr"
23:03:34:WU00:FS00:Cleaning up
23:03:34:ERROR:WU00:FS00:Exception: Failed to remove directory './work/00': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\00\wudata_01.tpr"
23:03:48:WU01:FS01:0x18:Completed 1975000 out of 2500000 steps (79%)
23:04:34:WU00:FS00:Cleaning up
23:04:34:ERROR:WU00:FS00:Exception: Failed to remove directory './work/00': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\00\wudata_01.tpr"

Main(?) issue was a Gromacs mem allocation error. Not solved by a reboot. There was also an upload problem, followed by a cleanup problem.

Post by **sryckbos** » Fri Jul 24, 2015 11:31 pm

See viewtopic.php?f=18&t=27887.

This problem should be fixed as of a couple hours ago or so. If this was from before then then it should be fixed.

Still I'm looking into it to make sure my fix is being implemented properly.

Thanks for the info!
Steven

davidcoton · Post by **davidcoton** » Sat Jul 25, 2015 2:32 pm

The error was live when I posted (ie, not fixed). That is only the second part of three separate (but connected) problems with the WU.
First, a mem alloc error in Gromacs -- not fixed by rebooting the PC, and not seen on other WUs.
Second, the upload error, similar to the linked thread but occurring after the supposed fix.
Third, the FAH system was unable to clean up because it held a file open (fixed by closing FAH completely -- I'm 97% certain it was FAH itself locking the file).

Ricky · Post by **Ricky** » Mon Aug 10, 2015 5:23 pm

I have "cleanup" status on the following WU. What should I do?
9752 (388,0,344) 12 attempts Collection server 172.65.103.160
9752 (430,0,332) 11 attempts Collection server 172.65.103.160
9752 (1672,0,150) 11 attempts Collection server 172.65.103.160
7520 (34,32,64) 6 attempts Collection server 0.0.0.0

Post by **Joe_H** » Mon Aug 10, 2015 10:57 pm

Continual retries at returning WU's often points to an issue with your folding client not getting acknowledgement responses back from the server after completing an upload, or the connection timing out before completing an upload. You will need to post your log file to show enough information to determine what is going on. See this topic for directions - viewtopic.php?f=61&t=26036.

As for showing the Collection Server information, that is only useful if an upload fails to the originating Work Server. An entry of 0.0.0.0 means there is no CS for that project, uploads return to the WS they came from.

Post by **bruce** » Mon Aug 10, 2015 11:08 pm

...uploads return to the WS they came from.

Actually, uploads always return to the WS they came from even if you don't upload them to it. (If you upload to a CS, it just forwards it to the WS over a more reliable path.)

Why isn't your WU being returned directly to the Work Server 171.64.65.124 and how often did THAT fail?

Ricky · Post by **Ricky** » Mon Aug 10, 2015 11:51 pm

I paused, then restarted windows 8.1, the "cleanup" queue was cleared. Folding seemed to resume after with no new problems.
Here is the log:

Code: Select all

20:14:54:WU06:FS00:0xa4:Completed 495000 out of 500000 steps  (99%)
20:14:54:WU01:FS00:Connecting to 171.67.108.200:8080
20:14:55:WU01:FS00:Assigned to work server 171.64.65.99
20:14:55:WU01:FS00:Requesting new work unit for slot 00: RUNNING cpu:30 from 171.64.65.99
20:14:55:WU01:FS00:Connecting to 171.64.65.99:8080
20:14:55:WU00:FS02:0x17:Completed 15200000 out of 16000000 steps (95%)
20:14:58:WU01:FS00:Downloading 6.23MiB
20:15:00:WU01:FS00:Download complete
20:15:00:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9752 run:1457 clone:0 gen:168 core:0xa4 unit:0x000000e5ab4041635541739d694b9db6
20:16:49:WU06:FS00:0xa4:Completed 500000 out of 500000 steps  (100%)
20:16:51:WU06:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
20:17:01:WU06:FS00:0xa4:
20:17:01:WU06:FS00:0xa4:Finished Work Unit:
20:17:01:WU06:FS00:0xa4:- Reading up to 6827976 from "06/wudata_01.trr": Read 6827976
20:17:01:WU06:FS00:0xa4:trr file hash check passed.
20:17:01:WU06:FS00:0xa4:- Reading up to 7402364 from "06/wudata_01.xtc": Read 7402364
20:17:01:WU06:FS00:0xa4:xtc file hash check passed.
20:17:01:WU06:FS00:0xa4:edr file hash check passed.
20:17:01:WU06:FS00:0xa4:logfile size: 356255
20:17:01:WU06:FS00:0xa4:Leaving Run
20:17:01:WU06:FS00:0xa4:- Writing 14619623 bytes of core data to disk...
20:17:04:WU06:FS00:0xa4:Done: 14619111 -> 13916404 (compressed to 95.1 percent)
20:17:04:WU06:FS00:0xa4:  ... Done.
20:17:04:WU06:FS00:0xa4:- Shutting down core
20:17:04:WU06:FS00:0xa4:
20:17:04:WU06:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
20:17:05:WU06:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
20:17:05:WU06:FS00:Sending unit results: id:06 state:SEND error:NO_ERROR project:7520 run:38 clone:16 gen:31 core:0xa4 unit:0x0000002bfbcb017d54ebbafe82325448
20:17:05:WU06:FS00:Uploading 13.27MiB to 128.143.199.97
20:17:05:WU06:FS00:Connecting to 128.143.199.97:8080
20:17:05:WU01:FS00:Starting
20:17:05:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 704 -lifeline 5860 -checkpoint 15 -np 30
20:17:05:WU01:FS00:Started FahCore on PID 4392
20:17:05:WU01:FS00:Core PID:5472
20:17:05:WU01:FS00:FahCore 0xa4 started
20:17:06:WU01:FS00:0xa4:
20:17:06:WU01:FS00:0xa4:*------------------------------*
20:17:06:WU01:FS00:0xa4:Folding@Home Gromacs GB Core
20:17:06:WU01:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
20:17:06:WU01:FS00:0xa4:
20:17:06:WU01:FS00:0xa4:Preparing to commence simulation
20:17:06:WU01:FS00:0xa4:- Looking at optimizations...
20:17:06:WU01:FS00:0xa4:- Created dyn
20:17:06:WU01:FS00:0xa4:- Files status OK
20:17:08:WU01:FS00:0xa4:- Expanded 6536919 -> 22431316 (decompressed 343.1 percent)
20:17:08:WU01:FS00:0xa4:Called DecompressByteArray: compressed_data_size=6536919 data_size=22431316, decompressed_data_size=22431316 diff=0
20:17:08:WU01:FS00:0xa4:- Digital signature verified
20:17:08:WU01:FS00:0xa4:
20:17:08:WU01:FS00:0xa4:Project: 9752 (Run 1457, Clone 0, Gen 168)
20:17:08:WU01:FS00:0xa4:
20:17:08:WU01:FS00:0xa4:Assembly optimizations on if available.
20:17:08:WU01:FS00:0xa4:Entering M.D.
20:17:11:WU06:FS00:Upload 5.18%
20:17:14:WU01:FS00:0xa4:Mapping NT from 30 to 30 
20:17:16:WU01:FS00:0xa4:Completed 0 out of 80000 steps  (0%)
20:17:17:WU06:FS00:Upload 9.89%
20:17:23:WU06:FS00:Upload 15.07%
20:17:28:WU07:FS01:0x18:Completed 3550000 out of 5000000 steps (71%)
20:17:29:WU06:FS00:Upload 25.43%
20:17:35:WU06:FS00:Upload 33.43%
20:17:41:WU06:FS00:Upload 48.03%
20:17:47:WU06:FS00:Upload 55.57%
20:17:50:WU01:FS00:0xa4:Completed 800 out of 80000 steps  (1%)
20:17:53:WU06:FS00:Upload 64.99%
20:17:59:WU06:FS00:Upload 77.23%
20:18:07:WU06:FS00:Upload 80.53%
20:18:13:WU06:FS00:Upload 83.35%
20:18:19:WU06:FS00:Upload 87.59%
20:18:24:WU01:FS00:0xa4:Completed 1600 out of 80000 steps  (2%)
20:18:25:WU06:FS00:Upload 91.36%
20:18:31:WU06:FS00:Upload 94.65%
20:18:37:WU06:FS00:Upload 98.42%
20:18:44:WU06:FS00:Upload complete
20:18:44:WU06:FS00:Server responded WORK_ACK (400)
20:18:44:WU06:FS00:Final credit estimate, 10582.00 points
20:18:44:WU06:FS00:Cleaning up
20:18:57:WU01:FS00:0xa4:Completed 2400 out of 80000 steps  (3%)
20:19:31:WU01:FS00:0xa4:Completed 3200 out of 80000 steps  (4%)
20:19:54:WU07:FS01:0x18:Completed 3600000 out of 5000000 steps (72%)
20:20:04:WU01:FS00:0xa4:Completed 4000 out of 80000 steps  (5%)
20:20:37:WU01:FS00:0xa4:Completed 4800 out of 80000 steps  (6%)
20:21:10:WU01:FS00:0xa4:Completed 5600 out of 80000 steps  (7%)
20:21:30:WU00:FS02:0x17:Completed 15360000 out of 16000000 steps (96%)
20:21:43:WU01:FS00:0xa4:Completed 6400 out of 80000 steps  (8%)
20:22:16:WU01:FS00:0xa4:Completed 7200 out of 80000 steps  (9%)
20:22:28:WU07:FS01:0x18:Completed 3650000 out of 5000000 steps (73%)
20:22:49:WU01:FS00:0xa4:Completed 8000 out of 80000 steps  (10%)
20:23:22:WU01:FS00:0xa4:Completed 8800 out of 80000 steps  (11%)
20:23:55:WU01:FS00:0xa4:Completed 9600 out of 80000 steps  (12%)
20:24:28:WU01:FS00:0xa4:Completed 10400 out of 80000 steps  (13%)
20:24:53:WU07:FS01:0x18:Completed 3700000 out of 5000000 steps (74%)
20:25:01:WU01:FS00:0xa4:Completed 11200 out of 80000 steps  (14%)
20:25:34:WU01:FS00:0xa4:Completed 12000 out of 80000 steps  (15%)
20:26:06:WU01:FS00:0xa4:Completed 12800 out of 80000 steps  (16%)
20:26:39:WU01:FS00:0xa4:Completed 13600 out of 80000 steps  (17%)
20:27:12:WU01:FS00:0xa4:Completed 14400 out of 80000 steps  (18%)
20:27:18:WU07:FS01:0x18:Completed 3750000 out of 5000000 steps (75%)
20:27:45:WU01:FS00:0xa4:Completed 15200 out of 80000 steps  (19%)
20:28:04:WU00:FS02:0x17:Completed 15520000 out of 16000000 steps (97%)
20:28:18:WU01:FS00:0xa4:Completed 16000 out of 80000 steps  (20%)
20:28:50:WU01:FS00:0xa4:Completed 16800 out of 80000 steps  (21%)
20:29:23:WU01:FS00:0xa4:Completed 17600 out of 80000 steps  (22%)
20:29:53:WU07:FS01:0x18:Completed 3800000 out of 5000000 steps (76%)
20:29:56:WU01:FS00:0xa4:Completed 18400 out of 80000 steps  (23%)
20:30:29:WU01:FS00:0xa4:Completed 19200 out of 80000 steps  (24%)
20:31:02:WU01:FS00:0xa4:Completed 20000 out of 80000 steps  (25%)
20:31:36:WU01:FS00:0xa4:Completed 20800 out of 80000 steps  (26%)
20:32:09:WU01:FS00:0xa4:Completed 21600 out of 80000 steps  (27%)
20:32:18:WU07:FS01:0x18:Completed 3850000 out of 5000000 steps (77%)
20:32:42:WU01:FS00:0xa4:Completed 22400 out of 80000 steps  (28%)
20:33:15:WU01:FS00:0xa4:Completed 23200 out of 80000 steps  (29%)
20:33:48:WU01:FS00:0xa4:Completed 24000 out of 80000 steps  (30%)
20:34:20:WU01:FS00:0xa4:Completed 24800 out of 80000 steps  (31%)
20:34:37:WU00:FS02:0x17:Completed 15680000 out of 16000000 steps (98%)
20:34:53:WU01:FS00:0xa4:Completed 25600 out of 80000 steps  (32%)
20:34:53:WU07:FS01:0x18:Completed 3900000 out of 5000000 steps (78%)
20:35:25:WU01:FS00:0xa4:Completed 26400 out of 80000 steps  (33%)
20:35:58:WU01:FS00:0xa4:Completed 27200 out of 80000 steps  (34%)
20:36:31:WU01:FS00:0xa4:Completed 28000 out of 80000 steps  (35%)
20:37:04:WU01:FS00:0xa4:Completed 28800 out of 80000 steps  (36%)
20:37:18:WU07:FS01:0x18:Completed 3950000 out of 5000000 steps (79%)
20:37:37:WU01:FS00:0xa4:Completed 29600 out of 80000 steps  (37%)
20:38:10:WU01:FS00:0xa4:Completed 30400 out of 80000 steps  (38%)
20:38:42:WU01:FS00:0xa4:Completed 31200 out of 80000 steps  (39%)
20:39:15:WU01:FS00:0xa4:Completed 32000 out of 80000 steps  (40%)
20:39:43:WU07:FS01:0x18:Completed 4000000 out of 5000000 steps (80%)
20:39:48:WU01:FS00:0xa4:Completed 32800 out of 80000 steps  (41%)
20:40:21:WU01:FS00:0xa4:Completed 33600 out of 80000 steps  (42%)
20:40:54:WU01:FS00:0xa4:Completed 34400 out of 80000 steps  (43%)
20:41:11:WU00:FS02:0x17:Completed 15840000 out of 16000000 steps (99%)
20:41:11:WU02:FS02:Connecting to 171.67.108.200:80
20:41:12:WU02:FS02:Assigned to work server 171.64.65.93
20:41:12:WU02:FS02:Requesting new work unit for slot 02: RUNNING gpu:1:GM206 [GeForce GTX 960] from 171.64.65.93
20:41:12:WU02:FS02:Connecting to 171.64.65.93:8080
20:41:12:WU02:FS02:Downloading 3.53MiB
20:41:15:WU02:FS02:Download complete
20:41:15:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:9126 run:1 clone:8 gen:73 core:0x18 unit:0x000000570a3b1e815549358543485c57
20:41:27:WU01:FS00:0xa4:Completed 35200 out of 80000 steps  (44%)
20:42:01:WU01:FS00:0xa4:Completed 36000 out of 80000 steps  (45%)
20:42:19:WU07:FS01:0x18:Completed 4050000 out of 5000000 steps (81%)
20:42:34:WU01:FS00:0xa4:Completed 36800 out of 80000 steps  (46%)
20:43:06:WU01:FS00:0xa4:Completed 37600 out of 80000 steps  (47%)
20:43:39:WU01:FS00:0xa4:Completed 38400 out of 80000 steps  (48%)
20:44:11:WU01:FS00:0xa4:Completed 39200 out of 80000 steps  (49%)
20:44:44:WU07:FS01:0x18:Completed 4100000 out of 5000000 steps (82%)
20:44:44:WU01:FS00:0xa4:Completed 40000 out of 80000 steps  (50%)
20:45:17:WU01:FS00:0xa4:Completed 40800 out of 80000 steps  (51%)
20:45:49:WU01:FS00:0xa4:Completed 41600 out of 80000 steps  (52%)
20:46:22:WU01:FS00:0xa4:Completed 42400 out of 80000 steps  (53%)
20:46:56:WU01:FS00:0xa4:Completed 43200 out of 80000 steps  (54%)
20:47:19:WU07:FS01:0x18:Completed 4150000 out of 5000000 steps (83%)
20:47:29:WU01:FS00:0xa4:Completed 44000 out of 80000 steps  (55%)
20:47:43:WU00:FS02:0x17:Completed 16000000 out of 16000000 steps (100%)
20:47:49:WU00:FS02:0x17:Saving result file logfile_01.txt
20:47:49:WU00:FS02:0x17:Saving result file checkpointState.xml
20:47:49:WU00:FS02:0x17:Saving result file checkpt.crc
20:47:49:WU00:FS02:0x17:Saving result file log.txt
20:47:49:WU00:FS02:0x17:Saving result file positions.xtc
20:47:55:WU00:FS02:0x17:Folding@home Core Shutdown: FINISHED_UNIT
20:47:55:WU00:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
20:47:55:WU00:FS02:Sending unit results: id:00 state:SEND error:NO_ERROR project:9411 run:358 clone:3 gen:0 core:0x17 unit:0x00000002ab4041385548205fe5bc7f33
20:47:55:WU00:FS02:Uploading 13.65MiB to 171.64.65.56
20:47:55:WU00:FS02:Connecting to 171.64.65.56:8080
20:47:55:WU02:FS02:Starting
20:47:55:WU02:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 02 -suffix 01 -version 704 -lifeline 5860 -checkpoint 15 -gpu 1 -gpu-vendor nvidia
20:47:55:WU02:FS02:Started FahCore on PID 644
20:47:55:WU02:FS02:Core PID:5392
20:47:55:WU02:FS02:FahCore 0x18 started
20:47:56:WU02:FS02:0x18:*********************** Log Started 2015-08-10T20:47:56Z ***********************
20:47:56:WU02:FS02:0x18:Project: 9126 (Run 1, Clone 8, Gen 73)
20:47:56:WU02:FS02:0x18:Unit: 0x000000570a3b1e815549358543485c57
20:47:56:WU02:FS02:0x18:CPU: 0x00000000000000000000000000000000
20:47:56:WU02:FS02:0x18:Machine: 2
20:47:56:WU02:FS02:0x18:Reading tar file state.xml
20:47:57:WU02:FS02:0x18:Reading tar file system.xml
20:47:58:WU02:FS02:0x18:Reading tar file integrator.xml
20:47:58:WU02:FS02:0x18:Reading tar file core.xml
20:47:58:WU02:FS02:0x18:Digital signatures verified
20:47:58:WU02:FS02:0x18:Folding@home GPU core18
20:47:58:WU02:FS02:0x18:Version 0.0.4
20:48:01:WU00:FS02:Upload 56.32%
20:48:04:WU01:FS00:0xa4:Completed 44800 out of 80000 steps  (56%)
20:48:09:WU00:FS02:Upload complete
20:48:09:WU00:FS02:Server responded WORK_ACK (400)
20:48:09:WU00:FS02:Final credit estimate, 67609.00 points
20:48:09:WU00:FS02:Cleaning up
20:48:38:WU01:FS00:0xa4:Completed 45600 out of 80000 steps  (57%)
20:48:40:WU02:FS02:0x18:Completed 0 out of 2500000 steps (0%)
20:48:40:WU02:FS02:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
20:49:12:WU01:FS00:0xa4:Completed 46400 out of 80000 steps  (58%)
20:49:42:WU07:FS01:0x18:Completed 4200000 out of 5000000 steps (84%)
20:49:46:WU01:FS00:0xa4:Completed 47200 out of 80000 steps  (59%)
20:50:21:WU01:FS00:0xa4:Completed 48000 out of 80000 steps  (60%)
20:50:56:WU01:FS00:0xa4:Completed 48800 out of 80000 steps  (61%)
20:51:31:WU01:FS00:0xa4:Completed 49600 out of 80000 steps  (62%)
20:51:55:WU02:FS02:0x18:Completed 25000 out of 2500000 steps (1%)
20:52:06:WU07:FS01:0x18:Completed 4250000 out of 5000000 steps (85%)
20:52:06:WU01:FS00:0xa4:Completed 50400 out of 80000 steps  (63%)
20:52:41:WU01:FS00:0xa4:Completed 51200 out of 80000 steps  (64%)
20:53:16:WU01:FS00:0xa4:Completed 52000 out of 80000 steps  (65%)
20:53:51:WU01:FS00:0xa4:Completed 52800 out of 80000 steps  (66%)
20:54:25:WU01:FS00:0xa4:Completed 53600 out of 80000 steps  (67%)
20:54:42:WU07:FS01:0x18:Completed 4300000 out of 5000000 steps (86%)
20:54:56:WU02:FS02:0x18:Completed 50000 out of 2500000 steps (2%)
20:55:00:WU01:FS00:0xa4:Completed 54400 out of 80000 steps  (68%)
20:55:35:WU01:FS00:0xa4:Completed 55200 out of 80000 steps  (69%)
20:56:10:WU01:FS00:0xa4:Completed 56000 out of 80000 steps  (70%)
20:56:44:WU01:FS00:0xa4:Completed 56800 out of 80000 steps  (71%)
20:57:07:WU07:FS01:0x18:Completed 4350000 out of 5000000 steps (87%)
20:57:19:WU01:FS00:0xa4:Completed 57600 out of 80000 steps  (72%)
20:57:54:WU01:FS00:0xa4:Completed 58400 out of 80000 steps  (73%)
20:57:57:WU02:FS02:0x18:Completed 75000 out of 2500000 steps (3%)
20:58:29:WU01:FS00:0xa4:Completed 59200 out of 80000 steps  (74%)
20:59:03:WU01:FS00:0xa4:Completed 60000 out of 80000 steps  (75%)
20:59:38:WU01:FS00:0xa4:Completed 60800 out of 80000 steps  (76%)
20:59:43:WU07:FS01:0x18:Completed 4400000 out of 5000000 steps (88%)
21:00:13:WU01:FS00:0xa4:Completed 61600 out of 80000 steps  (77%)
21:00:48:WU01:FS00:0xa4:Completed 62400 out of 80000 steps  (78%)
21:00:57:WU02:FS02:0x18:Completed 100000 out of 2500000 steps (4%)
21:01:21:WU01:FS00:0xa4:Completed 63200 out of 80000 steps  (79%)
21:01:57:WU01:FS00:0xa4:Completed 64000 out of 80000 steps  (80%)
21:02:09:WU07:FS01:0x18:Completed 4450000 out of 5000000 steps (89%)
21:02:32:WU01:FS00:0xa4:Completed 64800 out of 80000 steps  (81%)
21:03:07:WU01:FS00:0xa4:Completed 65600 out of 80000 steps  (82%)
21:03:41:WU01:FS00:0xa4:Completed 66400 out of 80000 steps  (83%)
21:04:15:WU02:FS02:0x18:Completed 125000 out of 2500000 steps (5%)
21:04:16:WU01:FS00:0xa4:Completed 67200 out of 80000 steps  (84%)
21:04:34:WU07:FS01:0x18:Completed 4500000 out of 5000000 steps (90%)
21:04:51:WU01:FS00:0xa4:Completed 68000 out of 80000 steps  (85%)
21:05:26:WU01:FS00:0xa4:Completed 68800 out of 80000 steps  (86%)
21:06:00:WU01:FS00:0xa4:Completed 69600 out of 80000 steps  (87%)
21:06:35:WU01:FS00:0xa4:Completed 70400 out of 80000 steps  (88%)
21:07:10:WU07:FS01:0x18:Completed 4550000 out of 5000000 steps (91%)
21:07:10:WU01:FS00:0xa4:Completed 71200 out of 80000 steps  (89%)
21:07:17:WU02:FS02:0x18:Completed 150000 out of 2500000 steps (6%)
21:07:44:WU01:FS00:0xa4:Completed 72000 out of 80000 steps  (90%)
21:08:19:WU01:FS00:0xa4:Completed 72800 out of 80000 steps  (91%)
21:08:54:WU01:FS00:0xa4:Completed 73600 out of 80000 steps  (92%)
21:09:28:WU01:FS00:0xa4:Completed 74400 out of 80000 steps  (93%)
21:09:35:WU07:FS01:0x18:Completed 4600000 out of 5000000 steps (92%)
21:10:03:WU01:FS00:0xa4:Completed 75200 out of 80000 steps  (94%)
21:10:17:WU02:FS02:0x18:Completed 175000 out of 2500000 steps (7%)
21:10:38:WU01:FS00:0xa4:Completed 76000 out of 80000 steps  (95%)
21:11:13:WU01:FS00:0xa4:Completed 76800 out of 80000 steps  (96%)
21:11:48:WU01:FS00:0xa4:Completed 77600 out of 80000 steps  (97%)
21:12:11:WU07:FS01:0x18:Completed 4650000 out of 5000000 steps (93%)
21:12:23:WU01:FS00:0xa4:Completed 78400 out of 80000 steps  (98%)
21:12:58:WU01:FS00:0xa4:Completed 79200 out of 80000 steps  (99%)
21:12:58:WU00:FS00:Connecting to 171.67.108.200:8080
21:12:59:WU00:FS00:Assigned to work server 171.64.65.99
21:12:59:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:30 from 171.64.65.99
21:12:59:WU00:FS00:Connecting to 171.64.65.99:8080
21:13:02:WU00:FS00:Downloading 6.22MiB
21:13:05:WU00:FS00:Download complete
21:13:05:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9752 run:248 clone:0 gen:347 core:0xa4 unit:0x000001d0ab404163554171e990ee5f05
21:13:18:WU02:FS02:0x18:Completed 200000 out of 2500000 steps (8%)
21:13:32:WU01:FS00:0xa4:Completed 80000 out of 80000 steps  (100%)
21:13:34:WU01:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
21:13:44:WU01:FS00:0xa4:
21:13:44:WU01:FS00:0xa4:Finished Work Unit:
21:13:44:WU01:FS00:0xa4:- Reading up to 4123296 from "01/wudata_01.trr": Read 4123296
21:13:44:WU01:FS00:0xa4:trr file hash check passed.
21:13:44:WU01:FS00:0xa4:- Reading up to 3193480 from "01/wudata_01.xtc": Read 3193480
21:13:44:WU01:FS00:0xa4:xtc file hash check passed.
21:13:44:WU01:FS00:0xa4:edr file hash check passed.
21:13:44:WU01:FS00:0xa4:logfile size: 19893
21:13:44:WU01:FS00:0xa4:Leaving Run
21:13:47:WU01:FS00:0xa4:- Writing 7339061 bytes of core data to disk...
21:13:49:WU01:FS00:0xa4:Done: 7338549 -> 7067390 (compressed to 96.3 percent)
21:13:49:WU01:FS00:0xa4:  ... Done.
21:13:50:WU01:FS00:0xa4:- Shutting down core
21:13:50:WU01:FS00:0xa4:
21:13:50:WU01:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
21:13:50:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
21:13:50:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9752 run:1457 clone:0 gen:168 core:0xa4 unit:0x000000e5ab4041635541739d694b9db6
21:13:50:WU01:FS00:Uploading 6.74MiB to 171.64.65.99
21:13:50:WU01:FS00:Connecting to 171.64.65.99:8080
21:13:51:WU00:FS00:Starting
21:13:51:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 704 -lifeline 5860 -checkpoint 15 -np 30
21:13:51:WU00:FS00:Started FahCore on PID 5312
21:13:51:WU00:FS00:Core PID:3164
21:13:51:WU00:FS00:FahCore 0xa4 started
21:13:51:WU00:FS00:0xa4:
21:13:51:WU00:FS00:0xa4:*------------------------------*
21:13:51:WU00:FS00:0xa4:Folding@Home Gromacs GB Core
21:13:51:WU00:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
21:13:51:WU00:FS00:0xa4:
21:13:51:WU00:FS00:0xa4:Preparing to commence simulation
21:13:51:WU00:FS00:0xa4:- Looking at optimizations...
21:13:51:WU00:FS00:0xa4:- Created dyn
21:13:51:WU00:FS00:0xa4:- Files status OK
21:13:53:WU00:FS00:0xa4:- Expanded 6525109 -> 22366436 (decompressed 342.7 percent)
21:13:53:WU00:FS00:0xa4:Called DecompressByteArray: compressed_data_size=6525109 data_size=22366436, decompressed_data_size=22366436 diff=0
21:13:54:WU00:FS00:0xa4:- Digital signature verified
21:13:54:WU00:FS00:0xa4:
21:13:54:WU00:FS00:0xa4:Project: 9752 (Run 248, Clone 0, Gen 347)
21:13:54:WU00:FS00:0xa4:
21:13:54:WU00:FS00:0xa4:Assembly optimizations on if available.
21:13:54:WU00:FS00:0xa4:Entering M.D.
21:13:56:WU01:FS00:Upload 93.65%
21:13:58:WU01:FS00:Upload complete
21:13:58:WU01:FS00:Server responded WORK_ACK (400)
21:13:58:WU01:FS00:Final credit estimate, 4276.00 points
21:13:58:WU01:FS00:Cleaning up
21:14:00:WU00:FS00:0xa4:Mapping NT from 30 to 30 
21:14:02:WU00:FS00:0xa4:Completed 0 out of 80000 steps  (0%)
21:14:36:WU07:FS01:0x18:Completed 4700000 out of 5000000 steps (94%)
21:14:37:WU00:FS00:0xa4:Completed 800 out of 80000 steps  (1%)
21:15:12:WU00:FS00:0xa4:Completed 1600 out of 80000 steps  (2%)
21:15:47:WU00:FS00:0xa4:Completed 2400 out of 80000 steps  (3%)
21:16:22:WU00:FS00:0xa4:Completed 3200 out of 80000 steps  (4%)
21:16:35:WU02:FS02:0x18:Completed 225000 out of 2500000 steps (9%)
21:16:56:WU00:FS00:0xa4:Completed 4000 out of 80000 steps  (5%)
21:17:01:WU07:FS01:0x18:Completed 4750000 out of 5000000 steps (95%)
21:17:32:WU00:FS00:0xa4:Completed 4800 out of 80000 steps  (6%)
21:18:06:WU00:FS00:0xa4:Completed 5600 out of 80000 steps  (7%)
21:18:41:WU00:FS00:0xa4:Completed 6400 out of 80000 steps  (8%)
21:19:16:WU00:FS00:0xa4:Completed 7200 out of 80000 steps  (9%)
21:19:36:WU02:FS02:0x18:Completed 250000 out of 2500000 steps (10%)
21:19:37:WU07:FS01:0x18:Completed 4800000 out of 5000000 steps (96%)
21:19:51:WU00:FS00:0xa4:Completed 8000 out of 80000 steps  (10%)
21:20:26:WU00:FS00:0xa4:Completed 8800 out of 80000 steps  (11%)
21:21:01:WU00:FS00:0xa4:Completed 9600 out of 80000 steps  (12%)
21:21:35:WU00:FS00:0xa4:Completed 10400 out of 80000 steps  (13%)
21:22:03:WU07:FS01:0x18:Completed 4850000 out of 5000000 steps (97%)
21:22:11:WU00:FS00:0xa4:Completed 11200 out of 80000 steps  (14%)
21:22:36:WU02:FS02:0x18:Completed 275000 out of 2500000 steps (11%)
21:22:46:WU00:FS00:0xa4:Completed 12000 out of 80000 steps  (15%)
21:23:21:WU00:FS00:0xa4:Completed 12800 out of 80000 steps  (16%)
21:23:56:WU00:FS00:0xa4:Completed 13600 out of 80000 steps  (17%)
21:24:31:WU00:FS00:0xa4:Completed 14400 out of 80000 steps  (18%)
21:24:39:WU07:FS01:0x18:Completed 4900000 out of 5000000 steps (98%)
21:25:06:WU00:FS00:0xa4:Completed 15200 out of 80000 steps  (19%)
21:25:37:WU02:FS02:0x18:Completed 300000 out of 2500000 steps (12%)
21:25:41:WU00:FS00:0xa4:Completed 16000 out of 80000 steps  (20%)
21:26:15:WU00:FS00:0xa4:Completed 16800 out of 80000 steps  (21%)
21:26:50:WU00:FS00:0xa4:Completed 17600 out of 80000 steps  (22%)
21:27:04:WU07:FS01:0x18:Completed 4950000 out of 5000000 steps (99%)
21:27:25:WU00:FS00:0xa4:Completed 18400 out of 80000 steps  (23%)
21:28:00:WU00:FS00:0xa4:Completed 19200 out of 80000 steps  (24%)
21:28:35:WU00:FS00:0xa4:Completed 20000 out of 80000 steps  (25%)
21:28:55:WU02:FS02:0x18:Completed 325000 out of 2500000 steps (13%)
21:29:10:WU00:FS00:0xa4:Completed 20800 out of 80000 steps  (26%)
21:29:29:WU07:FS01:0x18:Completed 5000000 out of 5000000 steps (100%)
21:29:30:WU01:FS01:Connecting to 171.67.108.200:80
21:29:31:WU01:FS01:Assigned to work server 171.64.65.93
21:29:31:WU01:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GM204 [GeForce GTX 980] from 171.64.65.93
21:29:31:WU01:FS01:Connecting to 171.64.65.93:8080
21:29:31:WU01:FS01:Downloading 3.59MiB
21:29:34:WU01:FS01:Download complete
21:29:34:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9129 run:11 clone:6 gen:67 core:0x18 unit:0x0000004c0a3b1e815549371e23703f44
21:29:41:WU07:FS01:0x18:Saving result file logfile_01.txt
21:29:41:WU07:FS01:0x18:Saving result file checkpointState.xml
21:29:43:WU07:FS01:0x18:Saving result file checkpt.crc
21:29:43:WU07:FS01:0x18:Saving result file log.txt
21:29:43:WU07:FS01:0x18:Saving result file positions.xtc
21:29:45:WU07:FS01:0x18:Folding@home Core Shutdown: FINISHED_UNIT
21:29:45:WU00:FS00:0xa4:Completed 21600 out of 80000 steps  (27%)
21:29:46:WU07:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
21:29:46:WU07:FS01:Sending unit results: id:07 state:SEND error:NO_ERROR project:10476 run:0 clone:60 gen:398 core:0x18 unit:0x000001e0538b3dba53f3811c64984db2
21:29:46:WU07:FS01:Uploading 6.12MiB to 140.163.4.234
21:29:46:WU07:FS01:Connecting to 140.163.4.234:8080
21:29:46:WU01:FS01:Starting
21:29:46:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 01 -suffix 01 -version 704 -lifeline 5860 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
21:29:46:WU01:FS01:Started FahCore on PID 5012
21:29:46:WU01:FS01:Core PID:4300
21:29:46:WU01:FS01:FahCore 0x18 started
21:29:46:WU01:FS01:0x18:*********************** Log Started 2015-08-10T21:29:46Z ***********************
21:29:46:WU01:FS01:0x18:Project: 9129 (Run 11, Clone 6, Gen 67)
21:29:46:WU01:FS01:0x18:Unit: 0x0000004c0a3b1e815549371e23703f44
21:29:46:WU01:FS01:0x18:CPU: 0x00000000000000000000000000000000
21:29:46:WU01:FS01:0x18:Machine: 1
21:29:46:WU01:FS01:0x18:Reading tar file state.xml
21:29:47:WU01:FS01:0x18:Reading tar file system.xml
21:29:48:WU01:FS01:0x18:Reading tar file integrator.xml
21:29:48:WU01:FS01:0x18:Reading tar file core.xml
21:29:48:WU01:FS01:0x18:Digital signatures verified
21:29:48:WU01:FS01:0x18:Folding@home GPU core18
21:29:48:WU01:FS01:0x18:Version 0.0.4
21:30:02:WU07:FS01:Upload 52.09%
21:30:08:WU07:FS01:Upload 53.11%
21:30:17:WU07:FS01:Upload complete
21:30:17:WU07:FS01:Server responded WORK_ACK (400)
21:30:17:WU07:FS01:Final credit estimate, 41881.00 points
21:30:17:WU07:FS01:Cleaning up
21:30:21:WU00:FS00:0xa4:Completed 22400 out of 80000 steps  (28%)
21:30:30:WU01:FS01:0x18:Completed 0 out of 2500000 steps (0%)
21:30:30:WU01:FS01:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
21:30:57:WU00:FS00:0xa4:Completed 23200 out of 80000 steps  (29%)
21:31:32:WU00:FS00:0xa4:Completed 24000 out of 80000 steps  (30%)
21:31:56:WU02:FS02:0x18:Completed 350000 out of 2500000 steps (14%)
21:32:09:WU00:FS00:0xa4:Completed 24800 out of 80000 steps  (31%)
21:32:39:WU01:FS01:0x18:Completed 25000 out of 2500000 steps (1%)
21:32:44:WU00:FS00:0xa4:Completed 25600 out of 80000 steps  (32%)
21:33:20:WU00:FS00:0xa4:Completed 26400 out of 80000 steps  (33%)
21:33:55:WU00:FS00:0xa4:Completed 27200 out of 80000 steps  (34%)
21:34:31:WU00:FS00:0xa4:Completed 28000 out of 80000 steps  (35%)
21:34:36:WU01:FS01:0x18:Completed 50000 out of 2500000 steps (2%)
21:34:57:WU02:FS02:0x18:Completed 375000 out of 2500000 steps (15%)
21:35:07:WU00:FS00:0xa4:Completed 28800 out of 80000 steps  (36%)
21:35:43:WU00:FS00:0xa4:Completed 29600 out of 80000 steps  (37%)
21:36:18:WU00:FS00:0xa4:Completed 30400 out of 80000 steps  (38%)
21:36:34:WU01:FS01:0x18:Completed 75000 out of 2500000 steps (3%)
21:36:54:WU00:FS00:0xa4:Completed 31200 out of 80000 steps  (39%)
21:37:29:WU00:FS00:0xa4:Completed 32000 out of 80000 steps  (40%)
21:37:58:WU02:FS02:0x18:Completed 400000 out of 2500000 steps (16%)
21:38:05:WU00:FS00:0xa4:Completed 32800 out of 80000 steps  (41%)
21:38:32:WU01:FS01:0x18:Completed 100000 out of 2500000 steps (4%)
21:38:40:WU00:FS00:0xa4:Completed 33600 out of 80000 steps  (42%)
21:39:15:WU00:FS00:0xa4:Completed 34400 out of 80000 steps  (43%)
21:39:51:WU00:FS00:0xa4:Completed 35200 out of 80000 steps  (44%)
21:40:27:WU00:FS00:0xa4:Completed 36000 out of 80000 steps  (45%)
21:40:45:WU01:FS01:0x18:Completed 125000 out of 2500000 steps (5%)
21:41:02:WU00:FS00:0xa4:Completed 36800 out of 80000 steps  (46%)
21:41:16:WU02:FS02:0x18:Completed 425000 out of 2500000 steps (17%)
21:41:38:WU00:FS00:0xa4:Completed 37600 out of 80000 steps  (47%)
21:42:14:WU00:FS00:0xa4:Completed 38400 out of 80000 steps  (48%)
21:42:43:WU01:FS01:0x18:Completed 150000 out of 2500000 steps (6%)
21:42:50:WU00:FS00:0xa4:Completed 39200 out of 80000 steps  (49%)
21:43:26:WU00:FS00:0xa4:Completed 40000 out of 80000 steps  (50%)
21:44:01:WU00:FS00:0xa4:Completed 40800 out of 80000 steps  (51%)
21:44:17:WU02:FS02:0x18:Completed 450000 out of 2500000 steps (18%)
21:44:37:WU00:FS00:0xa4:Completed 41600 out of 80000 steps  (52%)
21:44:41:WU01:FS01:0x18:Completed 175000 out of 2500000 steps (7%)
21:45:13:WU00:FS00:0xa4:Completed 42400 out of 80000 steps  (53%)
21:45:49:WU00:FS00:0xa4:Completed 43200 out of 80000 steps  (54%)
21:46:24:WU00:FS00:0xa4:Completed 44000 out of 80000 steps  (55%)
21:46:39:WU01:FS01:0x18:Completed 200000 out of 2500000 steps (8%)
21:47:01:WU00:FS00:0xa4:Completed 44800 out of 80000 steps  (56%)
21:47:18:WU02:FS02:0x18:Completed 475000 out of 2500000 steps (19%)
21:47:36:WU00:FS00:0xa4:Completed 45600 out of 80000 steps  (57%)
21:48:12:WU00:FS00:0xa4:Completed 46400 out of 80000 steps  (58%)
21:48:48:WU00:FS00:0xa4:Completed 47200 out of 80000 steps  (59%)
21:48:52:WU01:FS01:0x18:Completed 225000 out of 2500000 steps (9%)
21:49:23:WU00:FS00:0xa4:Completed 48000 out of 80000 steps  (60%)
21:49:59:WU00:FS00:0xa4:Completed 48800 out of 80000 steps  (61%)
21:50:19:WU02:FS02:0x18:Completed 500000 out of 2500000 steps (20%)
21:50:34:WU00:FS00:0xa4:Completed 49600 out of 80000 steps  (62%)
21:50:50:WU01:FS01:0x18:Completed 250000 out of 2500000 steps (10%)
21:51:09:WU00:FS00:0xa4:Completed 50400 out of 80000 steps  (63%)
21:51:45:WU00:FS00:0xa4:Completed 51200 out of 80000 steps  (64%)
21:52:21:WU00:FS00:0xa4:Completed 52000 out of 80000 steps  (65%)
21:52:48:WU01:FS01:0x18:Completed 275000 out of 2500000 steps (11%)
21:52:57:WU00:FS00:0xa4:Completed 52800 out of 80000 steps  (66%)
21:53:33:WU00:FS00:0xa4:Completed 53600 out of 80000 steps  (67%)
21:53:37:WU02:FS02:0x18:Completed 525000 out of 2500000 steps (21%)
21:54:08:WU00:FS00:0xa4:Completed 54400 out of 80000 steps  (68%)
21:54:44:WU00:FS00:0xa4:Completed 55200 out of 80000 steps  (69%)
21:54:46:WU01:FS01:0x18:Completed 300000 out of 2500000 steps (12%)
21:55:20:WU00:FS00:0xa4:Completed 56000 out of 80000 steps  (70%)
21:55:55:WU00:FS00:0xa4:Completed 56800 out of 80000 steps  (71%)
21:56:31:WU00:FS00:0xa4:Completed 57600 out of 80000 steps  (72%)
21:56:38:WU02:FS02:0x18:Completed 550000 out of 2500000 steps (22%)
21:56:59:WU01:FS01:0x18:Completed 325000 out of 2500000 steps (13%)
21:57:07:WU00:FS00:0xa4:Completed 58400 out of 80000 steps  (73%)
21:57:42:WU00:FS00:0xa4:Completed 59200 out of 80000 steps  (74%)
21:58:18:WU00:FS00:0xa4:Completed 60000 out of 80000 steps  (75%)
21:58:53:WU00:FS00:0xa4:Completed 60800 out of 80000 steps  (76%)
21:58:57:WU01:FS01:0x18:Completed 350000 out of 2500000 steps (14%)
21:59:29:WU00:FS00:0xa4:Completed 61600 out of 80000 steps  (77%)
21:59:39:WU02:FS02:0x18:Completed 575000 out of 2500000 steps (23%)
22:00:05:WU00:FS00:0xa4:Completed 62400 out of 80000 steps  (78%)
22:00:40:WU00:FS00:0xa4:Completed 63200 out of 80000 steps  (79%)
22:00:55:WU01:FS01:0x18:Completed 375000 out of 2500000 steps (15%)
22:01:16:WU00:FS00:0xa4:Completed 64000 out of 80000 steps  (80%)
22:01:52:WU00:FS00:0xa4:Completed 64800 out of 80000 steps  (81%)
22:02:29:WU00:FS00:0xa4:Completed 65600 out of 80000 steps  (82%)
22:02:40:WU02:FS02:0x18:Completed 600000 out of 2500000 steps (24%)
22:02:53:WU01:FS01:0x18:Completed 400000 out of 2500000 steps (16%)
22:03:03:WU00:FS00:0xa4:Completed 66400 out of 80000 steps  (83%)
22:03:39:WU00:FS00:0xa4:Completed 67200 out of 80000 steps  (84%)
22:04:14:WU00:FS00:0xa4:Completed 68000 out of 80000 steps  (85%)
22:04:50:WU00:FS00:0xa4:Completed 68800 out of 80000 steps  (86%)
22:05:06:WU01:FS01:0x18:Completed 425000 out of 2500000 steps (17%)
22:05:26:WU00:FS00:0xa4:Completed 69600 out of 80000 steps  (87%)
22:05:57:WU02:FS02:0x18:Completed 625000 out of 2500000 steps (25%)
22:06:02:WU00:FS00:0xa4:Completed 70400 out of 80000 steps  (88%)
22:06:37:WU00:FS00:0xa4:Completed 71200 out of 80000 steps  (89%)
22:07:04:WU01:FS01:0x18:Completed 450000 out of 2500000 steps (18%)
22:07:13:WU00:FS00:0xa4:Completed 72000 out of 80000 steps  (90%)
22:07:49:WU00:FS00:0xa4:Completed 72800 out of 80000 steps  (91%)
22:08:24:WU00:FS00:0xa4:Completed 73600 out of 80000 steps  (92%)
22:08:59:WU02:FS02:0x18:Completed 650000 out of 2500000 steps (26%)
22:09:00:WU00:FS00:0xa4:Completed 74400 out of 80000 steps  (93%)
22:09:01:WU01:FS01:0x18:Completed 475000 out of 2500000 steps (19%)
22:09:35:WU00:FS00:0xa4:Completed 75200 out of 80000 steps  (94%)
22:10:11:WU00:FS00:0xa4:Completed 76000 out of 80000 steps  (95%)
22:10:47:WU00:FS00:0xa4:Completed 76800 out of 80000 steps  (96%)
22:10:59:WU01:FS01:0x18:Completed 500000 out of 2500000 steps (20%)
22:11:22:WU00:FS00:0xa4:Completed 77600 out of 80000 steps  (97%)
22:11:58:WU00:FS00:0xa4:Completed 78400 out of 80000 steps  (98%)
22:11:59:WU02:FS02:0x18:Completed 675000 out of 2500000 steps (27%)
22:12:35:WU00:FS00:0xa4:Completed 79200 out of 80000 steps  (99%)
22:12:35:WU03:FS00:Connecting to 171.67.108.200:8080
22:12:35:WU03:FS00:Assigned to work server 171.64.65.99
22:12:35:WU03:FS00:Requesting new work unit for slot 00: RUNNING cpu:30 from 171.64.65.99
22:12:35:WU03:FS00:Connecting to 171.64.65.99:8080
22:12:38:WU03:FS00:Downloading 6.26MiB
22:12:42:WU03:FS00:Download complete
22:12:42:WU03:FS00:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:9752 run:2237 clone:0 gen:122 core:0xa4 unit:0x000000a0ab4041635541748a82475a52
22:13:10:WU00:FS00:0xa4:Completed 80000 out of 80000 steps  (100%)
22:13:13:WU00:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
22:13:13:WU01:FS01:0x18:Completed 525000 out of 2500000 steps (21%)
22:13:23:WU00:FS00:0xa4:
22:13:23:WU00:FS00:0xa4:Finished Work Unit:
22:13:23:WU00:FS00:0xa4:- Reading up to 4117896 from "00/wudata_01.trr": Read 4117896
22:13:23:WU00:FS00:0xa4:trr file hash check passed.
22:13:23:WU00:FS00:0xa4:- Reading up to 3189492 from "00/wudata_01.xtc": Read 3189492
22:13:23:WU00:FS00:0xa4:xtc file hash check passed.
22:13:23:WU00:FS00:0xa4:edr file hash check passed.
22:13:23:WU00:FS00:0xa4:logfile size: 19893
22:13:23:WU00:FS00:0xa4:Leaving Run
22:13:23:WU00:FS00:0xa4:- Writing 7329673 bytes of core data to disk...
22:13:24:WU00:FS00:0xa4:Done: 7329161 -> 7058897 (compressed to 96.3 percent)
22:13:24:WU00:FS00:0xa4:  ... Done.
22:13:26:WU00:FS00:0xa4:- Shutting down core
22:13:26:WU00:FS00:0xa4:
22:13:26:WU00:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
22:13:26:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:13:26:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:9752 run:248 clone:0 gen:347 core:0xa4 unit:0x000001d0ab404163554171e990ee5f05
22:13:26:WU00:FS00:Uploading 6.73MiB to 171.64.65.99
22:13:26:WU00:FS00:Connecting to 171.64.65.99:8080
22:13:26:WU03:FS00:Starting
22:13:26:WU03:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 03 -suffix 01 -version 704 -lifeline 5860 -checkpoint 15 -np 30
22:13:26:WU03:FS00:Started FahCore on PID 3016
22:13:26:WU03:FS00:Core PID:3124
22:13:26:WU03:FS00:FahCore 0xa4 started
22:13:27:WU03:FS00:0xa4:
22:13:27:WU03:FS00:0xa4:*------------------------------*
22:13:27:WU03:FS00:0xa4:Folding@Home Gromacs GB Core
22:13:27:WU03:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
22:13:27:WU03:FS00:0xa4:
22:13:27:WU03:FS00:0xa4:Preparing to commence simulation
22:13:27:WU03:FS00:0xa4:- Looking at optimizations...
22:13:27:WU03:FS00:0xa4:- Created dyn
22:13:27:WU03:FS00:0xa4:- Files status OK
22:13:29:WU03:FS00:0xa4:- Expanded 6564599 -> 22561724 (decompressed 343.6 percent)
22:13:29:WU03:FS00:0xa4:Called DecompressByteArray: compressed_data_size=6564599 data_size=22561724, decompressed_data_size=22561724 diff=0
22:13:29:WU03:FS00:0xa4:- Digital signature verified
22:13:29:WU03:FS00:0xa4:
22:13:29:WU03:FS00:0xa4:Project: 9752 (Run 2237, Clone 0, Gen 122)
22:13:29:WU03:FS00:0xa4:
22:13:30:WU03:FS00:0xa4:Assembly optimizations on if available.
22:13:30:WU03:FS00:0xa4:Entering M.D.
22:13:32:WU00:FS00:Upload 50.13%
22:13:36:WU00:FS00:Upload complete
22:13:36:WU03:FS00:0xa4:Mapping NT from 30 to 30 
22:13:36:WU00:FS00:Server responded WORK_ACK (400)
22:13:36:WU00:FS00:Final credit estimate, 4221.00 points
22:13:36:WU00:FS00:Cleaning up
22:13:38:WU03:FS00:0xa4:Completed 0 out of 80000 steps  (0%)
22:14:14:WU03:FS00:0xa4:Completed 800 out of 80000 steps  (1%)
22:14:50:WU03:FS00:0xa4:Completed 1600 out of 80000 steps  (2%)
22:15:00:WU02:FS02:0x18:Completed 700000 out of 2500000 steps (28%)
22:15:10:WU01:FS01:0x18:Completed 550000 out of 2500000 steps (22%)
22:15:25:WU03:FS00:0xa4:Completed 2400 out of 80000 steps  (3%)
22:16:01:WU03:FS00:0xa4:Completed 3200 out of 80000 steps  (4%)
22:16:37:WU03:FS00:0xa4:Completed 4000 out of 80000 steps  (5%)
22:17:08:WU01:FS01:0x18:Completed 575000 out of 2500000 steps (23%)
22:17:13:WU03:FS00:0xa4:Completed 4800 out of 80000 steps  (6%)
22:17:48:WU03:FS00:0xa4:Completed 5600 out of 80000 steps  (7%)
22:18:18:WU02:FS02:0x18:Completed 725000 out of 2500000 steps (29%)
22:18:24:WU03:FS00:0xa4:Completed 6400 out of 80000 steps  (8%)
22:19:00:WU03:FS00:0xa4:Completed 7200 out of 80000 steps  (9%)
22:19:06:WU01:FS01:0x18:Completed 600000 out of 2500000 steps (24%)
22:19:36:WU03:FS00:0xa4:Completed 8000 out of 80000 steps  (10%)
22:20:12:WU03:FS00:0xa4:Completed 8800 out of 80000 steps  (11%)
22:20:48:WU03:FS00:0xa4:Completed 9600 out of 80000 steps  (12%)
22:21:19:WU02:FS02:0x18:Completed 750000 out of 2500000 steps (30%)
22:21:19:WU01:FS01:0x18:Completed 625000 out of 2500000 steps (25%)
22:21:23:WU03:FS00:0xa4:Completed 10400 out of 80000 steps  (13%)
22:21:59:WU03:FS00:0xa4:Completed 11200 out of 80000 steps  (14%)
22:22:36:WU03:FS00:0xa4:Completed 12000 out of 80000 steps  (15%)
22:23:12:WU03:FS00:0xa4:Completed 12800 out of 80000 steps  (16%)
22:23:19:WU01:FS01:0x18:Completed 650000 out of 2500000 steps (26%)
22:23:48:WU03:FS00:0xa4:Completed 13600 out of 80000 steps  (17%)
22:24:20:WU02:FS02:0x18:Completed 775000 out of 2500000 steps (31%)
22:24:23:WU03:FS00:0xa4:Completed 14400 out of 80000 steps  (18%)
22:24:59:WU03:FS00:0xa4:Completed 15200 out of 80000 steps  (19%)
22:25:17:WU01:FS01:0x18:Completed 675000 out of 2500000 steps (27%)
22:25:35:WU03:FS00:0xa4:Completed 16000 out of 80000 steps  (20%)
22:26:11:WU03:FS00:0xa4:Completed 16800 out of 80000 steps  (21%)
22:26:47:WU03:FS00:0xa4:Completed 17600 out of 80000 steps  (22%)
22:27:15:WU01:FS01:0x18:Completed 700000 out of 2500000 steps (28%)
22:27:20:WU02:FS02:0x18:Completed 800000 out of 2500000 steps (32%)
22:27:22:WU03:FS00:0xa4:Completed 18400 out of 80000 steps  (23%)
22:27:58:WU03:FS00:0xa4:Completed 19200 out of 80000 steps  (24%)
22:28:33:WU03:FS00:0xa4:Completed 20000 out of 80000 steps  (25%)
22:29:09:WU03:FS00:0xa4:Completed 20800 out of 80000 steps  (26%)
22:29:28:WU01:FS01:0x18:Completed 725000 out of 2500000 steps (29%)
22:29:45:WU03:FS00:0xa4:Completed 21600 out of 80000 steps  (27%)
22:30:21:WU03:FS00:0xa4:Completed 22400 out of 80000 steps  (28%)
22:30:38:WU02:FS02:0x18:Completed 825000 out of 2500000 steps (33%)
22:30:57:WU03:FS00:0xa4:Completed 23200 out of 80000 steps  (29%)
22:31:26:WU01:FS01:0x18:Completed 750000 out of 2500000 steps (30%)
22:31:33:WU03:FS00:0xa4:Completed 24000 out of 80000 steps  (30%)
22:32:10:WU03:FS00:0xa4:Completed 24800 out of 80000 steps  (31%)
22:32:46:WU03:FS00:0xa4:Completed 25600 out of 80000 steps  (32%)
22:33:22:WU03:FS00:0xa4:Completed 26400 out of 80000 steps  (33%)
22:33:24:WU01:FS01:0x18:Completed 775000 out of 2500000 steps (31%)
22:33:39:WU02:FS02:0x18:Completed 850000 out of 2500000 steps (34%)
22:33:57:WU03:FS00:0xa4:Completed 27200 out of 80000 steps  (34%)
22:34:33:WU03:FS00:0xa4:Completed 28000 out of 80000 steps  (35%)
22:35:09:WU03:FS00:0xa4:Completed 28800 out of 80000 steps  (36%)
22:35:21:WU01:FS01:0x18:Completed 800000 out of 2500000 steps (32%)
22:35:45:WU03:FS00:0xa4:Completed 29600 out of 80000 steps  (37%)
22:36:21:WU03:FS00:0xa4:Completed 30400 out of 80000 steps  (38%)
22:36:40:WU02:FS02:0x18:Completed 875000 out of 2500000 steps (35%)
22:36:57:WU03:FS00:0xa4:Completed 31200 out of 80000 steps  (39%)
22:37:32:WU03:FS00:0xa4:Completed 32000 out of 80000 steps  (40%)
22:37:35:WU01:FS01:0x18:Completed 825000 out of 2500000 steps (33%)
22:38:08:WU03:FS00:0xa4:Completed 32800 out of 80000 steps  (41%)
22:38:44:WU03:FS00:0xa4:Completed 33600 out of 80000 steps  (42%)
22:39:20:WU03:FS00:0xa4:Completed 34400 out of 80000 steps  (43%)
22:39:33:WU01:FS01:0x18:Completed 850000 out of 2500000 steps (34%)
22:39:41:WU02:FS02:0x18:Completed 900000 out of 2500000 steps (36%)
22:39:55:WU03:FS00:0xa4:Completed 35200 out of 80000 steps  (44%)
22:40:31:WU03:FS00:0xa4:Completed 36000 out of 80000 steps  (45%)
22:41:07:WU03:FS00:0xa4:Completed 36800 out of 80000 steps  (46%)
22:41:31:WU01:FS01:0x18:Completed 875000 out of 2500000 steps (35%)
22:41:42:WU03:FS00:0xa4:Completed 37600 out of 80000 steps  (47%)
22:42:19:WU03:FS00:0xa4:Completed 38400 out of 80000 steps  (48%)
22:42:55:WU03:FS00:0xa4:Completed 39200 out of 80000 steps  (49%)
22:42:59:WU02:FS02:0x18:Completed 925000 out of 2500000 steps (37%)
22:43:29:WU01:FS01:0x18:Completed 900000 out of 2500000 steps (36%)
22:43:31:WU03:FS00:0xa4:Completed 40000 out of 80000 steps  (50%)
22:44:07:WU03:FS00:0xa4:Completed 40800 out of 80000 steps  (51%)
22:44:43:WU03:FS00:0xa4:Completed 41600 out of 80000 steps  (52%)
22:45:19:WU03:FS00:0xa4:Completed 42400 out of 80000 steps  (53%)
22:45:42:WU01:FS01:0x18:Completed 925000 out of 2500000 steps (37%)
22:45:55:WU03:FS00:0xa4:Completed 43200 out of 80000 steps  (54%)
22:46:00:WU02:FS02:0x18:Completed 950000 out of 2500000 steps (38%)
22:46:31:WU03:FS00:0xa4:Completed 44000 out of 80000 steps  (55%)
22:47:07:WU03:FS00:0xa4:Completed 44800 out of 80000 steps  (56%)
22:47:40:WU01:FS01:0x18:Completed 950000 out of 2500000 steps (38%)
22:47:43:WU03:FS00:0xa4:Completed 45600 out of 80000 steps  (57%)
22:48:18:WU03:FS00:0xa4:Completed 46400 out of 80000 steps  (58%)
22:48:54:WU03:FS00:0xa4:Completed 47200 out of 80000 steps  (59%)
22:49:01:WU02:FS02:0x18:Completed 975000 out of 2500000 steps (39%)
22:49:30:WU03:FS00:0xa4:Completed 48000 out of 80000 steps  (60%)
22:49:38:WU01:FS01:0x18:Completed 975000 out of 2500000 steps (39%)
22:50:06:WU03:FS00:0xa4:Completed 48800 out of 80000 steps  (61%)
22:50:42:WU03:FS00:0xa4:Completed 49600 out of 80000 steps  (62%)
22:51:18:WU03:FS00:0xa4:Completed 50400 out of 80000 steps  (63%)
22:51:36:WU01:FS01:0x18:Completed 1000000 out of 2500000 steps (40%)
22:51:54:WU03:FS00:0xa4:Completed 51200 out of 80000 steps  (64%)
22:52:02:WU02:FS02:0x18:Completed 1000000 out of 2500000 steps (40%)
22:52:30:WU03:FS00:0xa4:Completed 52000 out of 80000 steps  (65%)
22:53:05:WU03:FS00:0xa4:Completed 52800 out of 80000 steps  (66%)
22:53:41:WU03:FS00:0xa4:Completed 53600 out of 80000 steps  (67%)
22:53:49:WU01:FS01:0x18:Completed 1025000 out of 2500000 steps (41%)
22:54:17:WU03:FS00:0xa4:Completed 54400 out of 80000 steps  (68%)
22:54:53:WU03:FS00:0xa4:Completed 55200 out of 80000 steps  (69%)
22:55:20:WU02:FS02:0x18:Completed 1025000 out of 2500000 steps (41%)
22:55:29:WU03:FS00:0xa4:Completed 56000 out of 80000 steps  (70%)
22:55:47:WU01:FS01:0x18:Completed 1050000 out of 2500000 steps (42%)
22:56:05:WU03:FS00:0xa4:Completed 56800 out of 80000 steps  (71%)
22:56:41:WU03:FS00:0xa4:Completed 57600 out of 80000 steps  (72%)
22:57:16:WU03:FS00:0xa4:Completed 58400 out of 80000 steps  (73%)
22:57:45:WU01:FS01:0x18:Completed 1075000 out of 2500000 steps (43%)
22:57:52:WU03:FS00:0xa4:Completed 59200 out of 80000 steps  (74%)
22:58:22:WU02:FS02:0x18:Completed 1050000 out of 2500000 steps (42%)
22:58:28:WU03:FS00:0xa4:Completed 60000 out of 80000 steps  (75%)
22:59:04:WU03:FS00:0xa4:Completed 60800 out of 80000 steps  (76%)
22:59:40:WU03:FS00:0xa4:Completed 61600 out of 80000 steps  (77%)
22:59:43:WU01:FS01:0x18:Completed 1100000 out of 2500000 steps (44%)
23:00:16:WU03:FS00:0xa4:Completed 62400 out of 80000 steps  (78%)
23:00:52:WU03:FS00:0xa4:Completed 63200 out of 80000 steps  (79%)
23:01:22:WU02:FS02:0x18:Completed 1075000 out of 2500000 steps (43%)
23:01:28:WU03:FS00:0xa4:Completed 64000 out of 80000 steps  (80%)
23:01:57:WU01:FS01:0x18:Completed 1125000 out of 2500000 steps (45%)
23:02:03:WU03:FS00:0xa4:Completed 64800 out of 80000 steps  (81%)
23:02:40:WU03:FS00:0xa4:Completed 65600 out of 80000 steps  (82%)
23:03:16:WU03:FS00:0xa4:Completed 66400 out of 80000 steps  (83%)
23:03:52:WU03:FS00:0xa4:Completed 67200 out of 80000 steps  (84%)
23:03:55:WU01:FS01:0x18:Completed 1150000 out of 2500000 steps (46%)
23:04:23:WU02:FS02:0x18:Completed 1100000 out of 2500000 steps (44%)
23:04:27:WU03:FS00:0xa4:Completed 68000 out of 80000 steps  (85%)
23:05:02:WU03:FS00:0xa4:Completed 68800 out of 80000 steps  (86%)
23:05:38:WU03:FS00:0xa4:Completed 69600 out of 80000 steps  (87%)
23:05:53:WU01:FS01:0x18:Completed 1175000 out of 2500000 steps (47%)
23:06:14:WU03:FS00:0xa4:Completed 70400 out of 80000 steps  (88%)
23:06:50:WU03:FS00:0xa4:Completed 71200 out of 80000 steps  (89%)
23:07:26:WU03:FS00:0xa4:Completed 72000 out of 80000 steps  (90%)
23:07:41:WU02:FS02:0x18:Completed 1125000 out of 2500000 steps (45%)
23:07:51:WU01:FS01:0x18:Completed 1200000 out of 2500000 steps (48%)
23:08:02:WU03:FS00:0xa4:Completed 72800 out of 80000 steps  (91%)
23:08:38:WU03:FS00:0xa4:Completed 73600 out of 80000 steps  (92%)
23:09:13:WU03:FS00:0xa4:Completed 74400 out of 80000 steps  (93%)
23:09:49:WU03:FS00:0xa4:Completed 75200 out of 80000 steps  (94%)
23:10:04:WU01:FS01:0x18:Completed 1225000 out of 2500000 steps (49%)
23:10:25:WU03:FS00:0xa4:Completed 76000 out of 80000 steps  (95%)
23:10:42:WU02:FS02:0x18:Completed 1150000 out of 2500000 steps (46%)
23:11:01:WU03:FS00:0xa4:Completed 76800 out of 80000 steps  (96%)
23:11:36:WU03:FS00:0xa4:Completed 77600 out of 80000 steps  (97%)
23:12:02:WU01:FS01:0x18:Completed 1250000 out of 2500000 steps (50%)
23:12:12:WU03:FS00:0xa4:Completed 78400 out of 80000 steps  (98%)
23:12:49:WU03:FS00:0xa4:Completed 79200 out of 80000 steps  (99%)
23:12:50:WU00:FS00:Connecting to 171.67.108.200:8080
23:12:50:WU00:FS00:Assigned to work server 171.64.65.99
23:12:50:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:30 from 171.64.65.99
23:12:50:WU00:FS00:Connecting to 171.64.65.99:8080
23:12:53:WU00:FS00:Downloading 6.29MiB
23:12:56:WU00:FS00:Download complete
23:12:56:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9752 run:2908 clone:0 gen:45 core:0xa4 unit:0x0000003fab404163554175574aa3d596
23:13:25:WU03:FS00:0xa4:Completed 80000 out of 80000 steps  (100%)
23:13:27:WU03:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
23:13:38:WU03:FS00:0xa4:
23:13:38:WU03:FS00:0xa4:Finished Work Unit:
23:13:38:WU03:FS00:0xa4:- Reading up to 4134744 from "03/wudata_01.trr": Read 4134744
23:13:38:WU03:FS00:0xa4:trr file hash check passed.
23:13:38:WU03:FS00:0xa4:- Reading up to 3203080 from "03/wudata_01.xtc": Read 3203080
23:13:38:WU03:FS00:0xa4:xtc file hash check passed.
23:13:38:WU03:FS00:0xa4:edr file hash check passed.
23:13:38:WU03:FS00:0xa4:logfile size: 19882
23:13:38:WU03:FS00:0xa4:Leaving Run
23:13:39:WU03:FS00:0xa4:- Writing 7360098 bytes of core data to disk...
23:13:40:WU03:FS00:0xa4:Done: 7359586 -> 7089537 (compressed to 96.3 percent)
23:13:40:WU03:FS00:0xa4:  ... Done.
23:13:41:WU03:FS00:0xa4:- Shutting down core
23:13:41:WU03:FS00:0xa4:
23:13:41:WU03:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
23:13:42:WU03:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
23:13:42:WU03:FS00:Sending unit results: id:03 state:SEND error:NO_ERROR project:9752 run:2237 clone:0 gen:122 core:0xa4 unit:0x000000a0ab4041635541748a82475a52
23:13:42:WU03:FS00:Uploading 6.76MiB to 171.64.65.99
23:13:42:WU03:FS00:Connecting to 171.64.65.99:8080
23:13:42:WU00:FS00:Starting
23:13:42:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 704 -lifeline 5860 -checkpoint 15 -np 30
23:13:42:WU00:FS00:Started FahCore on PID 5472
23:13:42:WU00:FS00:Core PID:3680
23:13:42:WU00:FS00:FahCore 0xa4 started
23:13:42:WU00:FS00:0xa4:
23:13:42:WU00:FS00:0xa4:*------------------------------*
23:13:42:WU00:FS00:0xa4:Folding@Home Gromacs GB Core
23:13:42:WU00:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
23:13:42:WU00:FS00:0xa4:
23:13:42:WU00:FS00:0xa4:Preparing to commence simulation
23:13:42:WU00:FS00:0xa4:- Looking at optimizations...
23:13:42:WU00:FS00:0xa4:- Created dyn
23:13:42:WU00:FS00:0xa4:- Files status OK
23:13:43:WU02:FS02:0x18:Completed 1175000 out of 2500000 steps (47%)
23:13:45:WU00:FS00:0xa4:- Expanded 6599499 -> 22789968 (decompressed 345.3 percent)
23:13:45:WU00:FS00:0xa4:Called DecompressByteArray: compressed_data_size=6599499 data_size=22789968, decompressed_data_size=22789968 diff=0
23:13:45:WU00:FS00:0xa4:- Digital signature verified
23:13:45:WU00:FS00:0xa4:
23:13:45:WU00:FS00:0xa4:Project: 9752 (Run 2908, Clone 0, Gen 45)
23:13:45:WU00:FS00:0xa4:
23:13:45:WU00:FS00:0xa4:Assembly optimizations on if available.
23:13:45:WU00:FS00:0xa4:Entering M.D.
23:13:48:WU03:FS00:Upload 53.61%
23:13:51:WU00:FS00:0xa4:Mapping NT from 30 to 30 
23:13:52:WU03:FS00:Upload complete
23:13:53:WU03:FS00:Server responded WORK_ACK (400)
23:13:53:WU03:FS00:Final credit estimate, 4197.00 points
23:13:53:WU03:FS00:Cleaning up
23:13:53:WU00:FS00:0xa4:Completed 0 out of 80000 steps  (0%)
23:13:59:WU01:FS01:0x18:Completed 1275000 out of 2500000 steps (51%)
23:14:27:WU00:FS00:0xa4:Completed 800 out of 80000 steps  (1%)
23:15:01:WU00:FS00:0xa4:Completed 1600 out of 80000 steps  (2%)
23:15:35:WU00:FS00:0xa4:Completed 2400 out of 80000 steps  (3%)
23:15:57:WU01:FS01:0x18:Completed 1300000 out of 2500000 steps (52%)
23:16:10:WU00:FS00:0xa4:Completed 3200 out of 80000 steps  (4%)
23:16:41:WU02:FS02:0x18:Completed 1200000 out of 2500000 steps (48%)
23:16:43:WU00:FS00:0xa4:Completed 4000 out of 80000 steps  (5%)
23:17:17:WU00:FS00:0xa4:Completed 4800 out of 80000 steps  (6%)
23:17:52:WU00:FS00:0xa4:Completed 5600 out of 80000 steps  (7%)
23:18:12:WU01:FS01:0x18:Completed 1325000 out of 2500000 steps (53%)
23:18:26:WU00:FS00:0xa4:Completed 6400 out of 80000 steps  (8%)
23:19:00:WU00:FS00:0xa4:Completed 7200 out of 80000 steps  (9%)
23:19:35:WU00:FS00:0xa4:Completed 8000 out of 80000 steps  (10%)
23:19:56:WU02:FS02:0x18:Completed 1225000 out of 2500000 steps (49%)
23:20:08:WU00:FS00:0xa4:Completed 8800 out of 80000 steps  (11%)
23:20:11:WU01:FS01:0x18:Completed 1350000 out of 2500000 steps (54%)
23:20:42:WU00:FS00:0xa4:Completed 9600 out of 80000 steps  (12%)
23:21:15:WU00:FS00:0xa4:Completed 10400 out of 80000 steps  (13%)
23:21:49:WU00:FS00:0xa4:Completed 11200 out of 80000 steps  (14%)
23:22:10:WU01:FS01:0x18:Completed 1375000 out of 2500000 steps (55%)
23:22:23:WU00:FS00:0xa4:Completed 12000 out of 80000 steps  (15%)
23:22:54:WU02:FS02:0x18:Completed 1250000 out of 2500000 steps (50%)
23:22:56:WU00:FS00:0xa4:Completed 12800 out of 80000 steps  (16%)
23:23:30:WU00:FS00:0xa4:Completed 13600 out of 80000 steps  (17%)
23:24:03:WU00:FS00:0xa4:Completed 14400 out of 80000 steps  (18%)
23:24:08:WU01:FS01:0x18:Completed 1400000 out of 2500000 steps (56%)
23:24:36:WU00:FS00:0xa4:Completed 15200 out of 80000 steps  (19%)
23:25:09:WU00:FS00:0xa4:Completed 16000 out of 80000 steps  (20%)
23:25:43:WU00:FS00:0xa4:Completed 16800 out of 80000 steps  (21%)
23:25:52:WU02:FS02:0x18:Completed 1275000 out of 2500000 steps (51%)
23:26:16:WU00:FS00:0xa4:Completed 17600 out of 80000 steps  (22%)
23:26:23:WU01:FS01:0x18:Completed 1425000 out of 2500000 steps (57%)
23:26:49:WU00:FS00:0xa4:Completed 18400 out of 80000 steps  (23%)
23:27:23:WU00:FS00:0xa4:Completed 19200 out of 80000 steps  (24%)
23:27:56:WU00:FS00:0xa4:Completed 20000 out of 80000 steps  (25%)
23:28:21:WU01:FS01:0x18:Completed 1450000 out of 2500000 steps (58%)
23:28:29:WU00:FS00:0xa4:Completed 20800 out of 80000 steps  (26%)
23:28:50:WU02:FS02:0x18:Completed 1300000 out of 2500000 steps (52%)
23:29:03:WU00:FS00:0xa4:Completed 21600 out of 80000 steps  (27%)
23:29:36:WU00:FS00:0xa4:Completed 22400 out of 80000 steps  (28%)
******************************* Date: 2015-08-10 *******************************
23:30:09:WU00:FS00:0xa4:Completed 23200 out of 80000 steps  (29%)
23:30:19:WU01:FS01:0x18:Completed 1475000 out of 2500000 steps (59%)
23:30:42:WU00:FS00:0xa4:Completed 24000 out of 80000 steps  (30%)
23:31:16:WU00:FS00:0xa4:Completed 24800 out of 80000 steps  (31%)
23:31:49:WU00:FS00:0xa4:Completed 25600 out of 80000 steps  (32%)
23:32:06:WU02:FS02:0x18:Completed 1325000 out of 2500000 steps (53%)
23:32:18:WU01:FS01:0x18:Completed 1500000 out of 2500000 steps (60%)
23:32:22:WU00:FS00:0xa4:Completed 26400 out of 80000 steps  (33%)
23:32:56:WU00:FS00:0xa4:Completed 27200 out of 80000 steps  (34%)
23:33:30:WU00:FS00:0xa4:Completed 28000 out of 80000 steps  (35%)
23:34:03:WU00:FS00:0xa4:Completed 28800 out of 80000 steps  (36%)
23:34:32:WU01:FS01:0x18:Completed 1525000 out of 2500000 steps (61%)
23:34:37:WU00:FS00:0xa4:Completed 29600 out of 80000 steps  (37%)
23:35:04:WU02:FS02:0x18:Completed 1350000 out of 2500000 steps (54%)
23:35:10:WU00:FS00:0xa4:Completed 30400 out of 80000 steps  (38%)
23:35:44:WU00:FS00:0xa4:Completed 31200 out of 80000 steps  (39%)
23:36:17:WU00:FS00:0xa4:Completed 32000 out of 80000 steps  (40%)
23:36:32:WU01:FS01:0x18:Completed 1550000 out of 2500000 steps (62%)
23:36:51:WU00:FS00:0xa4:Completed 32800 out of 80000 steps  (41%)
23:37:25:WU00:FS00:0xa4:Completed 33600 out of 80000 steps  (42%)
23:37:58:WU00:FS00:0xa4:Completed 34400 out of 80000 steps  (43%)
23:38:02:WU02:FS02:0x18:Completed 1375000 out of 2500000 steps (55%)
23:38:31:WU01:FS01:0x18:Completed 1575000 out of 2500000 steps (63%)
23:38:32:WU00:FS00:0xa4:Completed 35200 out of 80000 steps  (44%)
23:39:05:WU00:FS00:0xa4:Completed 36000 out of 80000 steps  (45%)
23:39:39:WU00:FS00:0xa4:Completed 36800 out of 80000 steps  (46%)
23:40:13:WU00:FS00:0xa4:Completed 37600 out of 80000 steps  (47%)
23:40:30:WU01:FS01:0x18:Completed 1600000 out of 2500000 steps (64%)
23:40:31:100:127.0.0.1:New Web connection
23:40:49:WU00:FS00:0xa4:Completed 38400 out of 80000 steps  (48%)
23:41:01:WU02:FS02:0x18:Completed 1400000 out of 2500000 steps (56%)
23:41:24:WU00:FS00:0xa4:Completed 39200 out of 80000 steps  (49%)
23:41:59:WU00:FS00:0xa4:Completed 40000 out of 80000 steps  (50%)
23:42:34:WU00:FS00:0xa4:Completed 40800 out of 80000 steps  (51%)
23:42:44:WU01:FS01:0x18:Completed 1625000 out of 2500000 steps (65%)
23:43:08:WU00:FS00:0xa4:Completed 41600 out of 80000 steps  (52%)
23:43:42:WU00:FS00:0xa4:Completed 42400 out of 80000 steps  (53%)
23:44:16:WU00:FS00:0xa4:Completed 43200 out of 80000 steps  (54%)
23:44:19:WU02:FS02:0x18:Completed 1425000 out of 2500000 steps (57%)

Ricky · Post by **Ricky** » Wed Aug 12, 2015 12:06 pm

OK, I finally found the log data that is relevant to the cleanup problem. Apparently, my appdata folder is hidden, so it was hard to find. I removed some of the progress lines to reduce the size of the log being posted. I have a suspicion that I need to reboot windows periodically to clear memory. At the time the problem occurred, windows had been running over a week. What do you think?

Code: Select all

09:44:56:WU00:FS02:0x17:*********************** Log Started 2015-08-10T09:44:56Z ***********************
09:44:56:WU00:FS02:0x17:Project: 9411 (Run 358, Clone 3, Gen 0)
09:44:56:WU00:FS02:0x17:Unit: 0x00000002ab4041385548205fe5bc7f33
09:44:56:WU00:FS02:0x17:CPU: 0x00000000000000000000000000000000
09:44:56:WU00:FS02:0x17:Machine: 2
09:44:56:WU00:FS02:0x17:Reading tar file system.xml
09:44:56:WU00:FS02:0x17:Reading tar file integrator.xml
09:44:56:WU00:FS02:0x17:Reading tar file state.xml
09:44:56:WU00:FS02:0x17:Reading tar file core.xml
09:44:56:WU00:FS02:0x17:Digital signatures verified
09:44:56:WU00:FS02:0x17:Folding@home GPU core17
09:44:56:WU00:FS02:0x17:Version 0.0.52
09:45:03:WU02:FS01:0x18:Completed 4250000 out of 5000000 steps (85%)
09:45:06:WU01:FS02:Upload complete
09:45:07:WU01:FS02:Server responded WORK_ACK (400)
09:45:07:WU01:FS02:Final credit estimate, 29015.00 points
09:45:07:WU01:FS02:Cleaning up
09:45:18:WU00:FS02:0x17:Completed 0 out of 16000000 steps (0%)
09:45:18:WU00:FS02:0x17:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
10:35:09:WU02:FS01:0x18:Completed 4750000 out of 5000000 steps (95%)
10:35:12:WU03:FS00:0xa4:Completed 485000 out of 500000 steps  (97%)
10:37:06:WU03:FS00:0xa4:Completed 490000 out of 500000 steps  (98%)
10:37:40:WU00:FS02:0x17:Completed 1280000 out of 16000000 steps (8%)
10:39:14:WU03:FS00:0xa4:Completed 495000 out of 500000 steps  (99%)
10:39:15:WU01:FS00:Connecting to 171.67.108.200:8080
10:39:15:WU01:FS00:Assigned to work server 171.64.65.99
10:39:15:WU01:FS00:Requesting new work unit for slot 00: RUNNING cpu:30 from 171.64.65.99
10:39:15:WU01:FS00:Connecting to 171.64.65.99:8080
10:39:18:WU01:FS00:Downloading 6.22MiB
10:39:21:WU01:FS00:Download complete
10:39:21:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9752 run:388 clone:0 gen:344 core:0xa4 unit:0x000001d1ab4041635541721fa7d25de8
10:40:23:WU02:FS01:0x18:Completed 4800000 out of 5000000 steps (96%)
10:41:37:WU03:FS00:0xa4:Completed 500000 out of 500000 steps  (100%)
10:41:38:WU03:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
10:41:48:WU03:FS00:0xa4:
10:41:48:WU03:FS00:0xa4:Finished Work Unit:
10:41:48:WU03:FS00:0xa4:- Reading up to 6842952 from "03/wudata_01.trr": Read 6842952
10:41:48:WU03:FS00:0xa4:trr file hash check passed.
10:41:48:WU03:FS00:0xa4:- Reading up to 7426028 from "03/wudata_01.xtc": Read 7426028
10:41:48:WU03:FS00:0xa4:xtc file hash check passed.
10:41:48:WU03:FS00:0xa4:edr file hash check passed.
10:41:48:WU03:FS00:0xa4:logfile size: 351055
10:41:48:WU03:FS00:0xa4:Leaving Run
10:41:49:WU03:FS00:0xa4:- Writing 14653063 bytes of core data to disk...
10:41:52:WU03:FS00:0xa4:Done: 14652551 -> 13954717 (compressed to 95.2 percent)
10:41:52:WU03:FS00:0xa4:  ... Done.
10:41:53:WU03:FS00:0xa4:- Shutting down core
10:41:53:WU03:FS00:0xa4:
10:41:53:WU03:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
10:41:53:WU03:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
10:41:53:WU03:FS00:Sending unit results: id:03 state:SEND error:NO_ERROR project:7520 run:23 clone:14 gen:20 core:0xa4 unit:0x0000001ffbcb017d54ebbafeb491236e
10:41:53:WU03:FS00:Uploading 13.31MiB to 128.143.199.97
10:41:53:WU03:FS00:Connecting to 128.143.199.97:8080
10:41:53:WU01:FS00:Starting
10:41:53:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 704 -lifeline 5160 -checkpoint 15 -np 30
10:41:53:WU01:FS00:Started FahCore on PID 9072
10:41:53:WU01:FS00:Core PID:8520
10:41:53:WU01:FS00:FahCore 0xa4 started
10:41:54:WU01:FS00:0xa4:
10:41:54:WU01:FS00:0xa4:*------------------------------*
10:41:54:WU01:FS00:0xa4:Folding@Home Gromacs GB Core
10:41:54:WU01:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
10:41:54:WU01:FS00:0xa4:
10:41:54:WU01:FS00:0xa4:Preparing to commence simulation
10:41:54:WU01:FS00:0xa4:- Looking at optimizations...
10:41:54:WU01:FS00:0xa4:- Created dyn
10:41:54:WU01:FS00:0xa4:- Files status OK
10:41:56:WU01:FS00:0xa4:- Expanded 6522617 -> 22366436 (decompressed 342.9 percent)
10:41:56:WU01:FS00:0xa4:Called DecompressByteArray: compressed_data_size=6522617 data_size=22366436, decompressed_data_size=22366436 diff=0
10:41:56:WU01:FS00:0xa4:- Digital signature verified
10:41:56:WU01:FS00:0xa4:
10:41:56:WU01:FS00:0xa4:Project: 9752 (Run 388, Clone 0, Gen 344)
10:41:56:WU01:FS00:0xa4:
10:41:56:WU01:FS00:0xa4:Assembly optimizations on if available.
10:41:56:WU01:FS00:0xa4:Entering M.D.
10:41:59:WU03:FS00:Upload 19.25%
10:42:03:WU01:FS00:0xa4:Mapping NT from 30 to 30 
10:42:04:WU01:FS00:0xa4:Completed 0 out of 80000 steps  (0%)
10:42:05:WU03:FS00:Upload 83.59%
10:42:09:WU03:FS00:Upload complete
10:42:09:WU03:FS00:Server responded WORK_ACK (400)
10:42:09:WU03:FS00:Final credit estimate, 11011.00 points
10:42:09:WU03:FS00:Cleaning up
10:42:13:ERROR:std::exception: bad allocation
10:42:45:WU01:FS00:0xa4:Completed 800 out of 80000 steps  (1%)
10:43:26:WU01:FS00:0xa4:Completed 1600 out of 80000 steps  (2%)
10:44:07:WU01:FS00:0xa4:Completed 2400 out of 80000 steps  (3%)
10:54:54:WU01:FS00:0xa4:Completed 17600 out of 80000 steps  (22%)
10:55:19:WU02:FS01:0x18:Completed 4950000 out of 5000000 steps (99%)
10:55:28:WU01:FS00:0xa4:Completed 18400 out of 80000 steps  (23%)
10:56:03:WU01:FS00:0xa4:Completed 19200 out of 80000 steps  (24%)
10:56:36:WU01:FS00:0xa4:Completed 20000 out of 80000 steps  (25%)
10:57:10:WU01:FS00:0xa4:Completed 20800 out of 80000 steps  (26%)
10:57:17:WU00:FS02:0x17:Completed 1760000 out of 16000000 steps (11%)
10:57:44:WU01:FS00:0xa4:Completed 21600 out of 80000 steps  (27%)
10:58:17:WU01:FS00:0xa4:Completed 22400 out of 80000 steps  (28%)
10:58:51:WU01:FS00:0xa4:Completed 23200 out of 80000 steps  (29%)
10:59:25:WU01:FS00:0xa4:Completed 24000 out of 80000 steps  (30%)
10:59:59:WU01:FS00:0xa4:Completed 24800 out of 80000 steps  (31%)
11:00:10:WU02:FS01:0x18:Completed 5000000 out of 5000000 steps (100%)
11:00:11:WU03:FS01:Connecting to 171.67.108.200:80
11:00:11:WU03:FS01:Assigned to work server 140.163.4.234
11:00:11:WU03:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GM204 [GeForce GTX 980] from 140.163.4.234
11:00:11:WU03:FS01:Connecting to 140.163.4.234:8080
11:00:12:WU03:FS01:Downloading 3.27MiB
11:00:14:WU03:FS01:Download complete
11:00:14:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:10478 run:1 clone:25 gen:89 core:0x18 unit:0x0000007e538b3dba548f6e0c7873c310
11:00:33:WU01:FS00:0xa4:Completed 25600 out of 80000 steps  (32%)
11:00:34:WU02:FS01:0x18:Saving result file logfile_01.txt
11:00:34:WU02:FS01:0x18:Saving result file checkpointState.xml
11:00:38:WU02:FS01:0x18:Saving result file checkpt.crc
11:00:38:WU02:FS01:0x18:Saving result file log.txt
11:00:38:WU02:FS01:0x18:Saving result file positions.xtc
11:00:42:WU02:FS01:0x18:Folding@home Core Shutdown: FINISHED_UNIT
11:00:42:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
11:00:42:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:10486 run:0 clone:96 gen:64 core:0x18 unit:0x00000063538b3dbb54aecba5860d0ecb
11:00:42:WU02:FS01:Uploading 13.10MiB to 140.163.4.235
11:00:42:WU02:FS01:Connecting to 140.163.4.235:8080
11:00:42:WU03:FS01:Starting
11:00:42:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 03 -suffix 01 -version 704 -lifeline 5160 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
11:00:42:WU03:FS01:Started FahCore on PID 6516
11:00:42:WU03:FS01:Core PID:7420
11:00:42:WU03:FS01:FahCore 0x18 started
11:00:43:WU03:FS01:0x18:*********************** Log Started 2015-08-10T11:00:43Z ***********************
11:00:43:WU03:FS01:0x18:Project: 10478 (Run 1, Clone 25, Gen 89)
11:00:43:WU03:FS01:0x18:Unit: 0x0000007e538b3dba548f6e0c7873c310
11:00:43:WU03:FS01:0x18:CPU: 0x00000000000000000000000000000000
11:00:43:WU03:FS01:0x18:Machine: 1
11:00:43:WU03:FS01:0x18:Reading tar file state.xml
11:00:44:WU03:FS01:0x18:Reading tar file system.xml
11:00:45:WU03:FS01:0x18:Reading tar file integrator.xml
11:00:45:WU03:FS01:0x18:Reading tar file core.xml
11:00:45:WU03:FS01:0x18:Digital signatures verified
11:00:45:WU03:FS01:0x18:Folding@home GPU core18
11:00:45:WU03:FS01:0x18:Version 0.0.4
11:00:48:WU02:FS01:Upload 37.69%
11:01:04:WU02:FS01:Upload complete
11:01:04:WU02:FS01:Server responded WORK_ACK (400)
11:01:04:WU02:FS01:Final credit estimate, 93855.00 points
11:01:04:WU02:FS01:Cleaning up
11:01:06:WU01:FS00:0xa4:Completed 26400 out of 80000 steps  (33%)
11:01:17:WU03:FS01:0x18:Completed 0 out of 5000000 steps (0%)
11:01:17:WU03:FS01:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
11:01:39:WU01:FS00:0xa4:Completed 27200 out of 80000 steps  (34%)
11:02:13:WU01:FS00:0xa4:Completed 28000 out of 80000 steps  (35%)
11:28:24:WU01:FS00:0xa4:Completed 65600 out of 80000 steps  (82%)
11:28:58:WU01:FS00:0xa4:Completed 66400 out of 80000 steps  (83%)
******************************* Date: 2015-08-10 *******************************
11:29:32:WU01:FS00:0xa4:Completed 67200 out of 80000 steps  (84%)
11:30:03:WU00:FS02:0x17:Completed 2560000 out of 16000000 steps (16%)
11:30:05:WU01:FS00:0xa4:Completed 68000 out of 80000 steps  (85%)
11:30:28:WU03:FS01:0x18:Completed 400000 out of 5000000 steps (8%)
11:37:21:WU01:FS00:0xa4:Completed 78400 out of 80000 steps  (98%)
11:37:33:WU03:FS01:0x18:Completed 500000 out of 5000000 steps (10%)
11:37:54:WU01:FS00:0xa4:Completed 79200 out of 80000 steps  (99%)
11:37:55:WU02:FS00:Connecting to 171.67.108.200:8080
11:37:55:WU02:FS00:Assigned to work server 171.64.65.99
11:37:55:WU02:FS00:Requesting new work unit for slot 00: RUNNING cpu:30 from 171.64.65.99
11:37:55:WU02:FS00:Connecting to 171.64.65.99:8080
11:37:59:WU02:FS00:Downloading 6.22MiB
11:38:02:WU02:FS00:Download complete
11:38:03:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:9752 run:430 clone:0 gen:332 core:0xa4 unit:0x000001ccab4041635541722c120a1ec8
11:38:28:WU01:FS00:0xa4:Completed 80000 out of 80000 steps  (100%)
11:38:30:WU01:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
11:38:40:WU01:FS00:0xa4:
11:38:40:WU01:FS00:0xa4:Finished Work Unit:
11:38:40:WU01:FS00:0xa4:- Reading up to 4117896 from "01/wudata_01.trr": Read 4117896
11:38:40:WU01:FS00:0xa4:trr file hash check passed.
11:38:40:WU01:FS00:0xa4:- Reading up to 3189772 from "01/wudata_01.xtc": Read 3189772
11:38:40:WU01:FS00:0xa4:xtc file hash check passed.
11:38:40:WU01:FS00:0xa4:edr file hash check passed.
11:38:40:WU01:FS00:0xa4:logfile size: 19899
11:38:40:WU01:FS00:0xa4:Leaving Run
11:38:40:WU01:FS00:0xa4:- Writing 7329959 bytes of core data to disk...
11:38:42:WU01:FS00:0xa4:Done: 7329447 -> 7058781 (compressed to 96.3 percent)
11:38:42:WU01:FS00:0xa4:  ... Done.
11:38:43:WU01:FS00:0xa4:- Shutting down core
11:38:43:WU01:FS00:0xa4:
11:38:43:WU01:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
11:38:44:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
11:38:44:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9752 run:388 clone:0 gen:344 core:0xa4 unit:0x000001d1ab4041635541721fa7d25de8
11:38:44:WU01:FS00:Uploading 6.73MiB to 171.64.65.99
11:38:44:WU02:FS00:Starting
11:38:44:WU01:FS00:Connecting to 171.64.65.99:8080
11:38:44:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 02 -suffix 01 -version 704 -lifeline 5160 -checkpoint 15 -np 30
11:38:44:WU02:FS00:Started FahCore on PID 7860
11:38:44:WU02:FS00:Core PID:8096
11:38:44:WU02:FS00:FahCore 0xa4 started
11:38:44:WU02:FS00:0xa4:
11:38:44:WU02:FS00:0xa4:*------------------------------*
11:38:44:WU02:FS00:0xa4:Folding@Home Gromacs GB Core
11:38:44:WU02:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
11:38:44:WU02:FS00:0xa4:
11:38:44:WU02:FS00:0xa4:Preparing to commence simulation
11:38:44:WU02:FS00:0xa4:- Looking at optimizations...
11:38:44:WU02:FS00:0xa4:- Created dyn
11:38:44:WU02:FS00:0xa4:- Files status OK
11:38:47:WU02:FS00:0xa4:- Expanded 6523209 -> 22366436 (decompressed 342.8 percent)
11:38:47:WU02:FS00:0xa4:Called DecompressByteArray: compressed_data_size=6523209 data_size=22366436, decompressed_data_size=22366436 diff=0
11:38:47:WU02:FS00:0xa4:- Digital signature verified
11:38:47:WU02:FS00:0xa4:
11:38:47:WU02:FS00:0xa4:Project: 9752 (Run 430, Clone 0, Gen 332)
11:38:47:WU02:FS00:0xa4:
11:38:47:WU02:FS00:0xa4:Assembly optimizations on if available.
11:38:47:WU02:FS00:0xa4:Entering M.D.
11:38:51:WU01:FS00:Upload complete
11:38:51:WU01:FS00:Server responded WORK_ACK (400)
11:38:51:WU01:FS00:Final credit estimate, 4257.00 points
11:38:51:WU01:FS00:Cleaning up
11:38:51:ERROR:WU01:FS00:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\01\wudata_01.tpr"
11:38:51:WU01:FS00:Cleaning up
11:38:51:ERROR:WU01:FS00:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\01\wudata_01.tpr"
11:38:53:WU02:FS00:0xa4:Mapping NT from 30 to 30 
11:38:55:WU02:FS00:0xa4:Completed 0 out of 80000 steps  (0%)
11:39:28:WU02:FS00:0xa4:Completed 800 out of 80000 steps  (1%)
11:39:51:WU01:FS00:Cleaning up
11:39:51:ERROR:WU01:FS00:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\01\wudata_01.tpr"
11:40:01:WU02:FS00:0xa4:Completed 1600 out of 80000 steps  (2%)
11:40:35:WU02:FS00:0xa4:Completed 2400 out of 80000 steps  (3%)
11:41:08:WU02:FS00:0xa4:Completed 3200 out of 80000 steps  (4%)
11:41:17:WU03:FS01:0x18:Completed 550000 out of 5000000 steps (11%)
11:41:28:WU01:FS00:Cleaning up
11:41:28:ERROR:WU01:FS00:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\01\wudata_01.tpr"
11:41:41:WU02:FS00:0xa4:Completed 4000 out of 80000 steps  (5%)
11:42:15:WU02:FS00:0xa4:Completed 4800 out of 80000 steps  (6%)
11:42:49:WU02:FS00:0xa4:Completed 5600 out of 80000 steps  (7%)
11:43:07:WU00:FS02:0x17:Completed 2880000 out of 16000000 steps (18%)
11:43:22:WU02:FS00:0xa4:Completed 6400 out of 80000 steps  (8%)
11:43:56:WU02:FS00:0xa4:Completed 7200 out of 80000 steps  (9%)
11:44:06:WU01:FS00:Cleaning up
11:44:06:ERROR:WU01:FS00:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\01\wudata_01.tpr"
11:44:29:WU02:FS00:0xa4:Completed 8000 out of 80000 steps  (10%)
11:44:49:WU03:FS01:0x18:Completed 600000 out of 5000000 steps (12%)
11:45:02:WU02:FS00:0xa4:Completed 8800 out of 80000 steps  (11%)
11:45:35:WU02:FS00:0xa4:Completed 9600 out of 80000 steps  (12%)
11:46:08:WU02:FS00:0xa4:Completed 10400 out of 80000 steps  (13%)
11:46:43:WU02:FS00:0xa4:Completed 11200 out of 80000 steps  (14%)
11:47:17:WU02:FS00:0xa4:Completed 12000 out of 80000 steps  (15%)
11:47:50:WU02:FS00:0xa4:Completed 12800 out of 80000 steps  (16%)
11:48:20:WU01:FS00:Cleaning up
11:48:20:ERROR:WU01:FS00:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\01\wudata_01.tpr"
11:48:23:WU02:FS00:0xa4:Completed 13600 out of 80000 steps  (17%)
11:53:57:WU02:FS00:0xa4:Completed 21600 out of 80000 steps  (27%)
11:54:30:WU02:FS00:0xa4:Completed 22400 out of 80000 steps  (28%)
11:55:04:WU02:FS00:0xa4:Completed 23200 out of 80000 steps  (29%)
11:55:11:WU01:FS00:Cleaning up
11:55:11:ERROR:WU01:FS00:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\01\wudata_01.tpr"
11:55:37:WU02:FS00:0xa4:Completed 24000 out of 80000 steps  (30%)
11:55:38:WU03:FS01:0x18:Completed 750000 out of 5000000 steps (15%)
11:56:11:WU00:FS02:0x17:Completed 3200000 out of 16000000 steps (20%)
11:56:11:WU02:FS00:0xa4:Completed 24800 out of 80000 steps  (31%)
12:05:07:WU02:FS00:0xa4:Completed 37600 out of 80000 steps  (47%)
12:05:40:WU02:FS00:0xa4:Completed 38400 out of 80000 steps  (48%)
12:06:14:WU02:FS00:0xa4:Completed 39200 out of 80000 steps  (49%)
12:06:17:WU01:FS00:Cleaning up
12:06:17:ERROR:WU01:FS00:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\01\wudata_01.tpr"
12:06:40:WU03:FS01:0x18:Completed 900000 out of 5000000 steps (18%)
12:09:34:WU02:FS00:0xa4:Completed 44000 out of 80000 steps  (55%)
12:10:07:WU02:FS00:0xa4:Completed 44800 out of 80000 steps  (56%)
12:10:11:WU03:FS01:0x18:Completed 950000 out of 5000000 steps (19%)
12:10:41:WU02:FS00:0xa4:Completed 45600 out of 80000 steps  (57%)
12:11:14:WU02:FS00:0xa4:Completed 46400 out of 80000 steps  (58%)
12:22:21:WU02:FS00:0xa4:Completed 62400 out of 80000 steps  (78%)
12:22:24:WU00:FS02:0x17:Completed 3840000 out of 16000000 steps (24%)
12:22:54:WU02:FS00:0xa4:Completed 63200 out of 80000 steps  (79%)
12:23:27:WU02:FS00:0xa4:Completed 64000 out of 80000 steps  (80%)
12:24:00:WU02:FS00:0xa4:Completed 64800 out of 80000 steps  (81%)
12:24:14:WU01:FS00:Cleaning up
12:24:14:ERROR:WU01:FS00:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\01\wudata_01.tpr"
12:24:33:WU02:FS00:0xa4:Completed 65600 out of 80000 steps  (82%)
12:24:44:WU03:FS01:0x18:Completed 1150000 out of 5000000 steps (23%)
12:25:07:WU02:FS00:0xa4:Completed 66400 out of 80000 steps  (83%)
12:32:21:WU02:FS00:0xa4:Completed 76800 out of 80000 steps  (96%)
12:32:55:WU02:FS00:0xa4:Completed 77600 out of 80000 steps  (97%)
12:33:28:WU02:FS00:0xa4:Completed 78400 out of 80000 steps  (98%)
12:34:02:WU02:FS00:0xa4:Completed 79200 out of 80000 steps  (99%)
12:34:03:WU04:FS00:Connecting to 171.67.108.200:8080
12:34:03:WU04:FS00:Assigned to work server 171.64.65.99
12:34:03:WU04:FS00:Requesting new work unit for slot 00: RUNNING cpu:30 from 171.64.65.99
12:34:03:WU04:FS00:Connecting to 171.64.65.99:8080
12:34:07:WU04:FS00:Downloading 6.24MiB
12:34:10:WU04:FS00:Download complete
12:34:10:WU04:FS00:Received Unit: id:04 state:DOWNLOAD error:NO_ERROR project:9752 run:1672 clone:0 gen:150 core:0xa4 unit:0x000000d2ab404163554173e0795808de
12:34:35:WU02:FS00:0xa4:Completed 80000 out of 80000 steps  (100%)
12:34:38:WU02:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
12:34:48:WU02:FS00:0xa4:
12:34:48:WU02:FS00:0xa4:Finished Work Unit:
12:34:48:WU02:FS00:0xa4:- Reading up to 4117896 from "02/wudata_01.trr": Read 4117896
12:34:48:WU02:FS00:0xa4:trr file hash check passed.
12:34:48:WU02:FS00:0xa4:- Reading up to 3189964 from "02/wudata_01.xtc": Read 3189964
12:34:48:WU02:FS00:0xa4:xtc file hash check passed.
12:34:48:WU02:FS00:0xa4:edr file hash check passed.
12:34:48:WU02:FS00:0xa4:logfile size: 19891
12:34:48:WU02:FS00:0xa4:Leaving Run
12:34:52:WU02:FS00:0xa4:- Writing 7330143 bytes of core data to disk...
12:34:53:WU02:FS00:0xa4:Done: 7329631 -> 7059348 (compressed to 96.3 percent)
12:34:53:WU02:FS00:0xa4:  ... Done.
12:34:55:WU02:FS00:0xa4:- Shutting down core
12:34:55:WU02:FS00:0xa4:
12:34:55:WU02:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
12:34:55:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
12:34:55:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:9752 run:430 clone:0 gen:332 core:0xa4 unit:0x000001ccab4041635541722c120a1ec8
12:34:55:WU02:FS00:Uploading 6.73MiB to 171.64.65.99
12:34:55:WU02:FS00:Connecting to 171.64.65.99:8080
12:34:55:WU04:FS00:Starting
12:34:55:WU04:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 04 -suffix 01 -version 704 -lifeline 5160 -checkpoint 15 -np 30
12:34:55:WU04:FS00:Started FahCore on PID 5980
12:34:55:WU04:FS00:Core PID:6772
12:34:55:WU04:FS00:FahCore 0xa4 started
12:34:56:WU04:FS00:0xa4:
12:34:56:WU04:FS00:0xa4:*------------------------------*
12:34:56:WU04:FS00:0xa4:Folding@Home Gromacs GB Core
12:34:56:WU04:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
12:34:56:WU04:FS00:0xa4:
12:34:56:WU04:FS00:0xa4:Preparing to commence simulation
12:34:56:WU04:FS00:0xa4:- Looking at optimizations...
12:34:56:WU04:FS00:0xa4:- Created dyn
12:34:56:WU04:FS00:0xa4:- Files status OK
12:34:58:WU04:FS00:0xa4:- Expanded 6544483 -> 22464536 (decompressed 343.2 percent)
12:34:58:WU04:FS00:0xa4:Called DecompressByteArray: compressed_data_size=6544483 data_size=22464536, decompressed_data_size=22464536 diff=0
12:34:58:WU04:FS00:0xa4:- Digital signature verified
12:34:58:WU04:FS00:0xa4:
12:34:58:WU04:FS00:0xa4:Project: 9752 (Run 1672, Clone 0, Gen 150)
12:34:58:WU04:FS00:0xa4:
12:34:58:WU04:FS00:0xa4:Assembly optimizations on if available.
12:34:58:WU04:FS00:0xa4:Entering M.D.
12:35:02:WU02:FS00:Upload complete
12:35:02:WU02:FS00:Server responded WORK_ACK (400)
12:35:02:WU02:FS00:Final credit estimate, 4349.00 points
12:35:02:WU02:FS00:Cleaning up
12:35:02:ERROR:WU02:FS00:Exception: Failed to remove directory './work/02': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\02\wudata_01.tpr"
12:35:02:WU02:FS00:Cleaning up
12:35:02:ERROR:WU02:FS00:Exception: Failed to remove directory './work/02': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\02\wudata_01.tpr"
12:35:04:WU04:FS00:0xa4:Mapping NT from 30 to 30 
12:35:07:WU04:FS00:0xa4:Completed 0 out of 80000 steps  (0%)
12:35:32:WU03:FS01:0x18:Completed 1300000 out of 5000000 steps (26%)
12:35:32:WU00:FS02:0x17:Completed 4160000 out of 16000000 steps (26%)
12:35:43:WU04:FS00:0xa4:Completed 800 out of 80000 steps  (1%)
12:36:02:WU02:FS00:Cleaning up
12:36:02:ERROR:WU02:FS00:Exception: Failed to remove directory './work/02': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\02\wudata_01.tpr"
12:36:19:WU04:FS00:0xa4:Completed 1600 out of 80000 steps  (2%)
12:36:56:WU04:FS00:0xa4:Completed 2400 out of 80000 steps  (3%)
12:37:33:WU04:FS00:0xa4:Completed 3200 out of 80000 steps  (4%)
12:37:40:WU02:FS00:Cleaning up
12:37:40:ERROR:WU02:FS00:Exception: Failed to remove directory './work/02': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\02\wudata_01.tpr"
12:38:09:WU04:FS00:0xa4:Completed 4000 out of 80000 steps  (5%)
12:38:45:WU04:FS00:0xa4:Completed 4800 out of 80000 steps  (6%)
12:39:10:WU03:FS01:0x18:Completed 1350000 out of 5000000 steps (27%)
12:39:20:WU04:FS00:0xa4:Completed 5600 out of 80000 steps  (7%)
12:39:56:WU04:FS00:0xa4:Completed 6400 out of 80000 steps  (8%)
12:40:17:WU02:FS00:Cleaning up
12:40:17:ERROR:WU02:FS00:Exception: Failed to remove directory './work/02': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\02\wudata_01.tpr"
12:40:32:WU04:FS00:0xa4:Completed 7200 out of 80000 steps  (9%)
12:41:08:WU04:FS00:0xa4:Completed 8000 out of 80000 steps  (10%)
12:43:32:WU04:FS00:0xa4:Completed 11200 out of 80000 steps  (14%)
12:44:08:WU04:FS00:0xa4:Completed 12000 out of 80000 steps  (15%)
12:44:31:WU02:FS00:Cleaning up
12:44:31:ERROR:WU02:FS00:Exception: Failed to remove directory './work/02': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\02\wudata_01.tpr"
12:44:44:WU04:FS00:0xa4:Completed 12800 out of 80000 steps  (16%)
12:47:46:WU04:FS00:0xa4:Completed 16800 out of 80000 steps  (21%)
12:48:23:1835:127.0.0.1:New Web connection
12:48:26:WU04:FS00:0xa4:Completed 17600 out of 80000 steps  (22%)
12:50:51:WU04:FS00:0xa4:Completed 20800 out of 80000 steps  (26%)
12:51:22:WU02:FS00:Cleaning up
12:51:22:ERROR:WU02:FS00:Exception: Failed to remove directory './work/02': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\02\wudata_01.tpr"
12:51:28:WU04:FS00:0xa4:Completed 21600 out of 80000 steps  (27%)
12:52:05:WU04:FS00:0xa4:Completed 22400 out of 80000 steps  (28%)
12:52:44:WU04:FS00:0xa4:Completed 23200 out of 80000 steps  (29%)
12:53:16:WU01:FS00:Cleaning up
12:53:16:ERROR:WU01:FS00:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\01\wudata_01.tpr"
12:53:22:WU04:FS00:0xa4:Completed 24000 out of 80000 steps  (30%)
12:53:59:WU04:FS00:0xa4:Completed 24800 out of 80000 steps  (31%)
113:01:44:WU00:FS02:0x17:Completed 4800000 out of 16000000 steps (30%)
13:01:55:WU04:FS00:0xa4:Completed 35200 out of 80000 steps  (44%)
13:02:28:WU02:FS00:Cleaning up
13:02:28:ERROR:WU02:FS00:Exception: Failed to remove directory './work/02': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\02\wudata_01.tpr"
13:02:32:WU04:FS00:0xa4:Completed 36000 out of 80000 steps  (45%)
13:20:15:WU03:FS01:0x18:Completed 1900000 out of 5000000 steps (38%)
13:20:16:WU04:FS00:0xa4:Completed 59200 out of 80000 steps  (74%)
13:20:25:WU02:FS00:Cleaning up
13:20:25:ERROR:WU02:FS00:Exception: Failed to remove directory './work/02': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\02\wudata_01.tpr"
13:20:52:WU04:FS00:0xa4:Completed 60000 out of 80000 steps  (75%)
13:21:28:WU04:FS00:0xa4:Completed 60800 out of 80000 steps  (76%)
13:34:05:WU04:FS00:0xa4:Completed 77600 out of 80000 steps  (97%)
13:34:38:WU00:FS02:0x17:Completed 5600000 out of 16000000 steps (35%)
13:34:41:WU04:FS00:0xa4:Completed 78400 out of 80000 steps  (98%)
13:34:59:WU03:FS01:0x18:Completed 2100000 out of 5000000 steps (42%)
13:35:17:WU04:FS00:0xa4:Completed 79200 out of 80000 steps  (99%)
13:35:18:WU05:FS00:Connecting to 171.67.108.200:8080
13:35:19:WU05:FS00:Assigned to work server 128.143.199.97
13:35:19:WU05:FS00:Requesting new work unit for slot 00: RUNNING cpu:30 from 128.143.199.97
13:35:19:WU05:FS00:Connecting to 128.143.199.97:8080
13:35:20:WU05:FS00:Downloading 2.23MiB
13:35:22:WU05:FS00:Download complete
13:35:23:WU05:FS00:Received Unit: id:05 state:DOWNLOAD error:NO_ERROR project:7520 run:34 clone:32 gen:64 core:0xa4 unit:0x00000061fbcb017d54ebbafed7eef861
13:35:53:WU04:FS00:0xa4:Completed 80000 out of 80000 steps  (100%)
13:35:55:WU04:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
13:36:05:WU04:FS00:0xa4:
13:36:05:WU04:FS00:0xa4:Finished Work Unit:
13:36:05:WU04:FS00:0xa4:- Reading up to 4126776 from "04/wudata_01.trr": Read 4126776
13:36:05:WU04:FS00:0xa4:trr file hash check passed.
13:36:05:WU04:FS00:0xa4:- Reading up to 3196844 from "04/wudata_01.xtc": Read 3196844
13:36:05:WU04:FS00:0xa4:xtc file hash check passed.
13:36:05:WU04:FS00:0xa4:edr file hash check passed.
13:36:05:WU04:FS00:0xa4:logfile size: 19953
13:36:05:WU04:FS00:0xa4:Leaving Run
13:36:10:WU04:FS00:0xa4:- Writing 7345965 bytes of core data to disk...
13:36:11:WU04:FS00:0xa4:Done: 7345453 -> 7073390 (compressed to 96.2 percent)
13:36:11:WU04:FS00:0xa4:  ... Done.
13:36:13:WU04:FS00:0xa4:- Shutting down core
13:36:13:WU04:FS00:0xa4:
13:36:13:WU04:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
13:36:13:WU04:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
13:36:13:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:9752 run:1672 clone:0 gen:150 core:0xa4 unit:0x000000d2ab404163554173e0795808de
13:36:13:WU04:FS00:Uploading 6.75MiB to 171.64.65.99
13:36:13:WU04:FS00:Connecting to 171.64.65.99:8080
13:36:14:WU05:FS00:Starting
13:36:14:WU05:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 05 -suffix 01 -version 704 -lifeline 5160 -checkpoint 15 -np 30
13:36:14:WU05:FS00:Started FahCore on PID 6316
13:36:14:WU05:FS00:Core PID:7364
13:36:14:WU05:FS00:FahCore 0xa4 started
13:36:14:WU05:FS00:0xa4:
13:36:14:WU05:FS00:0xa4:*------------------------------*
13:36:14:WU05:FS00:0xa4:Folding@Home Gromacs GB Core
13:36:14:WU05:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
13:36:14:WU05:FS00:0xa4:
13:36:14:WU05:FS00:0xa4:Preparing to commence simulation
13:36:14:WU05:FS00:0xa4:- Looking at optimizations...
13:36:14:WU05:FS00:0xa4:- Created dyn
13:36:14:WU05:FS00:0xa4:- Files status OK
13:36:14:WU05:FS00:0xa4:- Expanded 2337512 -> 3301368 (decompressed 141.2 percent)
13:36:14:WU05:FS00:0xa4:Called DecompressByteArray: compressed_data_size=2337512 data_size=3301368, decompressed_data_size=3301368 diff=0
13:36:14:WU05:FS00:0xa4:- Digital signature verified
13:36:14:WU05:FS00:0xa4:
13:36:14:WU05:FS00:0xa4:Project: 7520 (Run 34, Clone 32, Gen 64)
13:36:14:WU05:FS00:0xa4:
13:36:14:WU05:FS00:0xa4:Assembly optimizations on if available.
13:36:14:WU05:FS00:0xa4:Entering M.D.
13:36:20:WU05:FS00:0xa4:Mapping NT from 30 to 30 
13:36:21:WU05:FS00:0xa4:Completed 0 out of 500000 steps  (0%)
13:36:22:WU04:FS00:Upload complete
13:36:22:WU04:FS00:Server responded WORK_ACK (400)
13:36:22:WU04:FS00:Final credit estimate, 4163.00 points
13:36:22:WU04:FS00:Cleaning up
13:36:22:ERROR:WU04:FS00:Exception: Failed to remove directory './work/04': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\04\wudata_01.tpr"
13:36:22:WU04:FS00:Cleaning up
13:36:22:ERROR:WU04:FS00:Exception: Failed to remove directory './work/04': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\04\wudata_01.tpr"
13:36:34:ERROR:FS00:
13:36:34:ERROR:FS00:-------------------------------------------------------
13:36:34:ERROR:FS00:Program Folding@home, VERSION 4.5.4
13:36:34:ERROR:FS00:Source code file: gromacs-4.5.4\src\gmxlib\smalloc.c, line: 171
13:36:34:ERROR:FS00:
13:36:34:ERROR:FS00:Fatal error:
13:36:34:ERROR:FS00:Not enough memory. Failed to calloc ld elements of size ld for 
13:36:34:ERROR:FS00:(called from file (null), line 24)
13:36:34:ERROR:FS00:For more information and tips for troubleshooting, please check the GROMACS
13:36:34:ERROR:FS00:website at http://www.gromacs.org/Documentation/Errors
13:36:34:ERROR:FS00:-------------------------------------------------------
13:36:34:ERROR:FS00:
13:36:34:ERROR:FS00:Thanx for Using GROMACS - Have a Nice Day
13:36:35:ERROR:FS00:
13:36:35:ERROR:FS00:-------------------------------------------------------
13:36:35:ERROR:FS00:Program Folding@home, VERSION 4.5.4
13:36:35:ERROR:FS00:Source code file: gromacs-4.5.4\src\gmxlib\smalloc.c, line: 171
13:36:35:ERROR:FS00:
13:36:35:ERROR:FS00:Fatal error:
13:36:35:ERROR:FS00:Not enough memory. Failed to calloc ld elements of size ld for 
13:36:35:ERROR:FS00:(called from file (null), line 24)
13:36:35:ERROR:FS00:For more information and tips for troubleshooting, please check the GROMACS
13:36:35:ERROR:FS00:website at http://www.gromacs.org/Documentation/Errors
13:36:35:ERROR:FS00:-------------------------------------------------------
13:36:35:ERROR:FS00:
13:36:35:ERROR:FS00:Thanx for Using GROMACS - Have a Nice Day
13:37:22:WU04:FS00:Cleaning up
13:37:22:ERROR:WU04:FS00:Exception: Failed to remove directory './work/04': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\04\wudata_01.tpr"
13:38:24:WU05:FS00:0xa4:Completed 5000 out of 500000 steps  (1%)
13:38:47:WU03:FS01:0x18:Completed 2150000 out of 5000000 steps (43%)
13:39:00:WU04:FS00:Cleaning up
13:39:00:ERROR:WU04:FS00:Exception: Failed to remove directory './work/04': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\04\wudata_01.tpr"
13:40:15:WU01:FS00:Cleaning up
13:40:15:ERROR:WU01:FS00:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\01\wudata_01.tpr"
13:40:37:WU05:FS00:0xa4:Completed 10000 out of 500000 steps  (2%)
13:41:23:WU00:FS02:0x17:Completed 5760000 out of 16000000 steps (36%)
13:41:37:WU04:FS00:Cleaning up
13:41:37:ERROR:WU04:FS00:Exception: Failed to remove directory './work/04': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\04\wudata_01.tpr"
13:42:24:WU03:FS01:0x18:Completed 2200000 out of 5000000 steps (44%)
13:42:51:WU05:FS00:0xa4:Completed 15000 out of 500000 steps  (3%)
13:45:05:WU05:FS00:0xa4:Completed 20000 out of 500000 steps  (4%)
13:45:51:WU04:FS00:Cleaning up
13:45:51:ERROR:WU04:FS00:Exception: Failed to remove directory './work/04': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\04\wudata_01.tpr"
13:46:01:WU03:FS01:0x18:Completed 2250000 out of 5000000 steps (45%)
13:47:08:WU05:FS00:0xa4:Completed 25000 out of 500000 steps  (5%)
13:48:11:WU00:FS02:0x17:Completed 5920000 out of 16000000 steps (37%)
13:49:06:WU05:FS00:0xa4:Completed 30000 out of 500000 steps  (6%)
13:49:27:WU02:FS00:Cleaning up
13:49:27:ERROR:WU02:FS00:Exception: Failed to remove directory './work/02': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\02\wudata_01.tpr"
13:49:50:WU03:FS01:0x18:Completed 2300000 out of 5000000 steps (46%)
13:51:05:WU05:FS00:0xa4:Completed 35000 out of 500000 steps  (7%)
13:52:43:WU04:FS00:Cleaning up
13:52:43:ERROR:WU04:FS00:Exception: Failed to remove directory './work/04': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\04\wudata_01.tpr"
13:53:02:WU05:FS00:0xa4:Completed 40000 out of 500000 steps  (8%)
13:53:28:WU03:FS01:0x18:Completed 2350000 out of 5000000 steps (47%)
14:01:17:WU00:FS02:0x17:Completed 6240000 out of 16000000 steps (39%)
14:02:51:WU05:FS00:0xa4:Completed 65000 out of 500000 steps  (13%)
14:03:48:WU04:FS00:Cleaning up
14:03:48:ERROR:WU04:FS00:Exception: Failed to remove directory './work/04': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\04\wudata_01.tpr"
14:04:32:WU03:FS01:0x18:Completed 2500000 out of 5000000 steps (50%)
14:21:03:WU00:FS02:0x17:Completed 6720000 out of 16000000 steps (42%)
14:21:45:WU04:FS00:Cleaning up
14:21:45:ERROR:WU04:FS00:Exception: Failed to remove directory './work/04': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\04\wudata_01.tpr"
14:22:32:WU05:FS00:0xa4:Completed 115000 out of 500000 steps  (23%)
14:23:00:WU03:FS01:0x18:Completed 2750000 out of 5000000 steps (55%)
14:34:20:WU05:FS00:0xa4:Completed 145000 out of 500000 steps  (29%)
14:36:19:WU05:FS00:0xa4:Completed 150000 out of 500000 steps  (30%)
14:36:26:WU02:FS00:Cleaning up
14:36:26:ERROR:WU02:FS00:Exception: Failed to remove directory './work/02': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\02\wudata_01.tpr"
14:37:52:WU03:FS01:0x18:Completed 2950000 out of 5000000 steps (59%)
14:48:54:WU03:FS01:0x18:Completed 3100000 out of 5000000 steps (62%)
14:50:07:WU05:FS00:0xa4:Completed 185000 out of 500000 steps  (37%)
14:50:47:WU04:FS00:Cleaning up
14:50:47:ERROR:WU04:FS00:Exception: Failed to remove directory './work/04': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\04\wudata_01.tpr"
14:52:04:WU05:FS00:0xa4:Completed 190000 out of 500000 steps  (38%)
14:52:42:WU03:FS01:0x18:Completed 3150000 out of 5000000 steps (63%)
14:54:00:WU00:FS02:0x17:Completed 7520000 out of 16000000 steps (47%)
14:54:02:WU05:FS00:0xa4:Completed 195000 out of 500000 steps  (39%)
14:56:01:WU05:FS00:0xa4:Completed 200000 out of 500000 steps  (40%)
14:56:16:WU01:FS00:Cleaning up
14:56:16:ERROR:WU01:FS00:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\01\wudata_01.tpr"
14:56:19:WU03:FS01:0x18:Completed 3200000 out of 5000000 steps (64%)
15:18:23:WU03:FS01:0x18:Completed 3500000 out of 5000000 steps (70%)
15:19:40:WU05:FS00:0xa4:Completed 260000 out of 500000 steps  (52%)
15:35:28:WU05:FS00:0xa4:Completed 300000 out of 500000 steps  (60%)
15:36:52:WU03:FS01:0x18:Completed 3750000 out of 5000000 steps (75%)
15:37:26:WU05:FS00:0xa4:Completed 305000 out of 500000 steps  (61%)
15:37:46:WU04:FS00:Cleaning up
15:37:46:ERROR:WU04:FS00:Exception: Failed to remove directory './work/04': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\04\wudata_01.tpr"
15:39:25:WU05:FS00:0xa4:Completed 310000 out of 500000 steps  (62%)
15:40:02:WU00:FS02:0x17:Completed 8640000 out of 16000000 steps (54%)
15:49:16:WU05:FS00:0xa4:Completed 335000 out of 500000 steps  (67%)
15:51:14:WU05:FS00:0xa4:Completed 340000 out of 500000 steps  (68%)
15:51:48:WU03:FS01:0x18:Completed 3950000 out of 5000000 steps (79%)
15:52:27:WU02:FS00:Cleaning up
15:52:27:ERROR:WU02:FS00:Exception: Failed to remove directory './work/02': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\02\wudata_01.tpr"
15:53:13:WU05:FS00:0xa4:Completed 345000 out of 500000 steps  (69%)
16:21:25:WU03:FS01:0x18:Completed 4350000 out of 5000000 steps (87%)
16:22:50:WU05:FS00:0xa4:Completed 420000 out of 500000 steps  (84%)
16:24:48:WU05:FS00:0xa4:Completed 425000 out of 500000 steps  (85%)
16:25:17:WU03:FS01:0x18:Completed 4400000 out of 5000000 steps (88%)
16:26:09:WU00:FS02:0x17:Completed 9760000 out of 16000000 steps (61%)
16:26:47:WU05:FS00:0xa4:Completed 430000 out of 500000 steps  (86%)
16:50:30:WU05:FS00:0xa4:Completed 490000 out of 500000 steps  (98%)
16:51:07:WU03:FS01:0x18:Completed 4750000 out of 5000000 steps (95%)
16:52:27:WU05:FS00:0xa4:Completed 495000 out of 500000 steps  (99%)
16:52:28:WU06:FS00:Connecting to 171.67.108.200:8080
16:52:29:WU06:FS00:Assigned to work server 128.143.199.97
16:52:29:WU06:FS00:Requesting new work unit for slot 00: RUNNING cpu:30 from 128.143.199.97
16:52:29:WU06:FS00:Connecting to 128.143.199.97:8080
16:52:30:WU06:FS00:Downloading 2.23MiB
16:52:32:WU00:FS02:0x17:Completed 10400000 out of 16000000 steps (65%)
16:52:32:WU06:FS00:Download complete
16:52:32:WU06:FS00:Received Unit: id:06 state:DOWNLOAD error:NO_ERROR project:7520 run:38 clone:16 gen:31 core:0xa4 unit:0x0000002bfbcb017d54ebbafe82325448
16:53:47:WU04:FS00:Cleaning up
16:53:47:ERROR:WU04:FS00:Exception: Failed to remove directory './work/04': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\04\wudata_01.tpr"
16:54:26:WU05:FS00:0xa4:Completed 500000 out of 500000 steps  (100%)
16:54:27:WU05:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
16:54:37:WU05:FS00:0xa4:
16:54:37:WU05:FS00:0xa4:Finished Work Unit:
16:54:37:WU05:FS00:0xa4:- Reading up to 6834672 from "05/wudata_01.trr": Read 6834672
16:54:37:WU05:FS00:0xa4:trr file hash check passed.
16:54:37:WU05:FS00:0xa4:- Reading up to 7413108 from "05/wudata_01.xtc": Read 7413108
16:54:37:WU05:FS00:0xa4:xtc file hash check passed.
16:54:37:WU05:FS00:0xa4:edr file hash check passed.
16:54:37:WU05:FS00:0xa4:logfile size: 371761
16:54:37:WU05:FS00:0xa4:Leaving Run
16:54:42:WU05:FS00:0xa4:- Writing 14652569 bytes of core data to disk...
16:54:45:WU05:FS00:0xa4:Done: 14652057 -> 13931372 (compressed to 95.0 percent)
16:54:45:WU05:FS00:0xa4:  ... Done.
16:54:45:WU05:FS00:0xa4:- Shutting down core
16:54:45:WU05:FS00:0xa4:
16:54:45:WU05:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
16:54:46:WU05:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
16:54:46:WU05:FS00:Sending unit results: id:05 state:SEND error:NO_ERROR project:7520 run:34 clone:32 gen:64 core:0xa4 unit:0x00000061fbcb017d54ebbafed7eef861
16:54:46:WU05:FS00:Uploading 13.29MiB to 128.143.199.97
16:54:46:WU06:FS00:Starting
16:54:46:WU05:FS00:Connecting to 128.143.199.97:8080
16:54:46:WU06:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 06 -suffix 01 -version 704 -lifeline 5160 -checkpoint 15 -np 30
16:54:46:WU06:FS00:Started FahCore on PID 1852
16:54:46:WU06:FS00:Core PID:6736
16:54:46:WU06:FS00:FahCore 0xa4 started
16:54:47:WU06:FS00:0xa4:
16:54:47:WU06:FS00:0xa4:*------------------------------*
16:54:47:WU06:FS00:0xa4:Folding@Home Gromacs GB Core
16:54:47:WU06:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
16:54:47:WU06:FS00:0xa4:
16:54:47:WU06:FS00:0xa4:Preparing to commence simulation
16:54:47:WU06:FS00:0xa4:- Looking at optimizations...
16:54:47:WU06:FS00:0xa4:- Created dyn
16:54:47:WU06:FS00:0xa4:- Files status OK
16:54:47:WU06:FS00:0xa4:- Expanded 2336019 -> 3300728 (decompressed 141.2 percent)
16:54:47:WU06:FS00:0xa4:Called DecompressByteArray: compressed_data_size=2336019 data_size=3300728, decompressed_data_size=3300728 diff=0
16:54:47:WU06:FS00:0xa4:- Digital signature verified
16:54:47:WU06:FS00:0xa4:
16:54:47:WU06:FS00:0xa4:Project: 7520 (Run 38, Clone 16, Gen 31)
16:54:47:WU06:FS00:0xa4:
16:54:47:WU06:FS00:0xa4:Assembly optimizations on if available.
16:54:47:WU06:FS00:0xa4:Entering M.D.
16:54:52:WU05:FS00:Upload 24.46%
16:54:53:WU06:FS00:0xa4:Mapping NT from 30 to 30 
16:54:53:WU06:FS00:0xa4:Completed 0 out of 500000 steps  (0%)
16:54:57:WU03:FS01:0x18:Completed 4800000 out of 5000000 steps (96%)
16:54:58:WU05:FS00:Upload 40.45%
16:55:06:ERROR:FS00:
16:55:06:ERROR:FS00:-------------------------------------------------------
16:55:06:ERROR:FS00:Program Folding@home, VERSION 4.5.4
16:55:06:ERROR:FS00:Source code file: gromacs-4.5.4\src\gmxlib\smalloc.c, line: 171
16:55:06:ERROR:FS00:
16:55:06:ERROR:FS00:Fatal error:
16:55:06:ERROR:FS00:Not enough memory. Failed to calloc ld elements of size ld for 
16:55:06:ERROR:FS00:(called from file (null), line 24)
16:55:06:ERROR:FS00:For more information and tips for troubleshooting, please check the GROMACS
16:55:06:ERROR:FS00:website at http://www.gromacs.org/Documentation/Errors
16:55:06:ERROR:FS00:-------------------------------------------------------
16:55:06:ERROR:FS00:
16:55:06:ERROR:FS00:Thanx for Using GROMACS - Have a Nice Day
16:55:06:WU05:FS00:Upload complete
16:55:06:WU05:FS00:Server responded WORK_ACK (400)
16:55:06:WU05:FS00:Final credit estimate, 10752.00 points
16:55:06:WU05:FS00:Cleaning up
16:55:06:ERROR:WU05:FS00:Exception: Failed to remove directory './work/05': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\05\wudata_01.tpr"
16:55:06:WU05:FS00:Cleaning up
16:55:06:ERROR:WU05:FS00:Exception: Failed to remove directory './work/05': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\05\wudata_01.tpr"
16:55:07:ERROR:FS00:
16:55:07:ERROR:FS00:-------------------------------------------------------
16:55:07:ERROR:FS00:Program Folding@home, VERSION 4.5.4
16:55:07:ERROR:FS00:Source code file: gromacs-4.5.4\src\gmxlib\smalloc.c, line: 171
16:55:07:ERROR:FS00:
16:55:07:ERROR:FS00:Fatal error:
16:55:07:ERROR:FS00:Not enough memory. Failed to calloc ld elements of size ld for 
16:55:07:ERROR:FS00:(called from file (null), line 24)
16:55:07:ERROR:FS00:For more information and tips for troubleshooting, please check the GROMACS
16:55:07:ERROR:FS00:website at http://www.gromacs.org/Documentation/Errors
16:55:07:ERROR:FS00:-------------------------------------------------------
16:55:07:ERROR:FS00:
16:55:07:ERROR:FS00:Thanx for Using GROMACS - Have a Nice Day
16:56:07:WU05:FS00:Cleaning up
16:56:07:ERROR:WU05:FS00:Exception: Failed to remove directory './work/05': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\05\wudata_01.tpr"
16:56:45:WU06:FS00:0xa4:Completed 5000 out of 500000 steps  (1%)
16:57:44:WU05:FS00:Cleaning up
16:57:44:ERROR:WU05:FS00:Exception: Failed to remove directory './work/05': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\05\wudata_01.tpr"
16:58:33:WU03:FS01:0x18:Completed 4850000 out of 5000000 steps (97%)
16:58:39:WU06:FS00:0xa4:Completed 10000 out of 500000 steps  (2%)
16:59:07:WU00:FS02:0x17:Completed 10560000 out of 16000000 steps (66%)
16:59:15:WU01:FS00:Cleaning up
16:59:15:ERROR:WU01:FS00:Exception: Failed to remove directory './work/01': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\01\wudata_01.tpr"
17:00:21:WU05:FS00:Cleaning up
17:00:21:ERROR:WU05:FS00:Exception: Failed to remove directory './work/05': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\05\wudata_01.tpr"
17:00:31:WU06:FS00:0xa4:Completed 15000 out of 500000 steps  (3%)
17:02:21:WU03:FS01:0x18:Completed 4900000 out of 5000000 steps (98%)
17:02:22:WU06:FS00:0xa4:Completed 20000 out of 500000 steps  (4%)
17:04:14:WU06:FS00:0xa4:Completed 25000 out of 500000 steps  (5%)
17:04:35:WU05:FS00:Cleaning up
17:04:35:ERROR:WU05:FS00:Exception: Failed to remove directory './work/05': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\05\wudata_01.tpr"
17:05:03:1912:127.0.0.1:New Web connection
17:05:40:WU00:FS02:0x17:Completed 10720000 out of 16000000 steps (67%)
17:05:57:WU03:FS01:0x18:Completed 4950000 out of 5000000 steps (99%)
17:06:22:WU06:FS00:0xa4:Completed 30000 out of 500000 steps  (6%)
17:08:19:WU06:FS00:0xa4:Completed 35000 out of 500000 steps  (7%)
17:09:33:WU03:FS01:0x18:Completed 5000000 out of 5000000 steps (100%)
17:09:34:WU07:FS01:Connecting to 171.67.108.200:80
17:09:35:WU07:FS01:Assigned to work server 140.163.4.234
17:09:35:WU07:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GM204 [GeForce GTX 980] from 140.163.4.234
17:09:35:WU07:FS01:Connecting to 140.163.4.234:8080
17:09:35:WU07:FS01:Downloading 2.31MiB
17:09:37:WU07:FS01:Download complete
17:09:37:WU07:FS01:Received Unit: id:07 state:DOWNLOAD error:NO_ERROR project:10476 run:0 clone:60 gen:398 core:0x18 unit:0x000001e0538b3dba53f3811c64984db2
17:09:47:WU03:FS01:0x18:Saving result file logfile_01.txt
17:09:47:WU03:FS01:0x18:Saving result file checkpointState.xml
17:09:50:WU03:FS01:0x18:Saving result file checkpt.crc
17:09:50:WU03:FS01:0x18:Saving result file log.txt
17:09:50:WU03:FS01:0x18:Saving result file positions.xtc
17:09:52:WU03:FS01:0x18:Folding@home Core Shutdown: FINISHED_UNIT
17:09:52:WU03:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
17:09:52:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:10478 run:1 clone:25 gen:89 core:0x18 unit:0x0000007e538b3dba548f6e0c7873c310
17:09:52:WU03:FS01:Uploading 8.77MiB to 140.163.4.234
17:09:52:WU03:FS01:Connecting to 140.163.4.234:8080
17:09:53:WU07:FS01:Starting
17:09:53:WU07:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 07 -suffix 01 -version 704 -lifeline 5160 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
17:09:53:WU07:FS01:Started FahCore on PID 3860
17:09:56:WU07:FS01:Core PID:1720
17:09:56:WU07:FS01:FahCore 0x18 started
17:09:57:WU07:FS01:0x18:*********************** Log Started 2015-08-10T17:09:56Z ***********************
17:09:57:WU07:FS01:0x18:Project: 10476 (Run 0, Clone 60, Gen 398)
17:09:57:WU07:FS01:0x18:Unit: 0x000001e0538b3dba53f3811c64984db2
17:09:57:WU07:FS01:0x18:CPU: 0x00000000000000000000000000000000
17:09:57:WU07:FS01:0x18:Machine: 1
17:09:57:WU07:FS01:0x18:Reading tar file state.xml
17:09:57:WU07:FS01:0x18:Reading tar file system.xml
17:09:58:WU07:FS01:0x18:Reading tar file integrator.xml
17:09:58:WU07:FS01:0x18:Reading tar file core.xml
17:09:58:WU07:FS01:0x18:Digital signatures verified
17:09:58:WU07:FS01:0x18:Folding@home GPU core18
17:09:58:WU07:FS01:0x18:Version 0.0.4
17:09:58:WU03:FS01:Upload 14.26%
17:10:04:WU03:FS01:Upload 30.65%
17:10:10:WU03:FS01:Upload 52.04%
17:10:15:WU06:FS00:0xa4:Completed 40000 out of 500000 steps  (8%)
17:10:16:WU03:FS01:Upload 79.13%
17:10:23:WU07:FS01:0x18:Completed 0 out of 5000000 steps (0%)
17:10:23:WU07:FS01:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
17:10:27:WU03:FS01:Upload complete
17:10:27:WU03:FS01:Server responded WORK_ACK (400)
17:10:27:WU03:FS01:Final credit estimate, 59806.00 points
17:10:27:WU03:FS01:Cleaning up
17:11:26:WU05:FS00:Cleaning up
17:11:26:ERROR:WU05:FS00:Exception: Failed to remove directory './work/05': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\05\wudata_01.tpr"
17:12:10:WU06:FS00:0xa4:Completed 45000 out of 500000 steps  (9%)
17:12:16:WU00:FS02:0x17:Completed 10880000 out of 16000000 steps (68%)
17:12:54:WU07:FS01:0x18:Completed 50000 out of 5000000 steps (1%)
17:14:02:WU06:FS00:0xa4:Completed 50000 out of 500000 steps  (10%)
17:15:19:WU07:FS01:0x18:Completed 100000 out of 5000000 steps (2%)
17:15:57:WU06:FS00:0xa4:Completed 55000 out of 500000 steps  (11%)
17:17:50:WU06:FS00:0xa4:Completed 60000 out of 500000 steps  (12%)
17:17:53:WU07:FS01:0x18:Completed 150000 out of 5000000 steps (3%)
17:18:51:WU00:FS02:0x17:Completed 11040000 out of 16000000 steps (69%)
17:19:44:WU06:FS00:0xa4:Completed 65000 out of 500000 steps  (13%)
17:20:16:WU07:FS01:0x18:Completed 200000 out of 5000000 steps (4%)
17:21:38:WU06:FS00:0xa4:Completed 70000 out of 500000 steps  (14%)
17:22:32:WU05:FS00:Cleaning up
17:22:32:ERROR:WU05:FS00:Exception: Failed to remove directory './work/05': boost::filesystem::remove: The process cannot access the file because it is being used by another process: ".\work\05\wudata_01.tpr"
17:22:41:WU07:FS01:0x18:Completed 250000 out of 5000000 steps (5%)
17:23:30:WU06:FS00:0xa4:Completed 75000 out of 500000 steps  (15%)
17:25:14:WU07:FS01:0x18:Completed 300000 out of 5000000 steps (6%)
17:25:25:WU06:FS00:0xa4:Completed 80000 out of 500000 steps  (16%)
17:25:26:WU00:FS02:0x17:Completed 11200000 out of 16000000 steps (70%)
17:26:39:FS00:Paused
17:26:39:FS01:Paused
17:26:39:FS02:Paused
17:26:40:FS00:Shutting core down
17:26:40:FS01:Shutting core down
17:26:40:FS02:Shutting core down
17:26:40:WU07:FS01:0x18:WARNING:Console control signal 1 on PID 1720
17:26:40:WU00:FS02:0x17:WARNING:Console control signal 1 on PID 4280
17:26:40:WU07:FS01:0x18:Exiting, please wait. . .
17:26:40:WU00:FS02:0x17:Exiting, please wait. . .
17:26:40:WU00:FS02:0x17:Folding@home Core Shutdown: INTERRUPTED
17:26:41:WU07:FS01:0x18:Folding@home Core Shutdown: INTERRUPTED
17:26:41:WU00:FS02:FahCore returned: INTERRUPTED (102 = 0x66)
17:26:41:WU07:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
17:26:50:WU06:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
17:27:14:Saving configuration to config.xml
17:27:14:<config>
17:27:14:  <!-- Network -->
17:27:14:  <proxy v=':8080'/>
17:27:14:
17:27:14:  <!-- Slot Control -->
17:27:14:  <power v='full'/>
17:27:14:
17:27:14:  <!-- User Information -->
17:27:14:  <passkey v='********************************'/>
17:27:14:  <user v='Richard_Summers'/>
17:27:14:
17:27:14:  <!-- Folding Slots -->
17:27:14:  <slot id='0' type='CPU'>
17:27:14:    <paused v='true'/>
17:27:14:  </slot>
17:27:14:  <slot id='1' type='GPU'>
17:27:14:    <paused v='true'/>
17:27:14:  </slot>
17:27:14:  <slot id='2' type='GPU'>
17:27:14:    <paused v='true'/>
17:27:14:  </slot>
17:27:14:</config>

Post by **bruce** » Wed Aug 12, 2015 3:40 pm

Ricky wrote:OK, I finally found the log data that is relevant to the cleanup problem. Apparently, my appdata folder is hidden, so it was hard to find.

Yes, that's true.
Microsoft recommends that applications use the appdata folder for working data and they established the policy of hiding it.

I have a suspicion that I need to reboot windows periodically to clear memory. At the time the problem occurred, windows had been running over a week. What do you think?

That SHOULDN'T be needed, but if it works, then do it.

I run WIndows 7 for weeks at a time. I think all of the computers that Microsoft looks at expect Windows Update to reboot every week or two.

Ricky · Post by **Ricky** » Wed Aug 12, 2015 5:56 pm

Thanks for the reply Bruce.

I will keep an eye on the FAH status. If this happens again, then I will reboot. I will keep a log of how long I am able to run if it keeps happening.

davidcoton · Post by **davidcoton** » Wed Aug 12, 2015 8:32 pm

Ricky wrote: I have a suspicion that I need to reboot windows periodically to clear memory.

My Windows Vista needs a fortnightly restart for its backup, otherwise the process reports "out of resources". It isn't memory, but AFAICT there is a significant leak of handles. The number of handles shown in Task Manager (performance tab) just keeps rising with a random element from sample to sample. I have been unable to identify the source of the leak, it doesn't seem to be any normal user app, it might be something loaded at startup or Vista itself. I haven't noticed that it affects FAH performance, though.

Ricky · Post by **Ricky** » Thu Aug 13, 2015 1:19 am

Thanks David for the update. I will keep an eye on my threads.

davidcoton · Post by **davidcoton** » Thu Aug 13, 2015 8:35 am

Not threads. Handles. The top entry in the System box of the Performance tab of Task Manager.

Rel25917 · Post by **Rel25917** » Thu Aug 13, 2015 8:37 am

davidcoton wrote:Not threads. Handles. The top entry in the System box of the Performance tab of Task Manager.

You can add a column for handles under the processes tab to see which application is using them all up.

davidcoton · Post by **davidcoton** » Thu Aug 13, 2015 7:35 pm

Thanks. I'd hoped to find something like that. Here's the culprits:

Code: Select all

CLMSServer.exe ... 975,455
ETService.exe ... 962,337 (Acer Empowering Technology Framework Service)

Of course, M$ don't allow cut and paste, so that's a manual copy of just the relevant bits.
Now to find if and how to stop them -- I think I know how for the second one.

EDIT: Got them! Neither service was important to me, both stopped (releasing their handles) and removed from autostart. Now after 271 hours uptime, under 37K handles used (was over 2M).

Folding Forum

Project:9011 run:145 clone:2 gen:75

Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75

Re: Project:9011 run:145 clone:2 gen:75