Project: 3062 (Run 5, Clone 6, Gen 93) Repeated EUEs at 44%

jrweiss · Post by **jrweiss** » Mon Apr 28, 2008 7:30 am

C2D E6850, 2 GB RAM, NOT overclocked, has been stable for many months.

The same WU keeps re-appearing!

[10:34:31] *------------------------------*
[10:34:31] Folding@Home Gromacs SMP Core
[10:34:31] Version 1.74 (March 10, 2007)
[10:34:31] 
[10:34:31] Preparing to commence simulation
[10:34:31] - Ensuring status. Please wait.
[10:34:48] - Assembly optimizations manually forced on.
[10:34:48] - Not checking prior termination.
[10:34:49] - Expanded 607705 -> 3260637 (decompressed 536.5 percent)
[10:34:49] - Starting from initial work packet
[10:34:49] 
[10:34:49] Project: 3062 (Run 5, Clone 6, Gen 93)
[10:34:49] 
[10:34:49] Assembly optimizations on if available.
[10:34:49] Entering M.D.
[10:34:49] e.
[10:34:49] Entering M.D.
[10:34:55] mbdaProteinWriting local files
[10:34:55] Extra SSE boost OK.
[10:34:55] rExtra SSE boost OK.
[10:34:55] Writing local files
[10:34:55] Completed 0 out of 5000000 steps  (0 percent)
[10:50:42] Writing local files
[10:50:42] Completed 50000 out of 5000000 steps  (1 percent)
[11:06:29] Writing local files
[11:06:29] Completed 100000 out of 5000000 steps  (2 percent)
[11:22:17] Writing local files
[11:22:17] Completed 150000 out of 5000000 steps  (3 percent)
[11:38:06] Writing local files
[11:38:06] Completed 200000 out of 5000000 steps  (4 percent)
[11:53:54] Writing local files
[11:53:54] Completed 250000 out of 5000000 steps  (5 percent)
[12:09:42] Writing local files
[12:09:42] Completed 300000 out of 5000000 steps  (6 percent)
[12:25:31] Writing local files
[12:25:31] Completed 350000 out of 5000000 steps  (7 percent)
[12:41:20] Writing local files
[12:41:20] Completed 400000 out of 5000000 steps  (8 percent)
[12:57:08] Writing local files
[12:57:08] Completed 450000 out of 5000000 steps  (9 percent)
[13:12:56] Writing local files
[13:12:56] Completed 500000 out of 5000000 steps  (10 percent)
[13:28:46] Writing local files
[13:28:46] Completed 550000 out of 5000000 steps  (11 percent)
[13:44:34] Writing local files
[13:44:34] Completed 600000 out of 5000000 steps  (12 percent)
[14:00:22] Writing local files
[14:00:22] Completed 650000 out of 5000000 steps  (13 percent)
[14:16:07] Writing local files
[14:16:07] Completed 700000 out of 5000000 steps  (14 percent)
[14:31:54] Writing local files
[14:31:54] Completed 750000 out of 5000000 steps  (15 percent)
[14:47:42] Writing local files
[14:47:42] Completed 800000 out of 5000000 steps  (16 percent)
[15:03:30] Writing local files
[15:03:30] Completed 850000 out of 5000000 steps  (17 percent)
[15:19:17] Writing local files
[15:19:17] Completed 900000 out of 5000000 steps  (18 percent)
[15:35:03] Writing local files
[15:35:03] Completed 950000 out of 5000000 steps  (19 percent)
[15:35:30] - Autosending finished units...
[15:35:30] Trying to send all finished work units
[15:35:30] + No unsent completed units remaining.
[15:35:30] - Autosend completed
[15:50:51] Writing local files
[15:50:51] Completed 1000000 out of 5000000 steps  (20 percent)
[16:06:40] Writing local files
[16:06:40] Completed 1050000 out of 5000000 steps  (21 percent)
[16:22:27] Writing local files
[16:22:27] Completed 1100000 out of 5000000 steps  (22 percent)
[16:38:19] Writing local files
[16:38:19] Completed 1150000 out of 5000000 steps  (23 percent)
[16:54:07] Writing local files
[16:54:07] Completed 1200000 out of 5000000 steps  (24 percent)
[17:10:06] Writing local files
[17:10:06] Completed 1250000 out of 5000000 steps  (25 percent)
[17:26:23] Writing local files
[17:26:23] Completed 1300000 out of 5000000 steps  (26 percent)
[17:42:30] Writing local files
[17:42:30] Completed 1350000 out of 5000000 steps  (27 percent)
[17:58:24] Writing local files
[17:58:24] Completed 1400000 out of 5000000 steps  (28 percent)
[18:14:17] Writing local files
[18:14:17] Completed 1450000 out of 5000000 steps  (29 percent)
[18:30:09] Writing local files
[18:30:09] Completed 1500000 out of 5000000 steps  (30 percent)
[18:46:02] Writing local files
[18:46:02] Completed 1550000 out of 5000000 steps  (31 percent)
[19:01:58] Writing local files
[19:01:58] Completed 1600000 out of 5000000 steps  (32 percent)
[19:17:45] Writing local files
[19:17:45] Completed 1650000 out of 5000000 steps  (33 percent)
[19:34:18] Writing local files
[19:34:18] Completed 1700000 out of 5000000 steps  (34 percent)
[19:50:12] Writing local files
[19:50:12] Completed 1750000 out of 5000000 steps  (35 percent)
[20:06:04] Writing local files
[20:06:04] Completed 1800000 out of 5000000 steps  (36 percent)
[20:21:59] Writing local files
[20:21:59] Completed 1850000 out of 5000000 steps  (37 percent)
[20:37:51] Writing local files
[20:37:51] Completed 1900000 out of 5000000 steps  (38 percent)
[20:53:39] Writing local files
[20:53:39] Completed 1950000 out of 5000000 steps  (39 percent)
[21:09:28] Writing local files
[21:09:28] Completed 2000000 out of 5000000 steps  (40 percent)
[21:25:20] Writing local files
[21:25:20] Completed 2050000 out of 5000000 steps  (41 percent)
[21:35:30] - Autosending finished units...
[21:35:30] Trying to send all finished work units
[21:35:30] + No unsent completed units remaining.
[21:35:30] - Autosend completed
[21:41:09] Writing local files
[21:41:09] Completed 2100000 out of 5000000 steps  (42 percent)
[21:56:56] Writing local files
[21:56:56] Completed 2150000 out of 5000000 steps  (43 percent)
[22:12:44] Writing local files
[22:12:44] Completed 2200000 out of 5000000 steps  (44 percent)
[22:22:56] Warning:  long 1-4 interactions
[23:35:52] Killing all core threads
[23:35:52] Killing SMP core threads
[23:35:52] Could not get process id information.  Please kill core process manually

Folding@Home Client Shutdown at user request.
[23:35:52] ***** Got a SIGTERM signal (2)
[23:35:52] Killing all core threads
[23:35:52] Killing SMP core threads
[23:35:52] Could not get process id information.  Please kill core process manually

Folding@Home Client Shutdown.



--- Opening Log file [April 26 23:39:26] 


# SMP Client ##################################################################
###############################################################################

                       Folding@Home Client Version 5.91beta6

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Apps\F@H\SMP
Executable: C:\Apps\F@H\SMP\fah.exe
Arguments: -local -verbosity 9 -forceasm 

Warning:
 By using the -forceasm flag, you are overriding
 safeguards in the program. If you did not intend to
 do this, please restart the program without -forceasm.
 If work units are not completing fully (and particularly
 if your machine is overclocked), then please discontinue
 use of the flag.

[23:39:26] - Ask before connecting: No
[23:39:26] - User name: jrweiss (Team 3074)
[23:39:26] - User ID: 6A4CC513440C2753
[23:39:26] - Machine ID: 1
[23:39:26] 
[23:39:26] Loaded queue successfully.
[23:39:26] 
[23:39:26] - Autosending finished units...
[23:39:26] + Processing work unit
[23:39:26] Trying to send all finished work units
[23:39:26] Core required: FahCore_a1.exe
[23:39:26] + No unsent completed units remaining.
[23:39:26] - Autosend completed
[23:39:26] Core found.
[23:39:26] Working on Unit 07 [April 26 23:39:26]
[23:39:26] + Working ...
[23:39:26] - Calling 'mpiexec -channel auto -np 4 FahCore_a1.exe -dir work/ -suffix 07 -cpu 96 -checkpoint 30 -forceasm -verbose -lifeline 3768 -version 591'

[23:39:27] 
[23:39:27] *------------------------------*
[23:39:27] Folding@Home Gromacs SMP Core
[23:39:27] Version 1.74 (March 10, 2007)
[23:39:27] 
[23:39:27] Preparing to commence simulation
[23:39:27] - Ensuring status. Please wait.
[23:39:44] - Assembly optimizations manually forced on.
[23:39:44] - Not checking prior termination.
[23:39:44] 
[23:39:44] Folding@home Core Shutdown: MISSING_WORK_FILES
[23:39:44] Finalizing output
[23:41:48] CoreStatus = 1 (1)
[23:41:48] Client-core communications error: ERROR 0x1
[23:41:48] Deleting current work unit & continuing...
[23:44:08] - Warning: Could not delete all work unit files (7): Core returned invalid code
[23:44:08] Trying to send all finished work units
[23:44:08] + No unsent completed units remaining.
[23:44:08] - Preparing to get new work unit...
[23:44:08] + Attempting to get work packet
[23:44:08] - Will indicate memory of 1024 MB
[23:44:08] - Connecting to assignment server
[23:44:08] Connecting to http://assign.stanford.edu:8080/
[23:44:09] Posted data.
[23:44:09] Initial: 40AB; - Successful: assigned to (171.64.65.63).
[23:44:09] + News From Folding@Home: Welcome to Folding@Home
[23:44:09] Loaded queue successfully.
[23:44:09] Connecting to http://171.64.65.63:8080/
[23:44:10] Posted data.
[23:44:10] Initial: 0000; - Receiving payload (expected size: 608217)
[23:44:12] - Downloaded at ~296 kB/s
[23:44:12] - Averaged speed for that direction ~294 kB/s
[23:44:12] + Received work.
[23:44:12] + Closed connections
[23:44:17] 
[23:44:17] + Processing work unit
[23:44:17] Core required: FahCore_a1.exe
[23:44:17] Core found.
[23:44:17] Working on Unit 08 [April 26 23:44:17]
[23:44:17] + Working ...
[23:44:17] - Calling 'mpiexec -channel auto -np 4 FahCore_a1.exe -dir work/ -suffix 08 -cpu 96 -checkpoint 30 -forceasm -verbose -lifeline 3768 -version 591'

[23:44:18] 
[23:44:18] *------------------------------*
[23:44:18] Folding@Home Gromacs SMP Core
[23:44:18] Version 1.74 (March 10, 2007)
[23:44:18] 
[23:44:18] Preparing to commence simulation
[23:44:18] - Ensuring status. Please wait.
[23:44:35] - Assembly optimizations manually forced on.
[23:44:35] - Not checking prior termination.
[23:44:35] - Expanded 607705 -> 3260637 (decompressed 536.5 percent)
[23:44:35] - Starting from initial work packet
[23:44:35] 
[23:44:35] Project: 3062 (Run 5, Clone 6, Gen 93)
[23:44:35] 
[23:44:35] Assembly optimizations on if available.
[23:44:35] Entering M.D.
[23:44:35] one 6, Gen 93)
[23:44:35] 
[23:44:35] Assembly optimizations on if available.
[23:44:35] Entering M.D.
[23:44:42] xtra SSE boost OK.
[23:44:42] Writing local files
[23:44:42] Completed 0 out of 5000000 steps  (0 percent)
[00:00:42] Writing local files
[00:00:42] Completed 50000 out of 5000000 steps  (1 percent)
[00:16:35] Writing local files
[00:16:35] Completed 100000 out of 5000000 steps  (2 percent)
[00:32:35] Writing local files
[00:32:35] Completed 150000 out of 5000000 steps  (3 percent)
[00:48:32] Writing local files
[00:48:32] Completed 200000 out of 5000000 steps  (4 percent)
[01:04:22] Writing local files
[01:04:22] Completed 250000 out of 5000000 steps  (5 percent)
[01:20:13] Writing local files
[01:20:13] Completed 300000 out of 5000000 steps  (6 percent)
[01:36:05] Writing local files
[01:36:05] Completed 350000 out of 5000000 steps  (7 percent)
[01:51:57] Writing local files
[01:51:57] Completed 400000 out of 5000000 steps  (8 percent)
[02:07:48] Writing local files
[02:07:48] Completed 450000 out of 5000000 steps  (9 percent)
[02:23:48] Writing local files
[02:23:49] Completed 500000 out of 5000000 steps  (10 percent)
[02:39:47] Writing local files
[02:39:47] Completed 550000 out of 5000000 steps  (11 percent)
[02:55:46] Writing local files
[02:55:46] Completed 600000 out of 5000000 steps  (12 percent)
[03:11:44] Writing local files
[03:11:44] Completed 650000 out of 5000000 steps  (13 percent)
[03:27:31] Writing local files
[03:27:31] Completed 700000 out of 5000000 steps  (14 percent)
[03:43:19] Writing local files
[03:43:19] Completed 750000 out of 5000000 steps  (15 percent)
[03:59:09] Writing local files
[03:59:09] Completed 800000 out of 5000000 steps  (16 percent)
[04:14:57] Writing local files
[04:14:57] Completed 850000 out of 5000000 steps  (17 percent)
[04:30:46] Writing local files
[04:30:46] Completed 900000 out of 5000000 steps  (18 percent)
[04:46:34] Writing local files
[04:46:34] Completed 950000 out of 5000000 steps  (19 percent)
[05:02:23] Writing local files
[05:02:23] Completed 1000000 out of 5000000 steps  (20 percent)
[05:18:11] Writing local files
[05:18:11] Completed 1050000 out of 5000000 steps  (21 percent)
[05:34:03] Writing local files
[05:34:03] Completed 1100000 out of 5000000 steps  (22 percent)
[05:39:26] - Autosending finished units...
[05:39:26] Trying to send all finished work units
[05:39:26] + No unsent completed units remaining.
[05:39:26] - Autosend completed
[05:49:54] Writing local files
[05:49:54] Completed 1150000 out of 5000000 steps  (23 percent)
[06:05:46] Writing local files
[06:05:46] Completed 1200000 out of 5000000 steps  (24 percent)
[06:21:37] Writing local files
[06:21:37] Completed 1250000 out of 5000000 steps  (25 percent)
[06:37:28] Writing local files
[06:37:28] Completed 1300000 out of 5000000 steps  (26 percent)
[06:53:19] Writing local files
[06:53:19] Completed 1350000 out of 5000000 steps  (27 percent)
[07:09:09] Writing local files
[07:09:09] Completed 1400000 out of 5000000 steps  (28 percent)
[07:25:01] Writing local files
[07:25:01] Completed 1450000 out of 5000000 steps  (29 percent)
[07:40:52] Writing local files
[07:40:52] Completed 1500000 out of 5000000 steps  (30 percent)
[07:56:43] Writing local files
[07:56:43] Completed 1550000 out of 5000000 steps  (31 percent)
[08:12:37] Writing local files
[08:12:37] Completed 1600000 out of 5000000 steps  (32 percent)
[08:28:28] Writing local files
[08:28:28] Completed 1650000 out of 5000000 steps  (33 percent)
[08:44:19] Writing local files
[08:44:19] Completed 1700000 out of 5000000 steps  (34 percent)
[09:00:09] Writing local files
[09:00:09] Completed 1750000 out of 5000000 steps  (35 percent)
[09:16:01] Writing local files
[09:16:01] Completed 1800000 out of 5000000 steps  (36 percent)
[09:31:53] Writing local files
[09:31:53] Completed 1850000 out of 5000000 steps  (37 percent)
[09:47:47] Writing local files
[09:47:47] Completed 1900000 out of 5000000 steps  (38 percent)
[10:03:39] Writing local files
[10:03:39] Completed 1950000 out of 5000000 steps  (39 percent)
[10:19:32] Writing local files
[10:19:32] Completed 2000000 out of 5000000 steps  (40 percent)
[10:35:26] Writing local files
[10:35:26] Completed 2050000 out of 5000000 steps  (41 percent)
[10:51:18] Writing local files
[10:51:18] Completed 2100000 out of 5000000 steps  (42 percent)
[11:07:09] Writing local files
[11:07:09] Completed 2150000 out of 5000000 steps  (43 percent)
[11:23:02] Writing local files
[11:23:02] Completed 2200000 out of 5000000 steps  (44 percent)
[11:33:17] Warning:  long 1-4 interactions
[11:39:26] - Autosending finished units...
[11:39:26] Trying to send all finished work units
[11:39:26] + No unsent completed units remaining.
[11:39:26] - Autosend completed
[14:23:02] At least 3 hours since checkpoint written...
[14:25:02] 
[14:25:02] Folding@home Core Shutdown: EARLY_UNIT_END
[14:25:02] 
[14:25:02] Folding@home Core Shutdown: EARLY_UNIT_END
[14:25:06] CoreStatus = 7B (123)
[14:25:06] Client-core communications error: ERROR 0x7b
[14:25:06] Deleting current work unit & continuing...
[14:27:26] - Warning: Could not delete all work unit files (8): Core returned invalid code
[14:27:26] Trying to send all finished work units
[14:27:26] + No unsent completed units remaining.
[14:27:26] - Preparing to get new work unit...
[14:27:26] + Attempting to get work packet
[14:27:26] - Will indicate memory of 1024 MB
[14:27:26] - Connecting to assignment server
[14:27:26] Connecting to http://assign.stanford.edu:8080/
[14:27:26] Posted data.
[14:27:26] Initial: 40AB; - Successful: assigned to (171.64.65.63).
[14:27:26] + News From Folding@Home: Welcome to Folding@Home
[14:27:26] Loaded queue successfully.
[14:27:26] Connecting to http://171.64.65.63:8080/
[14:27:27] Posted data.
[14:27:27] Initial: 0000; - Receiving payload (expected size: 608217)
[14:27:30] - Downloaded at ~197 kB/s
[14:27:30] - Averaged speed for that direction ~275 kB/s
[14:27:30] + Received work.
[14:27:30] + Closed connections
[14:27:35] 
[14:27:35] + Processing work unit
[14:27:35] Core required: FahCore_a1.exe
[14:27:35] Core found.
[14:27:35] Working on Unit 09 [April 27 14:27:35]
[14:27:35] + Working ...
[14:27:35] - Calling 'mpiexec -channel auto -np 4 FahCore_a1.exe -dir work/ -suffix 09 -cpu 96 -checkpoint 30 -forceasm -verbose -lifeline 3768 -version 591'

[14:27:35] 
[14:27:35] *------------------------------*
[14:27:35] Folding@Home Gromacs SMP Core
[14:27:35] Version 1.74 (March 10, 2007)
[14:27:35] 
[14:27:35] Preparing to commence simulation
[14:27:35] - Ensuring status. Please wait.
[14:27:52] - Assembly optimizations manually forced on.
[14:27:52] - Not checking prior termination.
[14:27:53] - Expanded 607705 -> 3260637 (decompressed 536.5 percent)
[14:27:53] - Starting from initial work packet
[14:27:53] 
[14:27:53] Project: 3062 (Run 5, Clone 6, Gen 93)
[14:27:53] 
[14:27:53] Assembly optimizations on if available.
[14:27:53] Entering M.D.
[14:27:53] ct: 3062 (Run
[14:27:53] Project: 3062 (Run 5,Assembly optimizations on if available.
[14:27:53] Entering M.D.
[14:27:53] ble.
[14:27:53] Entering M.D.
[14:27:59] 
[14:27:59] Completed 0 out of 5000000 steps  (0 percent)
[14:27:59] .
[14:27:59] xtra SSE boost OK.
[14:43:48] ompleted 50000 out of 5000000 steps  (1 percent)
[14:59:37] Writing local files
[14:59:37] Completed 100000 out of 5000000 steps  (2 percent)
[15:15:26] Writing local files
[15:15:26] Completed 150000 out of 5000000 steps  (3 percent)
[15:31:14] Writing local files
[15:31:14] Completed 200000 out of 5000000 steps  (4 percent)
[15:47:02] Writing local files
[15:47:02] Completed 250000 out of 5000000 steps  (5 percent)
[16:02:52] Writing local files
[16:02:52] Completed 300000 out of 5000000 steps  (6 percent)
[16:18:42] Writing local files
[16:18:42] Completed 350000 out of 5000000 steps  (7 percent)
[16:34:32] Writing local files
[16:34:32] Completed 400000 out of 5000000 steps  (8 percent)
[16:50:21] Writing local files
[16:50:21] Completed 450000 out of 5000000 steps  (9 percent)
[17:06:09] Writing local files
[17:06:09] Completed 500000 out of 5000000 steps  (10 percent)
[17:21:58] Writing local files
[17:21:58] Completed 550000 out of 5000000 steps  (11 percent)
[17:37:46] Writing local files
[17:37:46] Completed 600000 out of 5000000 steps  (12 percent)
[17:39:26] - Autosending finished units...
[17:39:26] Trying to send all finished work units
[17:39:26] + No unsent completed units remaining.
[17:39:26] - Autosend completed
[17:54:44] Writing local files
[17:54:44] Completed 650000 out of 5000000 steps  (13 percent)
[18:11:32] Writing local files
[18:11:32] Completed 700000 out of 5000000 steps  (14 percent)
[18:27:27] Writing local files
[18:27:27] Completed 750000 out of 5000000 steps  (15 percent)
[18:43:23] Writing local files
[18:43:23] Completed 800000 out of 5000000 steps  (16 percent)
[18:59:18] Writing local files
[18:59:18] Completed 850000 out of 5000000 steps  (17 percent)
[19:15:11] Writing local files
[19:15:11] Completed 900000 out of 5000000 steps  (18 percent)
[19:31:06] Writing local files
[19:31:06] Completed 950000 out of 5000000 steps  (19 percent)
[19:46:56] Writing local files
[19:46:56] Completed 1000000 out of 5000000 steps  (20 percent)
[20:02:45] Writing local files
[20:02:45] Completed 1050000 out of 5000000 steps  (21 percent)
[20:18:33] Writing local files
[20:18:33] Completed 1100000 out of 5000000 steps  (22 percent)
[20:34:21] Writing local files
[20:34:21] Completed 1150000 out of 5000000 steps  (23 percent)
[20:50:10] Writing local files
[20:50:10] Completed 1200000 out of 5000000 steps  (24 percent)
[21:05:58] Writing local files
[21:05:58] Completed 1250000 out of 5000000 steps  (25 percent)
[21:21:47] Writing local files
[21:21:47] Completed 1300000 out of 5000000 steps  (26 percent)
[21:37:34] Writing local files
[21:37:34] Completed 1350000 out of 5000000 steps  (27 percent)
[21:53:21] Writing local files
[21:53:22] Completed 1400000 out of 5000000 steps  (28 percent)
[22:09:09] Writing local files
[22:09:09] Completed 1450000 out of 5000000 steps  (29 percent)
[22:24:57] Writing local files
[22:24:57] Completed 1500000 out of 5000000 steps  (30 percent)
[22:40:46] Writing local files
[22:40:46] Completed 1550000 out of 5000000 steps  (31 percent)
[22:56:35] Writing local files
[22:56:35] Completed 1600000 out of 5000000 steps  (32 percent)
[23:12:23] Writing local files
[23:12:23] Completed 1650000 out of 5000000 steps  (33 percent)
[23:28:11] Writing local files
[23:28:11] Completed 1700000 out of 5000000 steps  (34 percent)
[23:39:26] - Autosending finished units...
[23:39:26] Trying to send all finished work units
[23:39:26] + No unsent completed units remaining.
[23:39:26] - Autosend completed
[23:43:59] Writing local files
[23:43:59] Completed 1750000 out of 5000000 steps  (35 percent)
[23:59:48] Writing local files
[23:59:48] Completed 1800000 out of 5000000 steps  (36 percent)
[00:15:41] Writing local files
[00:15:41] Completed 1850000 out of 5000000 steps  (37 percent)
[00:31:30] Writing local files
[00:31:30] Completed 1900000 out of 5000000 steps  (38 percent)
[00:47:18] Writing local files
[00:47:18] Completed 1950000 out of 5000000 steps  (39 percent)
[01:03:08] Writing local files
[01:03:08] Completed 2000000 out of 5000000 steps  (40 percent)
[01:18:57] Writing local files
[01:18:57] Completed 2050000 out of 5000000 steps  (41 percent)
[01:34:47] Writing local files
[01:34:47] Completed 2100000 out of 5000000 steps  (42 percent)
[01:50:33] Writing local files
[01:50:33] Completed 2150000 out of 5000000 steps  (43 percent)
[02:06:22] Writing local files
[02:06:22] Completed 2200000 out of 5000000 steps  (44 percent)
[02:16:33] Warning:  long 1-4 interactions
[05:06:22] At least 3 hours since checkpoint written...
[05:08:22] 
[05:08:22] Folding@home Core Shutdown: EARLY_UNIT_END
[05:08:22] 
[05:08:22] Folding@home Core Shutdown: EARLY_UNIT_END
[05:08:25] CoreStatus = 7B (123)
[05:08:25] Client-core communications error: ERROR 0x7b
[05:08:25] - Attempting to download new core...
[05:08:25] + Downloading new core: FahCore_a1.exe
[05:08:25] Downloading core (/~pande/Win32/x86//Core_a1.fah from http://www.stanford.edu)
[05:08:26] Initial: AFDE; + 10240 bytes downloaded
[05:08:26] Initial: AD21; + 20480 bytes downloaded
[05:08:26] Initial: CC38; + 30720 bytes downloaded
[05:08:26] Initial: 8501; + 40960 bytes downloaded
[05:08:26] Initial: F56A; + 51200 bytes downloaded
[05:08:26] Initial: ABAE; + 61440 bytes downloaded
[05:08:26] Initial: B6B0; + 71680 bytes downloaded
[05:08:26] Initial: 783A; + 81920 bytes downloaded
[05:08:26] Initial: B2A6; + 92160 bytes downloaded
[05:08:26] Initial: 1409; + 102400 bytes downloaded
[05:08:26] Initial: BBF0; + 112640 bytes downloaded
[05:08:26] Initial: 1861; + 122880 bytes downloaded
[05:08:26] Initial: 5950; + 133120 bytes downloaded
[05:08:26] Initial: 1081; + 143360 bytes downloaded
[05:08:26] Initial: 26BC; + 153600 bytes downloaded
[05:08:26] Initial: FE4A; + 163840 bytes downloaded
[05:08:26] Initial: C1C3; + 174080 bytes downloaded
[05:08:26] Initial: 9B49; + 184320 bytes downloaded
[05:08:26] Initial: 9EE5; + 194560 bytes downloaded
[05:08:26] Initial: D79D; + 204800 bytes downloaded
[05:08:26] Initial: 7801; + 215040 bytes downloaded
[05:08:26] Initial: 8B51; + 225280 bytes downloaded
[05:08:26] Initial: E26E; + 235520 bytes downloaded
[05:08:27] Initial: EDB0; + 245760 bytes downloaded
[05:08:27] Initial: 0919; + 256000 bytes downloaded
[05:08:27] Initial: CDDE; + 266240 bytes downloaded
[05:08:27] Initial: 7A7E; + 276480 bytes downloaded
[05:08:27] Initial: 034E; + 286720 bytes downloaded
[05:08:27] Initial: 88D0; + 296960 bytes downloaded
[05:08:27] Initial: D66D; + 307200 bytes downloaded
[05:08:27] Initial: 6A52; + 317440 bytes downloaded
[05:08:27] Initial: B478; + 327680 bytes downloaded
[05:08:27] Initial: CF8A; + 337920 bytes downloaded
[05:08:27] Initial: 8407; + 348160 bytes downloaded
[05:08:27] Initial: 2246; + 358400 bytes downloaded
[05:08:27] Initial: 1C69; + 368640 bytes downloaded
[05:08:27] Initial: 1287; + 378880 bytes downloaded
[05:08:27] Initial: 19B3; + 389120 bytes downloaded
[05:08:27] Initial: 1AD1; + 399360 bytes downloaded
[05:08:27] Initial: 5791; + 409600 bytes downloaded
[05:08:27] Initial: 76C5; + 419840 bytes downloaded
[05:08:27] Initial: 9B77; + 430080 bytes downloaded
[05:08:27] Initial: E82F; + 440320 bytes downloaded
[05:08:27] Initial: D0D3; + 450560 bytes downloaded
[05:08:27] Initial: 0F5E; + 460800 bytes downloaded
[05:08:27] Initial: D743; + 471040 bytes downloaded
[05:08:27] Initial: 0B7C; + 481280 bytes downloaded
[05:08:27] Initial: FAFD; + 491520 bytes downloaded
[05:08:27] Initial: 0E14; + 501760 bytes downloaded
[05:08:27] Initial: 4048; + 512000 bytes downloaded
[05:08:27] Initial: 21A5; + 522240 bytes downloaded
[05:08:27] Initial: C1A5; + 532480 bytes downloaded
[05:08:28] Initial: F716; + 542720 bytes downloaded
[05:08:28] Initial: DD98; + 552960 bytes downloaded
[05:08:28] Initial: 9F7B; + 563200 bytes downloaded
[05:08:28] Initial: 1CC0; + 573440 bytes downloaded
[05:08:28] Initial: 4D37; + 583680 bytes downloaded
[05:08:28] Initial: 222A; + 593920 bytes downloaded
[05:08:28] Initial: 8E33; + 604160 bytes downloaded
[05:08:28] Initial: D3C9; + 614400 bytes downloaded
[05:08:28] Initial: 9821; + 624640 bytes downloaded
[05:08:28] Initial: 236E; + 634880 bytes downloaded
[05:08:28] Initial: 1A7A; + 645120 bytes downloaded
[05:08:28] Initial: 6D64; + 655360 bytes downloaded
[05:08:28] Initial: 4ADC; + 665600 bytes downloaded
[05:08:28] Initial: 3854; + 675840 bytes downloaded
[05:08:28] Initial: CB5C; + 686080 bytes downloaded
[05:08:28] Initial: 2A88; + 696320 bytes downloaded
[05:08:28] Initial: 1199; + 706560 bytes downloaded
[05:08:28] Initial: 0512; + 716800 bytes downloaded
[05:08:28] Initial: 316E; + 727040 bytes downloaded
[05:08:28] Initial: D89D; + 737280 bytes downloaded
[05:08:28] Initial: E6A3; + 747520 bytes downloaded
[05:08:28] Initial: B488; + 757760 bytes downloaded
[05:08:28] Initial: BAFD; + 768000 bytes downloaded
[05:08:28] Initial: 34A0; + 778240 bytes downloaded
[05:08:29] Initial: DD6C; + 788480 bytes downloaded
[05:08:29] Initial: D2E9; + 789667 bytes downloaded
[05:08:29] Verifying core Core_a1.fah...
[05:08:29] Signature is VALID
[05:08:29] 
[05:08:29] Trying to unzip core FahCore_a1.exe
[05:08:29] Decompressed FahCore_a1.exe (2035712 bytes) successfully
[05:08:29] + Core successfully engaged
[05:08:29] Deleting current work unit & continuing...
[05:10:49] - Warning: Could not delete all work unit files (9): Core returned invalid code
[05:10:49] Trying to send all finished work units
[05:10:49] + No unsent completed units remaining.
[05:10:49] - Preparing to get new work unit...
[05:10:49] + Attempting to get work packet
[05:10:49] - Will indicate memory of 1024 MB
[05:10:49] - Connecting to assignment server
[05:10:49] Connecting to http://assign.stanford.edu:8080/
[05:10:49] Posted data.
[05:10:49] Initial: 40AB; - Successful: assigned to (171.64.65.63).
[05:10:49] + News From Folding@Home: Welcome to Folding@Home
[05:10:49] Loaded queue successfully.
[05:10:49] Connecting to http://171.64.65.63:8080/
[05:10:50] Posted data.
[05:10:50] Initial: 0000; - Receiving payload (expected size: 608217)
[05:10:53] - Downloaded at ~197 kB/s
[05:10:53] - Averaged speed for that direction ~259 kB/s
[05:10:53] + Received work.
[05:10:53] + Closed connections
[05:10:58] 
[05:10:58] + Processing work unit
[05:10:58] Core required: FahCore_a1.exe
[05:10:58] Core found.
[05:10:58] Working on Unit 00 [April 28 05:10:58]
[05:10:58] + Working ...
[05:10:58] - Calling 'mpiexec -channel auto -np 4 FahCore_a1.exe -dir work/ -suffix 00 -cpu 96 -checkpoint 30 -forceasm -verbose -lifeline 3768 -version 591'

[05:10:58] 
[05:10:58] *------------------------------*
[05:10:58] Folding@Home Gromacs SMP Core
[05:10:58] Version 1.74 (March 10, 2007)
[05:10:58] 
[05:10:58] Preparing to commence simulation
[05:10:58] - Ensuring status. Please wait.
[05:11:15] - Assembly optimizations manually forced on.
[05:11:15] - Not checking prior termination.
[05:11:15] - Expanded 607705 -> 3260637 (decompressed 536.5 percent)
[05:11:15] - Failed to delete work/wudata_00.ar
[05:11:15] PWarning:  check for stray files
[05:11:15] - StartAssembly optimizations on if available.
[05:11:15] Entering M.D.
[05:11:15] one 6, Gen 93)
[05:11:15] 
[05:11:16] Assembly optimizations on if available.
[05:11:16] Entering M.D.
[05:11:22] xtra SSE boost OK.
[05:11:22] Writing local files
[05:11:22] Completed 0 out of 5000000 steps  (0 percent)
[05:27:06] Writing local files
[05:27:06] Completed 50000 out of 5000000 steps  (1 percent)
[05:39:26] - Autosending finished units...
[05:39:26] Trying to send all finished work units
[05:39:26] + No unsent completed units remaining.
[05:39:26] - Autosend completed
[05:42:51] Writing local files
[05:42:51] Completed 100000 out of 5000000 steps  (2 percent)
[05:58:48] Writing local files
[05:58:48] Completed 150000 out of 5000000 steps  (3 percent)
[06:14:40] Writing local files
[06:14:40] Completed 200000 out of 5000000 steps  (4 percent)
[06:30:30] Writing local files
[06:30:30] Completed 250000 out of 5000000 steps  (5 percent)
[06:46:18] Writing local files
[06:46:18] Completed 300000 out of 5000000 steps  (6 percent)
[07:02:05] Writing local files
[07:02:05] Completed 350000 out of 5000000 steps  (7 percent)
[07:17:54] Writing local files
[07:17:54] Completed 400000 out of 5000000 steps  (8 percent)

Post by **toTOW** » Mon Apr 28, 2008 1:13 pm

If you didn't do it yet, you might want to submit partial result with qfix : viewtopic.php?f=8&t=191

jrweiss · Post by **jrweiss** » Mon Apr 28, 2008 3:29 pm

The results are already uploaded after each EUE, and I get my 4 points or whatever. What more will qfix get me in this case?

Post by **toTOW** » Mon Apr 28, 2008 4:33 pm

If you don't use qfix, partial results are not uploaded as you can see here :

[14:25:02] Folding@home Core Shutdown: EARLY_UNIT_END
[14:25:02]
[14:25:02] Folding@home Core Shutdown: EARLY_UNIT_END
[14:25:06] CoreStatus = 7B (123)
[14:25:06] Client-core communications error: ERROR 0x7b
[14:25:06] Deleting current work unit & continuing...

In fact result file is still present, but entry is deleted from queue (this is a specific SMP bug with EUE) ... qfix will find the result file and restore the queue entry, allowing you to submit them

jrweiss · Post by **jrweiss** » Wed Apr 30, 2008 8:13 am

That machine is STILL repeatedly downloading the same WU and STILL getting to the SAME 44% point and EUE!!!

If I were home, I'd try to force it to do something else (maybe by shutting down F@H altogether!

Post by **bruce** » Wed Apr 30, 2008 5:53 pm

jrweiss wrote:The results are already uploaded after each EUE, and I get my 4 points or whatever. What more will qfix get me in this case?

I don't think you're getting your " points or whatever"

jrweiss wrote:That machine is STILL repeatedly downloading the same WU and STILL getting to the SAME 44% point and EUE!!!

If I were home, I'd try to force it to do something else (maybe by shutting down F@H altogether!

If the wuresults file is present, as toTOW suggests but has been removed from the queue due to the 0x7b error, qfix will put the entry back in the queue and the partial result will upload. Once you upload a result (partial or complete) that WU will not be reassigned to you.

If no wuresults file is generated, qfix will say there is nothing to fix.

jrweiss · Post by **jrweiss** » Mon May 05, 2008 3:53 pm

Well, when I got home yesterday I found it finally working on a new WU. There were still data files for several old WUs in the /work directory, so I let the current WU finish with the -oneunit flag and ran the qfix routine. It did not appear to send anything when I ran the -send all.

Whatever it was, it cost the project 9 days production.

codysluder · Post by **codysluder** » Mon May 05, 2008 6:54 pm

bruce wrote:If the wuresults file is present, as toTOW suggests but has been removed from the queue due to the 0x7b error, qfix will put the entry back in the queue and the partial result will upload. Once you upload a result (partial or complete) that WU will not be reassigned to you.

If no wuresults file is generated, qfix will say there is nothing to fix.

The biggest advantage of this procedure is that after a successful qfix and upload, the server will not send the same WU to you again. Consider any points you earn from this procedure a bonus, because it's more important to me to move on to another WU than to get partial credit.

jrweiss · Post by **jrweiss** » Fri May 09, 2008 6:59 pm

Well, another interesting twist...

After picking up a couple different WUs, the machine picked up this one AGAIN! First time it got another EUE at 44%:

[11:26:05] Completed 2200000 out of 5000000 steps (44 percent)
[11:36:24] Warning: long 1-4 interactions
[11:36:24] Quit 101 - NaN detected: (ener[0])
[11:36:24]
[11:36:24] Simulation instability has been encountered. The run has entered a
[11:36:24] state from which no further progress can be made.
[11:36:24] This may be the correct result of the simulation, however if you
[11:36:24] often see other project units terminating early like this
[11:36:24] too, you may wish to check the stability of your computer (issues
[11:36:24] such as high temperature, overclocking, etc.).
[11:36:24] Going to send back what have done.
[11:36:24] logfile size: 64101
[11:36:24] - Writing 64650 bytes of core data to disk...
[11:36:24] ... Done.
[11:36:24] - Failed to delete work/wudata_02.pdo
[11:36:24] - Failed to delete work/wudata_02.xvg
[11:36:24] Warning: check for stray files
[11:38:24]
[11:38:24] Folding@home Core Shutdown: EARLY_UNIT_END
[11:38:24]
[11:38:24] Folding@home Core Shutdown: EARLY_UNIT_END
[11:38:27] CoreStatus = 7B (123)
[11:38:27] Client-core communications error: ERROR 0x7b
[11:38:27] Deleting current work unit & continuing...
[11:40:47] - Warning: Could not delete all work unit files (2): Core returned invalid code
[11:40:47] Trying to send all finished work units
[11:40:47] + No unsent completed units remaining.

Next time it made it to 75%. Just coincidentally, I had rebooted the machine when the WU was at 40% (I just discovered this when I reviewed the log):

[12:03:17] Completed 3750000 out of 5000000 steps (75 percent)
[12:17:20] Gromacs cannot continue further.
[12:17:20] Going to send back what have done.
[12:17:20] logfile size: 156965
[12:17:20] - Writing 157501 bytes of core data to disk...
[12:17:20] ... Done.
[12:17:20] - Failed to delete work/wudata_03.arc
[12:17:20] - Failed to delete work/wudata_03.xtc
[12:17:20] No C.P. to delete.
[12:17:20] - Failed to delete work/wudata_03.sas
[12:17:20] - Failed to delete work/wudata_03.goe
[12:17:20] - Failed to delete work/wudata_03.xvg
[12:17:20] Warning: check for stray files
[12:19:20]
[12:19:20] Folding@home Core Shutdown: EARLY_UNIT_END
[12:19:20]
[12:19:20] Folding@home Core Shutdown: EARLY_UNIT_END
[12:19:23] CoreStatus = 7B (123)
[12:19:23] Client-core communications error: ERROR 0x7b
[12:19:23] Deleting current work unit & continuing...
[12:21:43] - Warning: Could not delete all work unit files (3): Core returned invalid code
[12:21:43] Trying to send all finished work units
[12:21:43] + No unsent completed units remaining.

And then it FINALLY finished!

I find it a bit strange that the WU sat on the server for 2 days, then was re-assigned to the same machine that barfed on it so many times before...

Post by **bruce** » Sat May 10, 2008 6:49 am

I've said it before, but apparently not in this thread.

After an EUE, one of two things happen. You may get a message "Going to send back what have done." in which case the client informs the server that it had an error and the server will not reissue the WU. Otherwise, you will get a message saying "Deleting current work unit & continuing...". In the latter case, the client does NOT inform the server that it had an error, only that it needed more work and the server decides to reissue the same WU.

That's why the qfix procedure is so important. After you've had a message saying "Deleting . . ." the file is undeleted and the client can send the EUE report.

jrweiss · Post by **jrweiss** » Sat May 10, 2008 7:50 pm

Well, I had already done the qfix procedure several days prior, so I thought I was "done" with that WU (either as a result of qfix or as a result of an earlier send that made qfix redundant).

I can't monitor F@H 24/7, so I won't have the ability for a timely qfix run every time the system burps. AFAICT, qfix can only be run when a WU is completed and before another one starts. Is that true or not?

Folding Forum

Project: 3062 (Run 5, Clone 6, Gen 93) Repeated EUEs at 44%

Project: 3062 (Run 5, Clone 6, Gen 93) Repeated EUEs at 44%

Re: Project: 3062 (Run 5, Clone 6, Gen 93) Repeated EUEs at 44%

Re: Project: 3062 (Run 5, Clone 6, Gen 93) Repeated EUEs at 44%

Re: Project: 3062 (Run 5, Clone 6, Gen 93) Repeated EUEs at 44%

Re: Project: 3062 (Run 5, Clone 6, Gen 93) Repeated EUEs at 44%

Re: Project: 3062 (Run 5, Clone 6, Gen 93) Repeated EUEs at 44%

Re: Project: 3062 (Run 5, Clone 6, Gen 93) Repeated EUEs at 44%

Re: Project: 3062 (Run 5, Clone 6, Gen 93) Repeated EUEs at 44%

Re: Project: 3062 (Run 5, Clone 6, Gen 93) Repeated EUEs at 44%

Re: Project: 3062 (Run 5, Clone 6, Gen 93) Repeated EUEs at 44%

Re: Project: 3062 (Run 5, Clone 6, Gen 93) Repeated EUEs at 44%