Task 15502816

Name	hadcm3n_o25e_2140_40_008269913_0
Workunit	8425037
Created	24 Dec 2012, 0:32:30 UTC
Sent	26 Dec 2012, 4:24:44 UTC
Report deadline	27 Mar 2013, 11:51:55 UTC
Received	26 Mar 2013, 11:43:21 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1235099
Run time	9 days 18 hours 7 min 52 sec
CPU time	9 days 3 hours 41 min 58 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.78 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:41:31 (1940): No heartbeat from core client for 30 sec - exiting 10:41:32 (1940): No heartbeat from core client for 30 sec - exiting 10:41:33 (1940): No heartbeat from core client for 30 sec - exiting 10:41:34 (1940): No heartbeat from core client for 30 sec - exiting 10:41:36 (1940): No heartbeat from core client for 30 sec - exiting 10:41:37 (1940): No heartbeat from core client for 30 sec - exiting 10:41:38 (1940): No heartbeat from core client for 30 sec - exiting 10:41:39 (1940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3548, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3548, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3548, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5084, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5084, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4176, iMonCtr=1 Model crash detected, will try to restart... 18:25:01 (5796): No heartbeat from core client for 30 sec - exiting 18:25:02 (5796): No heartbeat from core client for 30 sec - exiting 18:25:03 (5796): No heartbeat from core client for 30 sec - exiting 18:25:04 (5796): No heartbeat from core client for 30 sec - exiting 18:25:05 (5796): No heartbeat from core client for 30 sec - exiting 18:25:06 (5796): No heartbeat from core client for 30 sec - exiting 18:25:07 (5796): No heartbeat from core client for 30 sec - exiting 18:25:09 (5796): No heartbeat from core client for 30 sec - exiting 18:25:10 (5796): No heartbeat from core client for 30 sec - exiting 18:25:11 (5796): No heartbeat from core client for 30 sec - exiting 18:25:12 (5796): No heartbeat from core client for 30 sec - exiting 18:25:13 (5796): No heartbeat from core client for 30 sec - exiting 18:25:14 (5796): No heartbeat from core client for 30 sec - exiting 18:25:15 (5796): No heartbeat from core client for 30 sec - exiting 18:25:16 (5796): No heartbeat from core client for 30 sec - exiting 18:25:17 (5796): No heartbeat from core client for 30 sec - exiting 18:25:18 (5796): No heartbeat from core client for 30 sec - exiting 18:25:19 (5796): No heartbeat from core client for 30 sec - exiting 18:25:21 (5796): No heartbeat from core client for 30 sec - exiting 18:25:22 (5796): No heartbeat from core client for 30 sec - exiting 18:25:23 (5796): No heartbeat from core client for 30 sec - exiting 18:25:24 (5796): No heartbeat from core client for 30 sec - exiting 18:25:25 (5796): No heartbeat from core client for 30 sec - exiting 18:25:26 (5796): No heartbeat from core client for 30 sec - exiting 18:25:27 (5796): No heartbeat from core client for 30 sec - exiting 18:25:28 (5796): No heartbeat from core client for 30 sec - exiting 18:25:29 (5796): No heartbeat from core client for 30 sec - exiting 18:25:30 (5796): No heartbeat from core client for 30 sec - exiting 18:25:32 (5796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 12:43:51 (4608): No heartbeat from core client for 30 sec - exiting 12:43:52 (4608): No heartbeat from core client for 30 sec - exiting 12:43:53 (4608): No heartbeat from core client for 30 sec - exiting 12:43:54 (4608): No heartbeat from core client for 30 sec - exiting 12:43:55 (4608): No heartbeat from core client for 30 sec - exiting 12:43:56 (4608): No heartbeat from core client for 30 sec - exiting 12:43:57 (4608): No heartbeat from core client for 30 sec - exiting 12:43:58 (4608): No heartbeat from core client for 30 sec - exiting 12:43:59 (4608): No heartbeat from core client for 30 sec - exiting 12:44:01 (4608): No heartbeat from core client for 30 sec - exiting 12:44:02 (4608): No heartbeat from core client for 30 sec - exiting 12:44:03 (4608): No heartbeat from core client for 30 sec - exiting 12:44:04 (4608): No heartbeat from core client for 30 sec - exiting 12:44:05 (4608): No heartbeat from core client for 30 sec - exiting 12:44:06 (4608): No heartbeat from core client for 30 sec - exiting 12:44:07 (4608): No heartbeat from core client for 30 sec - exiting 12:44:08 (4608): No heartbeat from core client for 30 sec - exiting 12:44:09 (4608): No heartbeat from core client for 30 sec - exiting 12:44:10 (4608): No heartbeat from core client for 30 sec - exiting 12:44:11 (4608): No heartbeat from core client for 30 sec - exiting 12:44:13 (4608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:52:37 (5124): No heartbeat from core client for 30 sec - exiting 08:52:38 (5124): No heartbeat from core client for 30 sec - exiting 08:52:39 (5124): No heartbeat from core client for 30 sec - exiting 08:52:40 (5124): No heartbeat from core client for 30 sec - exiting 08:52:41 (5124): No heartbeat from core client for 30 sec - exiting 08:52:42 (5124): No heartbeat from core client for 30 sec - exiting 08:52:44 (5124): No heartbeat from core client for 30 sec - exiting 08:52:45 (5124): No heartbeat from core client for 30 sec - exiting 08:52:46 (5124): No heartbeat from core client for 30 sec - exiting 08:52:47 (5124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:52:48 (5124): No heartbeat from core client for 30 sec - exiting 08:52:49 (5124): No heartbeat from core client for 30 sec - exiting 08:52:50 (5124): No heartbeat from core client for 30 sec - exiting 08:52:51 (5124): No heartbeat from core client for 30 sec - exiting 08:52:52 (5124): No heartbeat from core client for 30 sec - exiting 08:52:53 (5124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:18:46 (6892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:08:27 (9860): No heartbeat from core client for 30 sec - exiting 00:08:28 (9860): No heartbeat from core client for 30 sec - exiting 00:08:29 (9860): No heartbeat from core client for 30 sec - exiting 00:08:30 (9860): No heartbeat from core client for 30 sec - exiting 00:08:31 (9860): No heartbeat from core client for 30 sec - exiting 00:08:32 (9860): No heartbeat from core client for 30 sec - exiting 00:08:33 (9860): No heartbeat from core client for 30 sec - exiting 00:08:34 (9860): No heartbeat from core client for 30 sec - exiting 00:08:36 (9860): No heartbeat from core client for 30 sec - exiting 00:08:37 (9860): No heartbeat from core client for 30 sec - exiting 00:08:38 (9860): No heartbeat from core client for 30 sec - exiting 00:08:39 (9860): No heartbeat from core client for 30 sec - exiting 00:08:40 (9860): No heartbeat from core client for 30 sec - exiting 00:08:41 (9860): No heartbeat from core client for 30 sec - exiting 00:08:42 (9860): No heartbeat from core client for 30 sec - exiting 00:08:43 (9860): No heartbeat from core client for 30 sec - exiting 00:08:44 (9860): No heartbeat from core client for 30 sec - exiting 00:08:45 (9860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... MainError: 02:44:46 PM No files match the supplied pattern. MainError: 02:44:46 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 09:36:56 AM No files match the supplied pattern. MainError: 09:36:56 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 12:00:22 AM No files match the supplied pattern. MainError: 12:00:22 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 08:37:12 AM No files match the supplied pattern. MainError: 08:37:12 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:37:11 (6300): No heartbeat from core client for 30 sec - exiting 18:37:12 (6300): No heartbeat from core client for 30 sec - exiting 18:37:13 (6300): No heartbeat from core client for 30 sec - exiting 18:37:14 (6300): No heartbeat from core client for 30 sec - exiting 18:37:16 (6300): No heartbeat from core client for 30 sec - exiting 18:37:17 (6300): No heartbeat from core client for 30 sec - exiting 18:37:18 (6300): No heartbeat from core client for 30 sec - exiting 18:37:19 (6300): No heartbeat from core client for 30 sec - exiting 18:37:20 (6300): No heartbeat from core client for 30 sec - exiting 18:37:21 (6300): No heartbeat from core client for 30 sec - exiting 18:37:22 (6300): No heartbeat from core client for 30 sec - exiting 18:37:23 (6300): No heartbeat from core client for 30 sec - exiting 18:37:25 (6300): No heartbeat from core client for 30 sec - exiting 18:37:26 (6300): No heartbeat from core client for 30 sec - exiting 18:37:27 (6300): No heartbeat from core client for 30 sec - exiting 18:37:28 (6300): No heartbeat from core client for 30 sec - exiting 18:37:29 (6300): No heartbeat from core client for 30 sec - exiting 18:37:30 (6300): No heartbeat from core client for 30 sec - exiting 18:37:31 (6300): No heartbeat from core client for 30 sec - exiting 18:37:32 (6300): No heartbeat from core client for 30 sec - exiting 18:37:33 (6300): No heartbeat from core client for 30 sec - exiting 18:37:34 (6300): No heartbeat from core client for 30 sec - exiting 18:37:35 (6300): No heartbeat from core client for 30 sec - exiting 18:37:37 (6300): No heartbeat from core client for 30 sec - exiting 18:37:38 (6300): No heartbeat from core client for 30 sec - exiting 18:37:39 (6300): No heartbeat from core client for 30 sec - exiting 18:37:40 (6300): No heartbeat from core client for 30 sec - exiting 18:37:41 (6300): No heartbeat from core client for 30 sec - exiting 18:37:42 (6300): No heartbeat from core client for 30 sec - exiting 18:37:43 (6300): No heartbeat from core client for 30 sec - exiting 18:37:44 (6300): No heartbeat from core client for 30 sec - exiting 18:37:45 (6300): No heartbeat from core client for 30 sec - exiting 18:37:46 (6300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:37:47 (6300): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... MainError: 11:19:43 AM No files match the supplied pattern. MainError: 11:19:43 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 04:49:32 AM No files match the supplied pattern. MainError: 04:49:32 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 12:52:27 AM No files match the supplied pattern. MainError: 12:52:27 AM No files match the supplied pattern. MainError: 06:29:25 PM No files match the supplied pattern. MainError: 06:29:25 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 01:07:24 AM No files match the supplied pattern. MainError: 01:07:24 AM No files match the supplied pattern. MainError: 07:55:21 AM No files match the supplied pattern. MainError: 07:55:21 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Error converting file to netcdf: dataout/o25eka.ph11c10 Error converting file to netcdf: dataout/o25eka.pg11c10 Error converting file to netcdf: dataout/o25eka.pe11c10 MainError: 10:31:36 AM No files match the supplied pattern. MainError: 10:31:36 AM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Mar 2013 10:32:40	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	777,600	796,413	1.0242
24 Mar 2013 07:56:53	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	751,680	772,233	1.0273
24 Mar 2013 01:09:33	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	725,760	747,826	1.0304
23 Mar 2013 19:39:59	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	699,840	725,306	1.0364
23 Mar 2013 12:54:53	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	673,920	705,285	1.0465
23 Mar 2013 05:21:57	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	648,000	678,203	1.0466
22 Mar 2013 11:44:55	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	622,080	651,799	1.0478
21 Mar 2013 09:19:23	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	596,160	624,647	1.0478
17 Mar 2013 12:04:43	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	570,240	597,982	1.0486
14 Mar 2013 09:40:05	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	544,320	571,451	1.0498
13 Mar 2013 09:15:00	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	518,400	544,754	1.0508
06 Mar 2013 06:50:22	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	492,480	517,564	1.0509
05 Mar 2013 09:00:57	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	466,560	490,254	1.0508
24 Feb 2013 05:24:04	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	440,640	462,794	1.0503
23 Feb 2013 06:11:36	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	414,720	435,599	1.0503
07 Feb 2013 14:26:03	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	388,800	408,158	1.0498
04 Feb 2013 09:12:53	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	362,880	380,952	1.0498
28 Jan 2013 06:59:13	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	336,960	355,453	1.0549
25 Jan 2013 10:55:41	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	311,040	328,654	1.0566
24 Jan 2013 08:21:03	1235099	15502816	hadcm3n_o25e_2140_40_008269913_0	285,120	302,007	1.0592