Task 13667329

Name	hadcm3n_u11h_1980_40_007545745_1
Workunit	7742977
Created	28 Nov 2011, 23:51:40 UTC
Sent	28 Nov 2011, 23:53:16 UTC
Report deadline	28 Feb 2012, 7:20:27 UTC
Received	17 Dec 2011, 17:36:45 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	943197
Run time	6 days 10 hours 9 min 13 sec
CPU time	4 days 20 hours 14 min 53 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.20 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_d23:54:29 (916): No heartbeat from core client for 30 sec - exiting 23:54:30 (916): No heartbeat from core client for 30 sec - exiting 23:54:31 (916): No heartbeat from core client for 30 sec - exiting 23:54:33 (916): No heartbeat from core client for 30 sec - exiting 23:54:34 (916): No heartbeat from core client for 30 sec - exiting 23:54:35 (916): No heartbeat from core client for 30 sec - exiting 23:54:36 (916): No heartbeat from core client for 30 sec - exiting 23:54:37 (916): No heartbeat from core client for 30 sec - exiting 23:54:38 (916): No heartbeat from core client for 30 sec - exiting 23:54:39 (916): No heartbeat from core client for 30 sec - exiting 23:54:40 (916): No heartbeat from core client for 30 sec - exiting 23:54:41 (916): No heartbeat from core client for 30 sec - exiting 23:54:42 (916): No heartbeat from core client for 30 sec - exiting 23:54:44 (916): No heartbeat from core client for 30 sec - exiting 23:54:45 (916): No heartbeat from core client for 30 sec - exiting 23:54:46 (916): No heartbeat from core client for 30 sec - exiting 23:54:47 (916): No heartbeat from core client for 30 sec - exiting 23:54:48 (916): No heartbeat from core client for 30 sec - exiting 23:54:49 (916): No heartbeat from core client for 30 sec - exiting 23:54:50 (916): No heartbeat from core client for 30 sec - exiting 23:54:51 (916): No heartbeat from core client for 30 sec - exiting 23:54:52 (916): No heartbeat from core client for 30 sec - exiting 23:54:53 (916): No heartbeat from core client for 30 sec - exiting 23:54:54 (916): No heartbeat from core client for 30 sec - exiting 23:54:56 (916): No heartbeat from core client for 30 sec - exiting 23:54:57 (916): No heartbeat from core client for 30 sec - exiting 23:54:58 (916): No heartbeat from core client for 30 sec - exiting 23:54:59 (916): No heartbeat from core client for 30 sec - exiting 23:55:00 (916): No heartbeat from core client for 30 sec - exiting 23:55:01 (916): No heartbeat from core client for 30 sec - exiting 23:55:02 (916): No heartbeat from core client for 30 sec - exiting 23:55:03 (916): No heartbeat from core client for 30 sec - exiting 23:55:04 (916): No heartbeat from core client for 30 sec - exiting 23:55:05 (916): No heartbeat from core client for 30 sec - exiting 23:55:07 (916): No heartbeat from core client for 30 sec - exiting 23:55:08 (916): No heartbeat from core client for 30 sec - exiting 23:55:09 (916): No heartbeat from core client for 30 sec - exiting 23:55:10 (916): No heartbeat from core client for 30 sec - exiting 23:55:11 (916): No heartbeat from core client for 30 sec - exiting 23:55:12 (916): No heartbeat from core client for 30 sec - exiting 23:55:13 (916): No heartbeat from core client for 30 sec - exiting 23:55:14 (916): No heartbeat from core client for 30 sec - exiting 23:55:15 (916): No heartbeat from core client for 30 sec - exiting 23:55:16 (916): No heartbeat from core client for 30 sec - exiting 23:55:17 (916): No heartbeat from core client for 30 sec - exiting 23:55:19 (916): No heartbeat from core client for 30 sec - exiting 23:55:20 (916): No heartbeat from core client for 30 sec - exiting 23:55:21 (916): No heartbeat from core client for 30 sec - exiting 23:55:22 (916): No heartbeat from core client for 30 sec - exiting 23:55:23 (916): No heartbeat from core client for 30 sec - exiting 23:55:24 (916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:55:25 (916): No heartbeat from core client for 30 sec - exiting 23:55:26 (916): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4824, iMonCtr=1 Model crash detected, will try to restart... 23:55:08 (4592): No heartbeat from core client for 30 sec - exiting 23:55:09 (4592): No heartbeat from core client for 30 sec - exiting 23:55:10 (4592): No heartbeat from core client for 30 sec - exiting 23:55:11 (4592): No heartbeat from core client for 30 sec - exiting 23:55:13 (4592): No heartbeat from core client for 30 sec - exiting 23:55:14 (4592): No heartbeat from core client for 30 sec - exiting 23:55:15 (4592): No heartbeat from core client for 30 sec - exiting 23:55:16 (4592): No heartbeat from core client for 30 sec - exiting 23:55:17 (4592): No heartbeat from core client for 30 sec - exiting 23:55:18 (4592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 16:32:25 (5084): No heartbeat from core client for 30 sec - exiting 16:32:27 (5084): No heartbeat from core client for 30 sec - exiting 16:32:28 (5084): No heartbeat from core client for 30 sec - exiting 16:32:29 (5084): No heartbeat from core client for 30 sec - exiting 16:32:30 (5084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:53:56 (5080): No heartbeat from core client for 30 sec - exiting 23:53:57 (5080): No heartbeat from core client for 30 sec - exiting 23:53:58 (5080): No heartbeat from core client for 30 sec - exiting 23:53:59 (5080): No heartbeat from core client for 30 sec - exiting 23:54:00 (5080): No heartbeat from core client for 30 sec - exiting 23:54:01 (5080): No heartbeat from core client for 30 sec - exiting 23:54:02 (5080): No heartbeat from core client for 30 sec - exiting 23:54:03 (5080): No heartbeat from core client for 30 sec - exiting 23:54:04 (5080): No heartbeat from core client for 30 sec - exiting 23:54:05 (5080): No heartbeat from core client for 30 sec - exiting 23:54:07 (5080): No heartbeat from core client for 30 sec - exiting 23:54:08 (5080): No heartbeat from core client for 30 sec - exiting 23:54:09 (5080): No heartbeat from core client for 30 sec - exiting 23:54:10 (5080): No heartbeat from core client for 30 sec - exiting 23:54:11 (5080): No heartbeat from core client for 30 sec - exiting 23:54:12 (5080): No heartbeat from core client for 30 sec - exiting 23:54:13 (5080): No heartbeat from core client for 30 sec - exiting 23:54:14 (5080): No heartbeat from core client for 30 sec - exiting 23:54:15 (5080): No heartbeat from core client for 30 sec - exiting 23:54:16 (5080): No heartbeat from core client for 30 sec - exiting 23:54:17 (5080): No heartbeat from core client for 30 sec - exiting 23:54:19 (5080): No heartbeat from core client for 30 sec - exiting 23:54:20 (5080): No heartbeat from core client for 30 sec - exiting 23:54:21 (5080): No heartbeat from core client for 30 sec - exiting 23:54:22 (5080): No heartbeat from core client for 30 sec - exiting 23:54:23 (5080): No heartbeat from core client for 30 sec - exiting 23:54:24 (5080): No heartbeat from core client for 30 sec - exiting 23:54:25 (5080): No heartbeat from core client for 30 sec - exiting 23:54:26 (5080): No heartbeat from core client for 30 sec - exiting 23:54:27 (5080): No heartbeat from core client for 30 sec - exiting 23:54:28 (5080): No heartbeat from core client for 30 sec - exiting 23:54:29 (5080): No heartbeat from core client for 30 sec - exiting 23:54:31 (5080): No heartbeat from core client for 30 sec - exiting 23:54:32 (5080): No heartbeat from core client for 30 sec - exiting 23:54:33 (5080): No heartbeat from core client for 30 sec - exiting 23:54:34 (5080): No heartbeat from core client for 30 sec - exiting 23:54:35 (5080): No heartbeat from core client for 30 sec - exiting 23:54:36 (5080): No heartbeat from core client for 30 sec - exiting 23:54:37 (5080): No heartbeat from core client for 30 sec - exiting 23:54:38 (5080): No heartbeat from core client for 30 sec - exiting 23:54:39 (5080): No heartbeat from core client for 30 sec - exiting 23:54:40 (5080): No heartbeat from core client for 30 sec - exiting 23:54:42 (5080): No heartbeat from core client for 30 sec - exiting 23:54:43 (5080): No heartbeat from core client for 30 sec - exiting 23:54:44 (5080): No heartbeat from core client for 30 sec - exiting 23:54:45 (5080): No heartbeat from core client for 30 sec - exiting 23:54:46 (5080): No heartbeat from core client for 30 sec - exiting 23:54:47 (5080): No heartbeat from core client for 30 sec - exiting 23:54:48 (5080): No heartbeat from core client for 30 sec - exiting 23:54:49 (5080): No heartbeat from core client for 30 sec - exiting 23:54:50 (5080): No heartbeat from core client for 30 sec - exiting 23:54:51 (5080): No heartbeat from core client for 30 sec - exiting 23:54:52 (5080): No heartbeat from core client for 30 sec - exiting 23:54:54 (5080): No heartbeat from core client for 30 sec - exiting 23:54:55 (5080): No heartbeat from core client for 30 sec - exiting 23:54:56 (5080): No heartbeat from core client for 30 sec - exiting 23:54:57 (5080): No heartbeat from core client for 30 sec - exiting 23:54:58 (5080): No heartbeat from core client for 30 sec - exiting 23:54:59 (5080): No heartbeat from core client for 30 sec - exiting 23:55:00 (5080): No heartbeat from core client for 30 sec - exiting 23:55:01 (5080): No heartbeat from core client for 30 sec - exiting 23:55:02 (5080): No heartbeat from core client for 30 sec - exiting 23:55:03 (5080): No heartbeat from core client for 30 sec - exiting 23:55:04 (5080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:31:00 (2604): No heartbeat from core client for 30 sec - exiting 01:31:01 (2604): No heartbeat from core client for 30 sec - exiting 01:31:02 (2604): No heartbeat from core client for 30 sec - exiting 01:31:03 (2604): No heartbeat from core client for 30 sec - exiting 01:31:04 (2604): No heartbeat from core client for 30 sec - exiting 01:31:05 (2604): No heartbeat from core client for 30 sec - exiting 01:31:06 (2604): No heartbeat from core client for 30 sec - exiting 01:31:07 (2604): No heartbeat from core client for 30 sec - exiting 01:31:08 (2604): No heartbeat from core client for 30 sec - exiting 01:31:09 (2604): No heartbeat from core client for 30 sec - exiting 01:31:11 (2604): No heartbeat from core client for 30 sec - exiting 01:31:12 (2604): No heartbeat from core client for 30 sec - exiting 01:31:13 (2604): No heartbeat from core client for 30 sec - exiting 01:31:14 (2604): No heartbeat from core client for 30 sec - exiting 01:31:15 (2604): No heartbeat from core client for 30 sec - exiting 01:31:16 (2604): No heartbeat from core client for 30 sec - exiting 01:31:17 (2604): No heartbeat from core client for 30 sec - exiting 01:31:18 (2604): No heartbeat from core client for 30 sec - exiting 01:31:19 (2604): No heartbeat from core client for 30 sec - exiting 01:31:20 (2604): No heartbeat from core client for 30 sec - exiting 01:31:21 (2604): No heartbeat from core client for 30 sec - exiting 01:31:23 (2604): No heartbeat from core client for 30 sec - exiting 01:31:24 (2604): No heartbeat from core client for 30 sec - exiting 01:31:25 (2604): No heartbeat from core client for 30 sec - exiting 01:31:26 (2604): No heartbeat from core client for 30 sec - exiting 01:31:27 (2604): No heartbeat from core client for 30 sec - exiting 01:31:28 (2604): No heartbeat from core client for 30 sec - exiting 01:31:29 (2604): No heartbeat from core client for 30 sec - exiting 01:31:30 (2604): No heartbeat from core client for 30 sec - exiting 01:31:31 (2604): No heartbeat from core client for 30 sec - exiting 01:31:32 (2604): No heartbeat from core client for 30 sec - exiting 01:31:33 (2604): No heartbeat from core client for 30 sec - exiting 01:31:35 (2604): No heartbeat from core client for 30 sec - exiting 01:31:36 (2604): No heartbeat from core client for 30 sec - exiting 01:31:37 (2604): No heartbeat from core client for 30 sec - exiting 01:31:38 (2604): No heartbeat from core client for 30 sec - exiting 01:31:39 (2604): No heartbeat from core client for 30 sec - exiting 01:31:40 (2604): No heartbeat from core client for 30 sec - exiting 01:31:41 (2604): No heartbeat from core client for 30 sec - exiting 01:31:42 (2604): No heartbeat from core client for 30 sec - exiting 01:31:43 (2604): No heartbeat from core client for 30 sec - exiting 01:31:44 (2604): No heartbeat from core client for 30 sec - exiting 01:31:46 (2604): No heartbeat from core client for 30 sec - exiting 01:31:47 (2604): No heartbeat from core client for 30 sec - exiting 01:31:48 (2604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:27:00 (4716): No heartbeat from core client for 30 sec - exiting 14:27:01 (4716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:54:36 (3844): No heartbeat from core client for 30 sec - exiting 23:54:37 (3844): No heartbeat from core client for 30 sec - exiting 23:54:38 (3844): No heartbeat from core client for 30 sec - exiting 23:54:39 (3844): No heartbeat from core client for 30 sec - exiting 23:54:40 (3844): No heartbeat from core client for 30 sec - exiting 23:54:41 (3844): No heartbeat from core client for 30 sec - exiting 23:54:42 (3844): No heartbeat from core client for 30 sec - exiting 23:54:44 (3844): No heartbeat from core client for 30 sec - exiting 23:54:45 (3844): No heartbeat from core client for 30 sec - exiting 23:54:46 (3844): No heartbeat from core client for 30 sec - exiting 23:54:47 (3844): No heartbeat from core client for 30 sec - exiting 23:54:48 (3844): No heartbeat from core client for 30 sec - exiting 23:54:49 (3844): No heartbeat from core client for 30 sec - exiting 23:54:50 (3844): No heartbeat from core client for 30 sec - exiting 23:54:51 (3844): No heartbeat from core client for 30 sec - exiting 23:54:52 (3844): No heartbeat from core client for 30 sec - exiting 23:54:53 (3844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:54:27 (4936): No heartbeat from core client for 30 sec - exiting 23:54:28 (4936): No heartbeat from core client for 30 sec - exiting 23:54:29 (4936): No heartbeat from core client for 30 sec - exiting 23:54:30 (4936): No heartbeat from core client for 30 sec - exiting 23:54:31 (4936): No heartbeat from core client for 30 sec - exiting 23:54:32 (4936): No heartbeat from core client for 30 sec - exiting 23:54:34 (4936): No heartbeat from core client for 30 sec - exiting 23:54:35 (4936): No heartbeat from core client for 30 sec - exiting 23:54:36 (4936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4732, iMonCtr=1 Model crash detected, will try to restart... 23:55:18 (4880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:55:19 (4808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... C15:11:34 (5112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Dec 2011 16:40:37	943197	13667329	hadcm3n_u11h_1980_40_007545745_1	259,200	418,491	1.6145
16 Dec 2011 03:40:14	943197	13667329	hadcm3n_u11h_1980_40_007545745_1	233,280	372,798	1.5981
14 Dec 2011 04:56:35	943197	13667329	hadcm3n_u11h_1980_40_007545745_1	207,360	326,236	1.5733
12 Dec 2011 06:27:48	943197	13667329	hadcm3n_u11h_1980_40_007545745_1	181,440	279,827	1.5423
11 Dec 2011 00:54:41	943197	13667329	hadcm3n_u11h_1980_40_007545745_1	155,520	234,261	1.5063
09 Dec 2011 03:46:28	943197	13667329	hadcm3n_u11h_1980_40_007545745_1	129,600	188,465	1.4542
07 Dec 2011 05:29:50	943197	13667329	hadcm3n_u11h_1980_40_007545745_1	103,680	170,491	1.6444
06 Dec 2011 00:57:11	943197	13667329	hadcm3n_u11h_1980_40_007545745_1	77,760	124,877	1.6059
02 Dec 2011 04:30:45	943197	13667329	hadcm3n_u11h_1980_40_007545745_1	51,840	79,656	1.5366
30 Nov 2011 05:55:38	943197	13667329	hadcm3n_u11h_1980_40_007545745_1	25,920	34,054	1.3138