Task 13649226

Name	hadcm3n_y8pn_1900_40_007522161_2
Workunit	7719636
Created	20 Nov 2011, 14:06:30 UTC
Sent	20 Nov 2011, 14:13:00 UTC
Report deadline	19 Feb 2012, 21:40:11 UTC
Received	16 Feb 2012, 20:27:03 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1128791
Run time	57 days 16 hours 42 min 9 sec
CPU time	39 days 13 hours 53 min 9 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	1.80 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6000, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5504, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 19:36:01 (4220): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10588, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10588, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10588, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10588, iMonCtr=1 Model crash detected, will try to restart... 20:16:37 (1264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:16:38 (1264): No heartbeat from core client for 30 sec - exiting 20:16:39 (1264): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4696, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4400, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. 18:51:28 (5824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5724, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4388, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. 15:12:30 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4224, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5364, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... C09:15:55 (5108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:15:57 (5108): No heartbeat from core client for 30 sec - exiting 09:15:58 (5108): No heartbeat from core client for 30 sec - exiting 09:15:59 (5108): No heartbeat from core client for 30 sec - exiting 09:16:00 (5108): No heartbeat from core client for 30 sec - exiting 09:16:01 (5108): No heartbeat from core client for 30 sec - exiting 09:16:02 (5108): No heartbeat from core client for 30 sec - exiting 09:16:03 (5108): No heartbeat from core client for 30 sec - exiting 09:16:04 (5108): No heartbeat from core client for 30 sec - exiting 09:16:05 (5108): No heartbeat from core client for 30 sec - exiting 09:16:06 (5108): No heartbeat from core client for 30 sec - exiting 18:46:35 (6920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:46:37 (6920): No heartbeat from core client for 30 sec - exiting 09:12:42 (6096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:59:37 (3448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:59:38 (3448): No heartbeat from core client for 30 sec - exiting 19:59:39 (3448): No heartbeat from core client for 30 sec - exiting 22:27:07 (6952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:08 (6952): No heartbeat from core client for 30 sec - exiting 22:27:09 (6952): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2900, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 01:46:42 (5436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:46:43 (5436): No heartbeat from core client for 30 sec - exiting 01:46:44 (5436): No heartbeat from core client for 30 sec - exiting 01:46:45 (5436): No heartbeat from core client for 30 sec - exiting 01:46:46 (5436): No heartbeat from core client for 30 sec - exiting 01:46:47 (5436): No heartbeat from core client for 30 sec - exiting 01:46:48 (5436): No heartbeat from core client for 30 sec - exiting 01:46:49 (5436): No heartbeat from core client for 30 sec - exiting 01:46:50 (5436): No heartbeat from core client for 30 sec - exiting 01:46:51 (5436): No heartbeat from core client for 30 sec - exiting 01:46:52 (5436): No heartbeat from core client for 30 sec - exiting 09:57:39 (11896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:53:09 (11728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:53:10 (11728): No heartbeat from core client for 30 sec - exiting 10:53:11 (11728): No heartbeat from core client for 30 sec - exiting 11:55:06 (9872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:55:07 (9872): No heartbeat from core client for 30 sec - exiting 19:16:47 (10300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:16:48 (10300): No heartbeat from core client for 30 sec - exiting 19:16:49 (10300): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:17:33 (5316): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 22:17:35 (5316): No heartbeat from core client for 30 sec - exiting 22:17:36 (5316): No heartbeat from core client for 30 sec - exiting 22:17:37 (5316): No heartbeat from core client for 30 sec - exiting 02:51:15 (10700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:51:16 (10700): No heartbeat from core client for 30 sec - exiting 04:11:01 (9828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:11:03 (9828): No heartbeat from core client for 30 sec - exiting 04:11:04 (9828): No heartbeat from core client for 30 sec - exiting 04:11:05 (9828): No heartbeat from core client for 30 sec - exiting 04:11:06 (9828): No heartbeat from core client for 30 sec - exiting 04:11:07 (9828): No heartbeat from core client for 30 sec - exiting 04:11:08 (9828): No heartbeat from core client for 30 sec - exiting 04:21:39 (9176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:43:56 (11428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:49:27 (7488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. 06:34:50 (10552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4188, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5600, iMonCtr=1 Model crash detected, will try to restart... 05:49:07 (5584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:49:09 (5584): No heartbeat from core client for 30 sec - exiting 05:49:10 (5584): No heartbeat from core client for 30 sec - exiting 05:49:11 (5584): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:45:03 (5976): No heartbeat from core client for 30 sec - exiting 13:45:04 (5976): No heartbeat from core client for 30 sec - exiting 13:45:05 (5976): No heartbeat from core client for 30 sec - exiting 13:45:06 (5976): No heartbeat from core client for 30 sec - exiting 13:45:07 (5976): No heartbeat from core client for 30 sec - exiting 13:45:08 (5976): No heartbeat from core client for 30 sec - exiting 13:45:09 (5976): No heartbeat from core client for 30 sec - exiting 13:45:10 (5976): No heartbeat from core client for 30 sec - exiting 13:45:11 (5976): No heartbeat from core client for 30 sec - exiting 13:45:12 (5976): No heartbeat from core client for 30 sec - exiting 13:45:13 (5976): No heartbeat from core client for 30 sec - exiting 13:45:14 (5976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. 20:47:06 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. 21:33:34 (5612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:33:35 (5612): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. 05:19:21 (6492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. 05:39:02 (7060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 14:02:28 (3176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5844, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 00:34:30 (2228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2324, iMonCtr=1 Model crash detected, will try to restart... 18:06:40 (5264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:23:37 (5472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:23:38 (5472): No heartbeat from core client for 30 sec - exiting 08:23:39 (5472): No heartbeat from core client for 30 sec - exiting 09:07:16 (6716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:07:17 (6716): No heartbeat from core client for 30 sec - exiting 09:07:18 (6716): No heartbeat from core client for 30 sec - exiting 09:07:19 (6716): No heartbeat from core client for 30 sec - exiting 09:07:20 (6716): No heartbeat from core client for 30 sec - exiting 09:07:21 (6716): No heartbeat from core client for 30 sec - exiting 09:07:22 (6716): No heartbeat from core client for 30 sec - exiting 09:07:23 (6716): No heartbeat from core client for 30 sec - exiting 09:07:24 (6716): No heartbeat from core client for 30 sec - exiting 09:07:25 (6716): No heartbeat from core client for 30 sec - exiting 09:07:26 (6716): No heartbeat from core client for 30 sec - exiting 09:15:42 (7360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:15:43 (7360): No heartbeat from core client for 30 sec - exiting 09:15:44 (7360): No heartbeat from core client for 30 sec - exiting 09:15:45 (7360): No heartbeat from core client for 30 sec - exiting 09:15:46 (7360): No heartbeat from core client for 30 sec - exiting 09:15:47 (7360): No heartbeat from core client for 30 sec - exiting 09:15:48 (7360): No heartbeat from core client for 30 sec - exiting 10:39:20 (1332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:01:41 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:01:42 (5504): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 20:44:31 (2932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:20:00 (6912): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 09:20:02 (6912): No heartbeat from core client for 30 sec - exiting 09:20:03 (6912): No heartbeat from core client for 30 sec - exiting 13:02:38 (7004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5028, iMonCtr=1 Model crash detected, will try to restart... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Feb 2012 16:07:02	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	1,036,800	3,418,269	3.2969
15 Feb 2012 04:46:32	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	1,010,880	3,329,175	3.2933
13 Feb 2012 12:42:30	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	984,960	3,236,255	3.2857
11 Feb 2012 19:12:48	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	959,040	3,141,882	3.2761
09 Feb 2012 22:11:54	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	933,120	3,045,334	3.2636
08 Feb 2012 05:55:13	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	907,200	2,949,289	3.2510
07 Feb 2012 03:30:53	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	881,280	2,872,299	3.2592
05 Feb 2012 20:30:13	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	855,360	2,787,343	3.2587
04 Feb 2012 12:04:27	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	829,440	2,703,028	3.2589
02 Feb 2012 19:00:08	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	803,520	2,612,640	3.2515
01 Feb 2012 02:28:13	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	777,600	2,521,210	3.2423
30 Jan 2012 14:19:16	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	751,680	2,436,501	3.2414
23 Jan 2012 00:13:21	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	725,760	2,378,372	3.2771
21 Jan 2012 11:11:17	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	699,840	2,276,512	3.2529
20 Jan 2012 01:23:09	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	673,920	2,174,650	3.2269
18 Jan 2012 17:26:21	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	648,000	2,076,632	3.2047
17 Jan 2012 08:13:02	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	622,080	1,978,592	3.1806
15 Jan 2012 21:00:54	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	596,160	1,887,823	3.1666
14 Jan 2012 14:43:25	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	570,240	1,802,442	3.1608
13 Jan 2012 05:26:58	1128791	13649226	hadcm3n_y8pn_1900_40_007522161_2	544,320	1,710,773	3.1430