Task 15921350

Name	hadcm3n_o1t6_1980_40_008402823_1
Workunit	8553679
Created	15 Aug 2013, 0:04:58 UTC
Sent	15 Aug 2013, 0:06:07 UTC
Report deadline	14 Nov 2013, 7:33:18 UTC
Received	1 Oct 2013, 9:29:37 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	871746
Run time	17 days 22 hours 25 min 17 sec
CPU time	16 days 8 hours 43 min 13 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	3.48 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 00:16:41 (11786): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:16:42 (11786): No heartbeat from core client for 30 sec - exiting 00:16:43 (11786): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:17:56 (268): No heartbeat from core client for 30 sec - exiting 10:17:58 (268): No heartbeat from core client for 30 sec - exiting 10:17:59 (268): No heartbeat from core client for 30 sec - exiting 10:18:00 (268): No heartbeat from core client for 30 sec - exiting 10:18:01 (268): No heartbeat from core client for 30 sec - exiting 10:18:02 (268): No heartbeat from core client for 30 sec - exiting 10:18:03 (268): No heartbeat from core client for 30 sec - exiting 10:18:04 (268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:45:04 (258): No heartbeat from core client for 30 sec - exiting 16:45:05 (258): No heartbeat from core client for 30 sec - exiting 16:45:06 (258): No heartbeat from core client for 30 sec - exiting 16:45:07 (258): No heartbeat from core client for 30 sec - exiting 16:45:08 (258): No heartbeat from core client for 30 sec - exiting 16:45:09 (258): No heartbeat from core client for 30 sec - exiting 16:45:10 (258): No heartbeat from core client for 30 sec - exiting 16:45:11 (258): No heartbeat from core client for 30 sec - exiting 16:45:14 (258): No heartbeat from core client for 30 sec - exiting 16:45:15 (258): No heartbeat from core client for 30 sec - exiting 16:45:16 (258): No heartbeat from core client for 30 sec - exiting 16:45:17 (258): No heartbeat from core client for 30 sec - exiting 16:45:18 (258): No heartbeat from core client for 30 sec - exiting 16:45:19 (258): No heartbeat from core client for 30 sec - exiting 16:45:20 (258): No heartbeat from core client for 30 sec - exiting 16:45:21 (258): No heartbeat from core client for 30 sec - exiting 16:45:22 (258): No heartbeat from core client for 30 sec - exiting 16:45:23 (258): No heartbeat from core client for 30 sec - exiting 16:45:24 (258): No heartbeat from core client for 30 sec - exiting 16:45:25 (258): No heartbeat from core client for 30 sec - exiting 16:45:26 (258): No heartbeat from core client for 30 sec - exiting 16:45:27 (258): No heartbeat from core client for 30 sec - exiting 16:45:28 (258): No heartbeat from core client for 30 sec - exiting 16:45:29 (258): No heartbeat from core client for 30 sec - exiting 16:45:30 (258): No heartbeat from core client for 30 sec - exiting 16:45:31 (258): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 17:27:44 (290): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:27:45 (290): No heartbeat from core client for 30 sec - exiting 17:27:46 (290): No heartbeat from core client for 30 sec - exiting BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 OPEN: File Creation Failed: No space left on device OPEN: Unable to Open File dataout/o1t6ko.daj8630 for Read/Write Model crashed: DUMPCTL : Fail to open output dump - may already exist tmp/pipe_dummy 2048 21:09:06 (2775): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: No space left onController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=254, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 08:43:26 (254): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(3057,0xa01f9540) malloc: * error for object 0x2808e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3057,0xa01f9540) malloc: * error for object 0x2809c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3057,0xa01f9540) malloc: * error for object 0x2809c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3057,0xa01f9540) malloc: * error for object 0x2809c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3057,0xa01f9540) malloc: * error for object 0x2809c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3057,0xa01f9540) malloc: * error for object 0x2809c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3057,0xa01f9540) malloc: * error for object 0x2809c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3057,0xa01f9540) malloc: * error for object 0x2809c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3057,0xa01f9540) malloc: * error for object 0x2809c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3057,0xa01f9540) malloc: * error for object 0x2809c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3057,0xa01f9540) malloc: * error for object 0x2809c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Oct 2013 07:04:57	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	1,036,800	1,413,254	1.3631
30 Sep 2013 20:40:09	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	1,010,880	1,377,950	1.3631
30 Sep 2013 10:10:54	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	984,960	1,342,611	1.3631
30 Sep 2013 00:26:25	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	959,040	1,307,050	1.3629
29 Sep 2013 09:28:57	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	933,120	1,270,846	1.3619
28 Sep 2013 00:51:14	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	907,200	1,235,540	1.3619
27 Sep 2013 14:19:50	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	881,280	1,199,879	1.3615
27 Sep 2013 03:35:39	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	855,360	1,164,206	1.3611
26 Sep 2013 16:55:48	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	829,440	1,127,924	1.3599
25 Sep 2013 13:42:44	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	803,520	1,091,653	1.3586
20 Sep 2013 19:41:25	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	777,600	1,056,709	1.3589
19 Sep 2013 13:41:34	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	751,680	1,021,373	1.3588
18 Sep 2013 02:44:23	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	725,760	985,485	1.3579
17 Sep 2013 16:18:12	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	699,840	950,039	1.3575
17 Sep 2013 06:07:18	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	673,920	915,167	1.3580
16 Sep 2013 19:39:50	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	648,000	879,980	1.3580
16 Sep 2013 09:02:03	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	622,080	844,638	1.3578
15 Sep 2013 23:18:08	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	596,160	809,072	1.3571
15 Sep 2013 11:49:59	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	570,240	772,839	1.3553
15 Sep 2013 00:31:47	871746	15921350	hadcm3n_o1t6_1980_40_008402823_1	544,320	736,819	1.3537