Task 15480628

Name	hadcm3n_zl1m_1920_40_008256016_0
Workunit	8411140
Created	16 Dec 2012, 19:30:29 UTC
Sent	16 Dec 2012, 19:30:32 UTC
Report deadline	18 Mar 2013, 2:57:43 UTC
Received	21 Mar 2013, 12:34:30 UTC
Server state	Over
Outcome	Computation error
Client state	Aborted by user
Exit status	-197 (0xFFFFFF3B) ERR_ABORTED_VIA_GUI
Computer ID	1239754
Run time	33 days 4 hours 20 min 15 sec
CPU time	17 days 11 hours 44 min 37 sec
Validate state	Invalid
Credit	9,642.24
Device peak FLOPS	2.67 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> aborted by user </message> <stderr_txt> RetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 09:07:59 (8288): No heartbeat from core client for 30 sec - exiting 09:09:49 (8288): No heartbeat from core client for 30 sec - exiting 09:09:50 (8288): No heartbeat from core client for 30 sec - exiting 09:09:51 (8288): No heartbeat from core client for 30 sec - exiting 09:09:52 (8288): No heartbeat from core client for 30 sec - exiting 09:09:53 (8288): No heartbeat from core client for 30 sec - exiting 09:33:07 (8288): No heartbeat from core client for 30 sec - exiting 09:33:08 (8288): No heartbeat from core client for 30 sec - exiting 09:33:09 (8288): No heartbeat from core client for 30 sec - exiting 09:33:10 (8288): No heartbeat from core client for 30 sec - exiting 09:33:11 (8288): No heartbeat from core client for 30 sec - exiting 09:33:12 (8288): No heartbeat from core client for 30 sec - exiting 09:33:13 (8288): No heartbeat from core client for 30 sec - exiting 09:33:14 (8288): No heartbeat from core client for 30 sec - exiting 09:33:15 (8288): No heartbeat from core client for 30 sec - exiting 09:33:16 (8288): No heartbeat from core client for 30 sec - exiting 09:33:17 (8288): No heartbeat from core client for 30 sec - exiting 09:33:52 (8288): No heartbeat from core client for 30 sec - exiting 09:33:53 (8288): No heartbeat from core client for 30 sec - exiting 09:33:54 (8288): No heartbeat from core client for 30 sec - exiting 09:33:55 (8288): No heartbeat from core client for 30 sec - exiting 09:33:56 (8288): No heartbeat from core client for 30 sec - exiting 09:33:57 (8288): No heartbeat from core client for 30 sec - exiting 09:33:58 (8288): No heartbeat from core client for 30 sec - exiting 09:33:59 (8288): No heartbeat from core client for 30 sec - exiting 09:34:00 (8288): No heartbeat from core client for 30 sec - exiting 09:34:01 (8288): No heartbeat from core client for 30 sec - exiting 09:34:02 (8288): No heartbeat from core client for 30 sec - exiting 09:35:03 (8288): No heartbeat from core client for 30 sec - exiting 09:35:04 (8288): No heartbeat from core client for 30 sec - exiting 09:35:05 (8288): No heartbeat from core client for 30 sec - exiting 09:35:06 (8288): No heartbeat from core client for 30 sec - exiting 09:35:07 (8288): No heartbeat from core client for 30 sec - exiting 09:35:08 (8288): No heartbeat from core client for 30 sec - exiting 09:35:09 (8288): No heartbeat from core client for 30 sec - exiting 09:35:10 (8288): No heartbeat from core client for 30 sec - exiting 09:35:11 (8288): No heartbeat from core client for 30 sec - exiting 09:35:12 (8288): No heartbeat from core client for 30 sec - exiting 09:36:39 (8288): No heartbeat from core client for 30 sec - exiting 09:36:40 (8288): No heartbeat from core client for 30 sec - exiting 09:36:41 (8288): No heartbeat from core client for 30 sec - exiting 09:36:42 (8288): No heartbeat from core client for 30 sec - exiting 09:36:43 (8288): No heartbeat from core client for 30 sec - exiting 09:36:44 (8288): No heartbeat from core client for 30 sec - exiting 09:36:45 (8288): No heartbeat from core client for 30 sec - exiting 09:36:46 (8288): No heartbeat from core client for 30 sec - exiting 09:36:47 (8288): No heartbeat from core client for 30 sec - exiting 09:36:48 (8288): No heartbeat from core client for 30 sec - exiting 09:36:49 (8288): No heartbeat from core client for 30 sec - exiting 02:05:06 (8288): No heartbeat from core client for 30 sec - exiting 02:05:07 (8288): No heartbeat from core client for 30 sec - exiting 02:05:08 (8288): No heartbeat from core client for 30 sec - exiting 02:05:09 (8288): No heartbeat from core client for 30 sec - exiting 02:05:10 (8288): No heartbeat from core client for 30 sec - exiting 02:05:11 (8288): No heartbeat from core client for 30 sec - exiting 02:05:12 (8288): No heartbeat from core client for 30 sec - exiting 02:05:13 (8288): No heartbeat from core client for 30 sec - exiting 02:05:14 (8288): No heartbeat from core client for 30 sec - exiting 02:05:15 (8288): No heartbeat from core client for 30 sec - exiting 02:05:16 (8288): No heartbeat from core client for 30 sec - exiting 02:05:50 (8288): No heartbeat from core client for 30 sec - exiting 02:05:51 (8288): No heartbeat from core client for 30 sec - exiting 02:05:52 (8288): No heartbeat from core client for 30 sec - exiting 02:05:53 (8288): No heartbeat from core client for 30 sec - exiting 02:05:54 (8288): No heartbeat from core client for 30 sec - exiting 02:05:55 (8288): No heartbeat from core client for 30 sec - exiting 02:05:56 (8288): No heartbeat from core client for 30 sec - exiting 02:05:57 (8288): No heartbeat from core client for 30 sec - exiting 02:05:58 (8288): No heartbeat from core client for 30 sec - exiting 02:05:59 (8288): No heartbeat from core client for 30 sec - exiting 02:06:00 (8288): No heartbeat from core client for 30 sec - exiting 02:07:01 (8288): No heartbeat from core client for 30 sec - exiting 02:07:02 (8288): No heartbeat from core client for 30 sec - exiting 02:07:03 (8288): No heartbeat from core client for 30 sec - exiting 02:07:04 (8288): No heartbeat from core client for 30 sec - exiting 02:07:05 (8288): No heartbeat from core client for 30 sec - exiting 02:07:06 (8288): No heartbeat from core client for 30 sec - exiting 02:07:07 (8288): No heartbeat from core client for 30 sec - exiting 02:07:08 (8288): No heartbeat from core client for 30 sec - exiting 02:07:09 (8288): No heartbeat from core client for 30 sec - exiting 02:07:10 (8288): No heartbeat from core client for 30 sec - exiting 02:07:11 (8288): No heartbeat from core client for 30 sec - exiting 02:09:57 (8288): No heartbeat from core client for 30 sec - exiting 02:09:58 (8288): No heartbeat from core client for 30 sec - exiting 02:09:59 (8288): No heartbeat from core client for 30 sec - exiting 02:10:00 (8288): No heartbeat from core client for 30 sec - exiting 02:10:01 (8288): No heartbeat from core client for 30 sec - exiting 02:10:02 (8288): No heartbeat from core client for 30 sec - exiting 02:10:03 (8288): No heartbeat from core client for 30 sec - exiting 02:10:04 (8288): No heartbeat from core client for 30 sec - exiting 02:10:05 (8288): No heartbeat from core client for 30 sec - exiting 02:10:06 (8288): No heartbeat from core client for 30 sec - exiting 02:10:07 (8288): No heartbeat from core client for 30 sec - exiting 02:13:44 (8288): No heartbeat from core client for 30 sec - exiting 02:13:45 (8288): No heartbeat from core client for 30 sec - exiting 02:13:46 (8288): No heartbeat from core client for 30 sec - exiting 02:13:47 (8288): No heartbeat from core client for 30 sec - exiting 02:13:48 (8288): No heartbeat from core client for 30 sec - exiting 02:13:49 (8288): No heartbeat from core client for 30 sec - exiting 02:13:50 (8288): No heartbeat from core client for 30 sec - exiting 02:13:51 (8288): No heartbeat from core client for 30 sec - exiting 02:13:52 (8288): No heartbeat from core client for 30 sec - exiting 02:13:53 (8288): No heartbeat from core client for 30 sec - exiting 02:13:54 (8288): No heartbeat from core client for 30 sec - exiting 09:13:53 (8288): No heartbeat from core client for 30 sec - exiting 09:13:54 (8288): No heartbeat from core client for 30 sec - exiting 09:13:55 (8288): No heartbeat from core client for 30 sec - exiting 09:13:56 (8288): No heartbeat from core client for 30 sec - exiting 09:13:57 (8288): No heartbeat from core client for 30 sec - exiting 09:13:58 (8288): No heartbeat from core client for 30 sec - exiting 09:13:59 (8288): No heartbeat from core client for 30 sec - exiting 12:15:42 (8288): No heartbeat from core client for 30 sec - exiting 12:15:43 (8288): No heartbeat from core client for 30 sec - exiting 12:15:44 (8288): No heartbeat from core client for 30 sec - exiting 12:15:45 (8288): No heartbeat from core client for 30 sec - exiting 12:15:46 (8288): No heartbeat from core client for 30 sec - exiting 12:15:47 (8288): No heartbeat from core client for 30 sec - exiting 12:15:48 (8288): No heartbeat from core client for 30 sec - exiting 12:15:49 (8288): No heartbeat from core client for 30 sec - exiting 12:15:50 (8288): No heartbeat from core client for 30 sec - exiting 12:15:51 (8288): No heartbeat from core client for 30 sec - exiting 12:15:52 (8288): No heartbeat from core client for 30 sec - exiting 12:15:53 (8288): No heartbeat from core client for 30 sec - exiting 12:15:54 (8288): No heartbeat from core client for 30 sec - exiting 12:15:55 (8288): No heartbeat from core client for 30 sec - exiting 12:15:56 (8288): No heartbeat from core client for 30 sec - exiting 12:15:57 (8288): No heartbeat from core client for 30 sec - exiting 12:15:58 (8288): No heartbeat from core client for 30 sec - exiting 12:17:15 (8288): No heartbeat from core client for 30 sec - exiting 12:17:16 (8288): No heartbeat from core client for 30 sec - exiting 12:17:17 (8288): No heartbeat from core client for 30 sec - exiting 12:17:18 (8288): No heartbeat from core client for 30 sec - exiting 12:17:19 (8288): No heartbeat from core client for 30 sec - exiting 12:18:49 (8288): No heartbeat from core client for 30 sec - exiting 12:18:50 (8288): No heartbeat from core client for 30 sec - exiting 12:18:51 (8288): No heartbeat from core client for 30 sec - exiting 12:18:52 (8288): No heartbeat from core client for 30 sec - exiting 12:18:53 (8288): No heartbeat from core client for 30 sec - exiting 12:18:54 (8288): No heartbeat from core client for 30 sec - exiting 12:18:55 (8288): No heartbeat from core client for 30 sec - exiting 12:18:56 (8288): No heartbeat from core client for 30 sec - exiting 12:20:45 (8288): No heartbeat from core client for 30 sec - exiting 12:20:46 (8288): No heartbeat from core client for 30 sec - exiting 12:20:47 (8288): No heartbeat from core client for 30 sec - exiting 12:20:48 (8288): No heartbeat from core client for 30 sec - exiting 12:20:49 (8288): No heartbeat from core client for 30 sec - exiting 12:20:50 (8288): No heartbeat from core client for 30 sec - exiting 11:30:19 (8288): No heartbeat from core client for 30 sec - exiting 11:30:20 (8288): No heartbeat from core client for 30 sec - exiting 11:30:21 (8288): No heartbeat from core client for 30 sec - exiting 11:30:22 (8288): No heartbeat from core client for 30 sec - exiting 11:30:23 (8288): No heartbeat from core client for 30 sec - exiting 11:30:24 (8288): No heartbeat from core client for 30 sec - exiting 11:30:25 (8288): No heartbeat from core client for 30 sec - exiting 11:30:26 (8288): No heartbeat from core client for 30 sec - exiting 11:30:27 (8288): No heartbeat from core client for 30 sec - exiting 11:30:28 (8288): No heartbeat from core client for 30 sec - exiting 11:30:29 (8288): No heartbeat from core client for 30 sec - exiting 11:31:03 (8288): No heartbeat from core client for 30 sec - exiting 11:31:04 (8288): No heartbeat from core client for 30 sec - exiting 11:31:05 (8288): No heartbeat from core client for 30 sec - exiting 11:31:06 (8288): No heartbeat from core client for 30 sec - exiting 11:31:07 (8288): No heartbeat from core client for 30 sec - exiting 11:31:08 (8288): No heartbeat from core client for 30 sec - exiting 11:31:09 (8288): No heartbeat from core client for 30 sec - exiting 11:31:10 (8288): No heartbeat from core client for 30 sec - exiting 11:31:11 (8288): No heartbeat from core client for 30 sec - exiting 11:31:12 (8288): No heartbeat from core client for 30 sec - exiting 11:31:13 (8288): No heartbeat from core client for 30 sec - exiting 11:32:14 (8288): No heartbeat from core client for 30 sec - exiting 11:32:15 (8288): No heartbeat from core client for 30 sec - exiting 11:32:16 (8288): No heartbeat from core client for 30 sec - exiting 11:32:17 (8288): No heartbeat from core client for 30 sec - exiting 11:32:18 (8288): No heartbeat from core client for 30 sec - exiting 11:32:19 (8288): No heartbeat from core client for 30 sec - exiting 11:32:20 (8288): No heartbeat from core client for 30 sec - exiting 11:32:21 (8288): No heartbeat from core client for 30 sec - exiting 11:32:22 (8288): No heartbeat from core client for 30 sec - exiting 11:32:23 (8288): No heartbeat from core client for 30 sec - exiting 11:32:24 (8288): No heartbeat from core client for 30 sec - exiting 11:34:50 (8288): No heartbeat from core client for 30 sec - exiting 11:34:51 (8288): No heartbeat from core client for 30 sec - exiting 11:34:52 (8288): No heartbeat from core client for 30 sec - exiting 11:34:53 (8288): No heartbeat from core client for 30 sec - exiting 11:34:54 (8288): No heartbeat from core client for 30 sec - exiting 11:34:55 (8288): No heartbeat from core client for 30 sec - exiting 11:34:56 (8288): No heartbeat from core client for 30 sec - exiting 11:34:57 (8288): No heartbeat from core client for 30 sec - exiting 11:34:58 (8288): No heartbeat from core client for 30 sec - exiting 11:34:59 (8288): No heartbeat from core client for 30 sec - exiting 11:35:31 (8288): No heartbeat from core client for 30 sec - exiting 11:35:32 (8288): No heartbeat from core client for 30 sec - exiting 11:35:33 (8288): No heartbeat from core client for 30 sec - exiting 11:35:34 (8288): No heartbeat from core client for 30 sec - exiting 11:35:35 (8288): No heartbeat from core client for 30 sec - exiting 11:35:36 (8288): No heartbeat from core client for 30 sec - exiting 11:35:37 (8288): No heartbeat from core client for 30 sec - exiting 11:35:38 (8288): No heartbeat from core client for 30 sec - exiting 11:35:39 (8288): No heartbeat from core client for 30 sec - exiting 11:37:41 (8288): No heartbeat from core client for 30 sec - exiting 11:37:42 (8288): No heartbeat from core client for 30 sec - exiting 11:37:43 (8288): No heartbeat from core client for 30 sec - exiting 11:37:44 (8288): No heartbeat from core client for 30 sec - exiting 11:37:45 (8288): No heartbeat from core client for 30 sec - exiting 11:37:46 (8288): No heartbeat from core client for 30 sec - exiting 11:37:47 (8288): No heartbeat from core client for 30 sec - exiting 11:37:48 (8288): No heartbeat from core client for 30 sec - exiting 11:37:49 (8288): No heartbeat from core client for 30 sec - exiting 11:37:50 (8288): No heartbeat from core client for 30 sec - exiting 11:37:51 (8288): No heartbeat from core client for 30 sec - exiting 11:39:49 (8288): No heartbeat from core client for 30 sec - exiting 11:39:50 (8288): No heartbeat from core client for 30 sec - exiting 11:39:51 (8288): No heartbeat from core client for 30 sec - exiting 11:39:52 (8288): No heartbeat from core client for 30 sec - exiting 11:39:53 (8288): No heartbeat from core client for 30 sec - exiting 11:39:54 (8288): No heartbeat from core client for 30 sec - exiting 11:39:55 (8288): No heartbeat from core client for 30 sec - exiting 11:39:56 (8288): No heartbeat from core client for 30 sec - exiting 11:39:57 (8288): No heartbeat from core client for 30 sec - exiting 11:39:58 (8288): No heartbeat from core client for 30 sec - exiting 11:39:59 (8288): No heartbeat from core client for 30 sec - exiting 11:41:55 (8288): No heartbeat from core client for 30 sec - exiting 11:41:56 (8288): No heartbeat from core client for 30 sec - exiting 11:41:57 (8288): No heartbeat from core client for 30 sec - exiting 11:41:58 (8288): No heartbeat from core client for 30 sec - exiting 11:41:59 (8288): No heartbeat from core client for 30 sec - exiting 11:42:00 (8288): No heartbeat from core client for 30 sec - exiting 11:42:01 (8288): No heartbeat from core client for 30 sec - exiting 11:42:02 (8288): No heartbeat from core client for 30 sec - exiting 11:42:03 (8288): No heartbeat from core client for 30 sec - exiting 11:42:04 (8288): No heartbeat from core client for 30 sec - exiting 11:42:05 (8288): No heartbeat from core client for 30 sec - exiting 11:45:51 (8288): No heartbeat from core client for 30 sec - exiting 11:45:52 (8288): No heartbeat from core client for 30 sec - exiting 11:45:53 (8288): No heartbeat from core client for 30 sec - exiting 11:45:54 (8288): No heartbeat from core client for 30 sec - exiting 11:45:55 (8288): No heartbeat from core client for 30 sec - exiting 11:45:56 (8288): No heartbeat from core client for 30 sec - exiting 11:45:57 (8288): No heartbeat from core client for 30 sec - exiting 11:45:58 (8288): No heartbeat from core client for 30 sec - exiting 11:45:59 (8288): No heartbeat from core client for 30 sec - exiting 11:46:00 (8288): No heartbeat from core client for 30 sec - exiting 11:46:01 (8288): No heartbeat from core client for 30 sec - exiting 11:49:23 (8288): No heartbeat from core client for 30 sec - exiting 11:49:24 (8288): No heartbeat from core client for 30 sec - exiting 11:49:25 (8288): No heartbeat from core client for 30 sec - exiting 11:49:26 (8288): No heartbeat from core client for 30 sec - exiting 11:49:27 (8288): No heartbeat from core client for 30 sec - exiting 11:49:28 (8288): No heartbeat from core client for 30 sec - exiting 11:49:29 (8288): No heartbeat from core client for 30 sec - exiting 11:49:30 (8288): No heartbeat from core client for 30 sec - exiting 11:49:31 (8288): No heartbeat from core client for 30 sec - exiting 11:49:32 (8288): No heartbeat from core client for 30 sec - exiting 11:50:51 (8288): No heartbeat from core client for 30 sec - exiting 11:50:52 (8288): No heartbeat from core client for 30 sec - exiting 11:50:53 (8288): No heartbeat from core client for 30 sec - exiting 11:50:54 (8288): No heartbeat from core client for 30 sec - exiting 11:50:55 (8288): No heartbeat from core client for 30 sec - exiting 11:50:56 (8288): No heartbeat from core client for 30 sec - exiting 11:50:57 (8288): No heartbeat from core client for 30 sec - exiting 11:50:58 (8288): No heartbeat from core client for 30 sec - exiting 11:50:59 (8288): No heartbeat from core client for 30 sec - exiting 11:51:00 (8288): No heartbeat from core client for 30 sec - exiting 11:53:37 (8288): No heartbeat from core client for 30 sec - exiting 11:53:38 (8288): No heartbeat from core client for 30 sec - exiting 11:53:39 (8288): No heartbeat from core client for 30 sec - exiting 11:53:40 (8288): No heartbeat from core client for 30 sec - exiting 11:53:41 (8288): No heartbeat from core client for 30 sec - exiting 11:53:42 (8288): No heartbeat from core client for 30 sec - exiting 11:53:43 (8288): No heartbeat from core client for 30 sec - exiting 11:53:44 (8288): No heartbeat from core client for 30 sec - exiting 11:53:45 (8288): No heartbeat from core client for 30 sec - exiting 11:53:46 (8288): No heartbeat from core client for 30 sec - exiting 11:53:47 (8288): No heartbeat from core client for 30 sec - exiting 11:57:29 (8288): No heartbeat from core client for 30 sec - exiting 11:57:30 (8288): No heartbeat from core client for 30 sec - exiting 11:57:31 (8288): No heartbeat from core client for 30 sec - exiting 11:57:32 (8288): No heartbeat from core client for 30 sec - exiting 11:57:33 (8288): No heartbeat from core client for 30 sec - exiting 11:57:34 (8288): No heartbeat from core client for 30 sec - exiting 11:57:35 (8288): No heartbeat from core client for 30 sec - exiting 11:57:36 (8288): No heartbeat from core client for 30 sec - exiting 11:57:37 (8288): No heartbeat from core client for 30 sec - exiting 11:57:38 (8288): No heartbeat from core client for 30 sec - exiting 12:01:15 (8288): No heartbeat from core client for 30 sec - exiting 12:01:16 (8288): No heartbeat from core client for 30 sec - exiting 12:01:17 (8288): No heartbeat from core client for 30 sec - exiting 12:01:18 (8288): No heartbeat from core client for 30 sec - exiting 12:01:19 (8288): No heartbeat from core client for 30 sec - exiting 12:01:20 (8288): No heartbeat from core client for 30 sec - exiting 12:01:21 (8288): No heartbeat from core client for 30 sec - exiting 12:01:22 (8288): No heartbeat from core client for 30 sec - exiting 12:01:23 (8288): No heartbeat from core client for 30 sec - exiting 12:01:24 (8288): No heartbeat from core client for 30 sec - exiting 12:05:41 (8288): No heartbeat from core client for 30 sec - exiting 12:05:42 (8288): No heartbeat from core client for 30 sec - exiting 12:05:43 (8288): No heartbeat from core client for 30 sec - exiting 12:05:44 (8288): No heartbeat from core client for 30 sec - exiting 12:05:45 (8288): No heartbeat from core client for 30 sec - exiting 12:05:46 (8288): No heartbeat from core client for 30 sec - exiting 12:05:47 (8288): No heartbeat from core client for 30 sec - exiting 12:05:48 (8288): No heartbeat from core client for 30 sec - exiting 12:05:49 (8288): No heartbeat from core client for 30 sec - exiting 12:05:50 (8288): No heartbeat from core client for 30 sec - exiting 12:05:51 (8288): No heartbeat from core client for 30 sec - exiting 12:06:56 (8288): No heartbeat from core client for 30 sec - exiting 12:06:57 (8288): No heartbeat from core client for 30 sec - exiting 12:06:58 (8288): No heartbeat from core client for 30 sec - exiting 12:06:59 (8288): No heartbeat from core client for 30 sec - exiting 12:07:00 (8288): No heartbeat from core client for 30 sec - exiting 12:07:01 (8288): No heartbeat from core client for 30 sec - exiting 12:07:02 (8288): No heartbeat from core client for 30 sec - exiting 12:07:03 (8288): No heartbeat from core client for 30 sec - exiting 12:07:04 (8288): No heartbeat from core client for 30 sec - exiting 12:07:05 (8288): No heartbeat from core client for 30 sec - exiting 12:07:06 (8288): No heartbeat from core client for 30 sec - exiting 12:08:37 (8288): No heartbeat from core client for 30 sec - exiting 12:08:38 (8288): No heartbeat from core client for 30 sec - exiting 12:08:39 (8288): No heartbeat from core client for 30 sec - exiting 12:08:40 (8288): No heartbeat from core client for 30 sec - exiting 12:08:41 (8288): No heartbeat from core client for 30 sec - exiting 12:08:42 (8288): No heartbeat from core client for 30 sec - exiting 12:08:43 (8288): No heartbeat from core client for 30 sec - exiting 12:08:44 (8288): No heartbeat from core client for 30 sec - exiting 12:08:45 (8288): No heartbeat from core client for 30 sec - exiting 12:08:46 (8288): No heartbeat from core client for 30 sec - exiting 12:08:47 (8288): No heartbeat from core client for 30 sec - exiting 12:12:28 (8288): No heartbeat from core client for 30 sec - exiting 12:12:29 (8288): No heartbeat from core client for 30 sec - exiting 12:12:30 (8288): No heartbeat from core client for 30 sec - exiting 12:12:31 (8288): No heartbeat from core client for 30 sec - exiting 12:12:32 (8288): No heartbeat from core client for 30 sec - exiting 12:12:33 (8288): No heartbeat from core client for 30 sec - exiting 12:12:34 (8288): No heartbeat from core client for 30 sec - exiting 12:12:35 (8288): No heartbeat from core client for 30 sec - exiting 12:12:36 (8288): No heartbeat from core client for 30 sec - exiting 12:12:37 (8288): No heartbeat from core client for 30 sec - exiting 12:12:38 (8288): No heartbeat from core client for 30 sec - exiting 12:15:55 (8288): No heartbeat from core client for 30 sec - exiting 12:15:56 (8288): No heartbeat from core client for 30 sec - exiting 12:15:57 (8288): No heartbeat from core client for 30 sec - exiting 12:15:58 (8288): No heartbeat from core client for 30 sec - exiting 12:15:59 (8288): No heartbeat from core client for 30 sec - exiting 12:16:00 (8288): No heartbeat from core client for 30 sec - exiting 12:16:01 (8288): No heartbeat from core client for 30 sec - exiting 12:16:02 (8288): No heartbeat from core client for 30 sec - exiting 12:16:03 (8288): No heartbeat from core client for 30 sec - exiting 12:16:04 (8288): No heartbeat from core client for 30 sec - exiting 12:17:22 (8288): No heartbeat from core client for 30 sec - exiting 12:17:23 (8288): No heartbeat from core client for 30 sec - exiting 12:17:24 (8288): No heartbeat from core client for 30 sec - exiting 12:17:25 (8288): No heartbeat from core client for 30 sec - exiting 12:17:26 (8288): No heartbeat from core client for 30 sec - exiting 12:17:27 (8288): No heartbeat from core client for 30 sec - exiting 12:17:28 (8288): No heartbeat from core client for 30 sec - exiting 12:17:29 (8288): No heartbeat from core client for 30 sec - exiting 12:17:30 (8288): No heartbeat from core client for 30 sec - exiting 12:17:31 (8288): No heartbeat from core client for 30 sec - exiting 12:17:32 (8288): No heartbeat from core client for 30 sec - exiting 12:19:43 (8288): No heartbeat from core client for 30 sec - exiting 12:19:44 (8288): No heartbeat from core client for 30 sec - exiting 12:19:45 (8288): No heartbeat from core client for 30 sec - exiting 12:19:46 (8288): No heartbeat from core client for 30 sec - exiting 12:19:47 (8288): No heartbeat from core client for 30 sec - exiting 12:19:48 (8288): No heartbeat from core client for 30 sec - exiting 12:19:49 (8288): No heartbeat from core client for 30 sec - exiting 12:19:50 (8288): No heartbeat from core client for 30 sec - exiting 12:19:51 (8288): No heartbeat from core client for 30 sec - exiting 12:19:52 (8288): No heartbeat from core client for 30 sec - exiting 12:19:53 (8288): No heartbeat from core client for 30 sec - exiting 12:24:24 (8288): No heartbeat from core client for 30 sec - exiting 12:24:25 (8288): No heartbeat from core client for 30 sec - exiting 12:24:26 (8288): No heartbeat from core client for 30 sec - exiting 12:24:27 (8288): No heartbeat from core client for 30 sec - exiting 12:24:28 (8288): No heartbeat from core client for 30 sec - exiting 12:24:29 (8288): No heartbeat from core client for 30 sec - exiting 12:24:30 (8288): No heartbeat from core client for 30 sec - exiting 12:24:31 (8288): No heartbeat from core client for 30 sec - exiting 12:24:32 (8288): No heartbeat from core client for 30 sec - exiting 12:24:33 (8288): No heartbeat from core client for 30 sec - exiting 12:24:34 (8288): No heartbeat from core client for 30 sec - exiting 12:25:51 (8288): No heartbeat from core client for 30 sec - exiting 12:25:52 (8288): No heartbeat from core client for 30 sec - exiting 12:25:53 (8288): No heartbeat from core client for 30 sec - exiting 12:25:54 (8288): No heartbeat from core client for 30 sec - exiting 12:25:55 (8288): No heartbeat from core client for 30 sec - exiting 12:25:57 (8288): No heartbeat from core client for 30 sec - exiting 12:25:58 (8288): No heartbeat from core client for 30 sec - exiting 12:25:59 (8288): No heartbeat from core client for 30 sec - exiting 12:26:00 (8288): No heartbeat from core client for 30 sec - exiting 12:26:01 (8288): No heartbeat from core client for 30 sec - exiting 12:26:02 (8288): No heartbeat from core client for 30 sec - exiting 12:27:54 (8288): No heartbeat from core client for 30 sec - exiting 12:27:55 (8288): No heartbeat from core client for 30 sec - exiting 12:27:56 (8288): No heartbeat from core client for 30 sec - exiting 12:27:57 (8288): No heartbeat from core client for 30 sec - exiting 12:27:58 (8288): No heartbeat from core client for 30 sec - exiting 12:27:59 (8288): No heartbeat from core client for 30 sec - exiting 12:28:00 (8288): No heartbeat from core client for 30 sec - exiting 12:28:01 (8288): No heartbeat from core client for 30 sec - exiting 12:28:02 (8288): No heartbeat from core client for 30 sec - exiting 12:28:03 (8288): No heartbeat from core client for 30 sec - exiting 12:28:04 (8288): No heartbeat from core client for 30 sec - exiting 12:28:46 (8288): No heartbeat from core client for 30 sec - exiting 12:28:47 (8288): No heartbeat from core client for 30 sec - exiting 12:28:48 (8288): No heartbeat from core client for 30 sec - exiting 12:28:49 (8288): No heartbeat from core client for 30 sec - exiting 12:28:50 (8288): No heartbeat from core client for 30 sec - exiting 12:28:51 (8288): No heartbeat from core client for 30 sec - exiting 12:28:52 (8288): No heartbeat from core client for 30 sec - exiting 12:28:53 (8288): No heartbeat from core client for 30 sec - exiting 12:28:54 (8288): No heartbeat from core client for 30 sec - exiting 12:28:55 (8288): No heartbeat from core client for 30 sec - exiting 12:30:18 (8288): No heartbeat from core client for 30 sec - exiting 12:30:19 (8288): No heartbeat from core client for 30 sec - exiting 12:30:20 (8288): No heartbeat from core client for 30 sec - exiting 12:30:21 (8288): No heartbeat from core client for 30 sec - exiting 12:30:22 (8288): No heartbeat from core client for 30 sec - exiting 12:30:23 (8288): No heartbeat from core client for 30 sec - exiting 12:30:24 (8288): No heartbeat from core client for 30 sec - exiting 12:30:25 (8288): No heartbeat from core client for 30 sec - exiting 12:30:26 (8288): No heartbeat from core client for 30 sec - exiting 12:30:27 (8288): No heartbeat from core client for 30 sec - exiting 12:30:28 (8288): No heartbeat from core client for 30 sec - exiting 18:40:34 (8288): No heartbeat from core client for 30 sec - exiting 18:40:35 (8288): No heartbeat from core client for 30 sec - exiting 18:40:36 (8288): No heartbeat from core client for 30 sec - exiting 18:40:37 (8288): No heartbeat from core client for 30 sec - exiting </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Jan 2013 10:33:24	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	803,520	1,511,077	1.8806
29 Jan 2013 12:35:06	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	777,600	1,455,637	1.8720
28 Jan 2013 23:02:49	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	751,680	1,408,325	1.8736
28 Jan 2013 09:34:38	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	725,760	1,361,293	1.8757
27 Jan 2013 20:10:17	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	699,840	1,314,491	1.8783
31 Dec 2012 22:18:57	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	673,920	1,266,714	1.8796
31 Dec 2012 08:33:23	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	648,000	1,218,632	1.8806
30 Dec 2012 18:51:23	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	622,080	1,170,770	1.8820
30 Dec 2012 05:03:31	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	596,160	1,122,618	1.8831
29 Dec 2012 15:22:09	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	570,240	1,074,640	1.8845
29 Dec 2012 01:28:33	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	544,320	1,026,455	1.8858
28 Dec 2012 11:40:08	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	518,400	978,278	1.8871
27 Dec 2012 21:56:42	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	492,480	930,269	1.8889
27 Dec 2012 08:09:15	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	466,560	882,044	1.8905
26 Dec 2012 18:18:12	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	440,640	833,720	1.8921
26 Dec 2012 04:33:07	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	414,720	785,659	1.8944
25 Dec 2012 14:45:48	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	388,800	737,375	1.8965
25 Dec 2012 01:00:50	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	362,880	689,266	1.8994
24 Dec 2012 11:08:26	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	336,960	641,039	1.9024
23 Dec 2012 20:14:42	1239754	15480628	hadcm3n_zl1m_1920_40_008256016_0	311,040	588,783	1.8929