Task 13352310

Name	hadcm3n_t68k_1940_40_007444010_2
Workunit	7641513
Created	9 Sep 2011, 8:47:15 UTC
Sent	9 Sep 2011, 8:50:51 UTC
Report deadline	9 Dec 2011, 16:18:02 UTC
Received	29 Sep 2011, 21:25:19 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1168571
Run time	7 days 13 hours 6 min 29 sec
CPU time	6 days 18 hours 52 min 55 sec
Validate state	Invalid
Credit	4,354.56
Device peak FLOPS	2.35 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4648, iMonCtr=1 Model crash detected, will try to restart... 05:30:25 (5716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:22:20 (4216): No heartbeat from core client for 30 sec - exiting 23:22:22 (4216): No heartbeat from core client for 30 sec - exiting 23:22:23 (4216): No heartbeat from core client for 30 sec - exiting 23:22:24 (4216): No heartbeat from core client for 30 sec - exiting 23:22:25 (4216): No heartbeat from core client for 30 sec - exiting 23:22:26 (4216): No heartbeat from core client for 30 sec - exiting 23:22:27 (4216): No heartbeat from core client for 30 sec - exiting 23:22:28 (4216): No heartbeat from core client for 30 sec - exiting 23:22:29 (4216): No heartbeat from core client for 30 sec - exiting 23:22:30 (4216): No heartbeat from core client for 30 sec - exiting 23:22:31 (4216): No heartbeat from core client for 30 sec - exiting 23:22:32 (4216): No heartbeat from core client for 30 sec - exiting 23:22:33 (4216): No heartbeat from core client for 30 sec - exiting 23:22:34 (4216): No heartbeat from core client for 30 sec - exiting 23:22:35 (4216): No heartbeat from core client for 30 sec - exiting 23:22:36 (4216): No heartbeat from core client for 30 sec - exiting 23:22:37 (4216): No heartbeat from core client for 30 sec - exiting 23:22:38 (4216): No heartbeat from core client for 30 sec - exiting 23:22:39 (4216): No heartbeat from core client for 30 sec - exiting 23:22:40 (4216): No heartbeat from core client for 30 sec - exiting 23:22:41 (4216): No heartbeat from core client for 30 sec - exiting 23:22:42 (4216): No heartbeat from core client for 30 sec - exiting 23:22:43 (4216): No heartbeat from core client for 30 sec - exiting 23:22:44 (4216): No heartbeat from core client for 30 sec - exiting 23:22:45 (4216): No heartbeat from core client for 30 sec - exiting 23:22:46 (4216): No heartbeat from core client for 30 sec - exiting 23:22:47 (4216): No heartbeat from core client for 30 sec - exiting 23:22:48 (4216): No heartbeat from core client for 30 sec - exiting 23:22:49 (4216): No heartbeat from core client for 30 sec - exiting 23:22:50 (4216): No heartbeat from core client for 30 sec - exiting 23:22:51 (4216): No heartbeat from core client for 30 sec - exiting 23:22:52 (4216): No heartbeat from core client for 30 sec - exiting 23:22:53 (4216): No heartbeat from core client for 30 sec - exiting 23:22:54 (4216): No heartbeat from core client for 30 sec - exiting 23:22:55 (4216): No heartbeat from core client for 30 sec - exiting 23:22:56 (4216): No heartbeat from core client for 30 sec - exiting 23:22:57 (4216): No heartbeat from core client for 30 sec - exiting 23:22:58 (4216): No heartbeat from core client for 30 sec - exiting 23:22:59 (4216): No heartbeat from core client for 30 sec - exiting 23:23:00 (4216): No heartbeat from core client for 30 sec - exiting 23:23:01 (4216): No heartbeat from core client for 30 sec - exiting 23:23:02 (4216): No heartbeat from core client for 30 sec - exiting 23:23:03 (4216): No heartbeat from core client for 30 sec - exiting 23:23:04 (4216): No heartbeat from core client for 30 sec - exiting 23:23:05 (4216): No heartbeat from core client for 30 sec - exiting 23:23:06 (4216): No heartbeat from core client for 30 sec - exiting 23:23:07 (4216): No heartbeat from core client for 30 sec - exiting 23:23:08 (4216): No heartbeat from core client for 30 sec - exiting 23:23:09 (4216): No heartbeat from core client for 30 sec - exiting 23:23:10 (4216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:24:04 (5924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7396, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:16:43 (5128): No heartbeat from core client for 30 sec - exiting 16:16:45 (5128): No heartbeat from core client for 30 sec - exiting 16:16:46 (5128): No heartbeat from core client for 30 sec - exiting 16:16:47 (5128): No heartbeat from core client for 30 sec - exiting 16:16:48 (5128): No heartbeat from core client for 30 sec - exiting 16:16:49 (5128): No heartbeat from core client for 30 sec - exiting 16:16:50 (5128): No heartbeat from core client for 30 sec - exiting 16:16:51 (5128): No heartbeat from core client for 30 sec - exiting 16:16:52 (5128): No heartbeat from core client for 30 sec - exiting 16:16:53 (5128): No heartbeat from core client for 30 sec - exiting 16:16:54 (5128): No heartbeat from core client for 30 sec - exiting 16:16:55 (5128): No heartbeat from core client for 30 sec - exiting 16:16:56 (5128): No heartbeat from core client for 30 sec - exiting 16:16:57 (5128): No heartbeat from core client for 30 sec - exiting 16:16:58 (5128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6988, iMonCtr=1 Model crash detected, will try to restart... 09:32:40 (4936): No heartbeat from core client for 30 sec - exiting 09:32:42 (4936): No heartbeat from core client for 30 sec - exiting 09:32:43 (4936): No heartbeat from core client for 30 sec - exiting 09:32:44 (4936): No heartbeat from core client for 30 sec - exiting 09:32:45 (4936): No heartbeat from core client for 30 sec - exiting 09:32:46 (4936): No heartbeat from core client for 30 sec - exiting 09:32:47 (4936): No heartbeat from core client for 30 sec - exiting 09:32:48 (4936): No heartbeat from core client for 30 sec - exiting 09:32:49 (4936): No heartbeat from core client for 30 sec - exiting 09:32:50 (4936): No heartbeat from core client for 30 sec - exiting 09:32:51 (4936): No heartbeat from core client for 30 sec - exiting 09:32:52 (4936): No heartbeat from core client for 30 sec - exiting 09:32:53 (4936): No heartbeat from core client for 30 sec - exiting 09:32:54 (4936): No heartbeat from core client for 30 sec - exiting 09:32:55 (4936): No heartbeat from core client for 30 sec - exiting 09:32:56 (4936): No heartbeat from core client for 30 sec - exiting 09:32:57 (4936): No heartbeat from core client for 30 sec - exiting 09:32:58 (4936): No heartbeat from core client for 30 sec - exiting 09:32:59 (4936): No heartbeat from core client for 30 sec - exiting 09:33:00 (4936): No heartbeat from core client for 30 sec - exiting 09:33:01 (4936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 00:52:12 (5964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:35:39 (5044): No heartbeat from core client for 30 sec - exiting 14:35:40 (5044): No heartbeat from core client for 30 sec - exiting 14:35:41 (5044): No heartbeat from core client for 30 sec - exiting 14:35:42 (5044): No heartbeat from core client for 30 sec - exiting 14:35:43 (5044): No heartbeat from core client for 30 sec - exiting 14:35:44 (5044): No heartbeat from core client for 30 sec - exiting 14:35:45 (5044): No heartbeat from core client for 30 sec - exiting 14:35:46 (5044): No heartbeat from core client for 30 sec - exiting 14:35:47 (5044): No heartbeat from core client for 30 sec - exiting 14:35:48 (5044): No heartbeat from core client for 30 sec - exiting 14:35:49 (5044): No heartbeat from core client for 30 sec - exiting 14:35:50 (5044): No heartbeat from core client for 30 sec - exiting 14:35:51 (5044): No heartbeat from core client for 30 sec - exiting 14:35:52 (5044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:55:10 (4120): No heartbeat from core client for 30 sec - exiting 20:55:11 (4120): No heartbeat from core client for 30 sec - exiting 20:55:12 (4120): No heartbeat from core client for 30 sec - exiting 20:55:13 (4120): No heartbeat from core client for 30 sec - exiting 20:55:14 (4120): No heartbeat from core client for 30 sec - exiting 20:55:15 (4120): No heartbeat from core client for 30 sec - exiting 20:55:16 (4120): No heartbeat from core client for 30 sec - exiting 20:55:17 (4120): No heartbeat from core client for 30 sec - exiting 20:55:18 (4120): No heartbeat from core client for 30 sec - exiting 20:55:19 (4120): No heartbeat from core client for 30 sec - exiting 20:55:20 (4120): No heartbeat from core client for 30 sec - exiting 20:55:21 (4120): No heartbeat from core client for 30 sec - exiting 20:55:22 (4120): No heartbeat from core client for 30 sec - exiting 20:55:23 (4120): No heartbeat from core client for 30 sec - exiting 20:55:24 (4120): No heartbeat from core client for 30 sec - exiting 20:55:25 (4120): No heartbeat from core client for 30 sec - exiting 20:55:26 (4120): No heartbeat from core client for 30 sec - exiting 20:55:27 (4120): No heartbeat from core client for 30 sec - exiting 20:55:28 (4120): No heartbeat from core client for 30 sec - exiting 20:55:29 (4120): No heartbeat from core client for 30 sec - exiting 20:55:30 (4120): No heartbeat from core client for 30 sec - exiting 20:55:31 (4120): No heartbeat from core client for 30 sec - exiting 20:55:32 (4120): No heartbeat from core client for 30 sec - exiting 20:55:33 (4120): No heartbeat from core client for 30 sec - exiting 20:55:34 (4120): No heartbeat from core client for 30 sec - exiting 20:55:35 (4120): No heartbeat from core client for 30 sec - exiting 20:55:36 (4120): No heartbeat from core client for 30 sec - exiting 20:55:37 (4120): No heartbeat from core client for 30 sec - exiting 20:55:38 (4120): No heartbeat from core client for 30 sec - exiting 20:55:39 (4120): No heartbeat from core client for 30 sec - exiting 20:55:40 (4120): No heartbeat from core client for 30 sec - exiting 20:55:41 (4120): No heartbeat from core client for 30 sec - exiting 20:55:42 (4120): No heartbeat from core client for 30 sec - exiting 20:55:43 (4120): No heartbeat from core client for 30 sec - exiting 20:55:44 (4120): No heartbeat from core client for 30 sec - exiting 20:55:45 (4120): No heartbeat from core client for 30 sec - exiting 20:55:46 (4120): No heartbeat from core client for 30 sec - exiting 20:55:47 (4120): No heartbeat from core client for 30 sec - exiting 20:55:48 (4120): No heartbeat from core client for 30 sec - exiting 20:55:49 (4120): No heartbeat from core client for 30 sec - exiting 20:55:50 (4120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:16:31 (6108): No heartbeat from core client for 30 sec - exiting 09:16:32 (6108): No heartbeat from core client for 30 sec - exiting 09:16:33 (6108): No heartbeat from core client for 30 sec - exiting 09:16:34 (6108): No heartbeat from core client for 30 sec - exiting 09:16:35 (6108): No heartbeat from core client for 30 sec - exiting 09:16:36 (6108): No heartbeat from core client for 30 sec - exiting 09:16:37 (6108): No heartbeat from core client for 30 sec - exiting 09:16:38 (6108): No heartbeat from core client for 30 sec - exiting 09:16:39 (6108): No heartbeat from core client for 30 sec - exiting 09:16:40 (6108): No heartbeat from core client for 30 sec - exiting 09:16:41 (6108): No heartbeat from core client for 30 sec - exiting 09:16:42 (6108): No heartbeat from core client for 30 sec - exiting 09:16:43 (6108): No heartbeat from core client for 30 sec - exiting 09:16:44 (6108): No heartbeat from core client for 30 sec - exiting 09:16:45 (6108): No heartbeat from core client for 30 sec - exiting 09:16:46 (6108): No heartbeat from core client for 30 sec - exiting 09:16:47 (6108): No heartbeat from core client for 30 sec - exiting 09:16:48 (6108): No heartbeat from core client for 30 sec - exiting 09:16:49 (6108): No heartbeat from core client for 30 sec - exiting 09:16:50 (6108): No heartbeat from core client for 30 sec - exiting 09:16:51 (6108): No heartbeat from core client for 30 sec - exiting 09:16:52 (6108): No heartbeat from core client for 30 sec - exiting 09:16:53 (6108): No heartbeat from core client for 30 sec - exiting 09:16:54 (6108): No heartbeat from core client for 30 sec - exiting 09:16:55 (6108): No heartbeat from core client for 30 sec - exiting 09:16:56 (6108): No heartbeat from core client for 30 sec - exiting 09:16:57 (6108): No heartbeat from core client for 30 sec - exiting 09:16:58 (6108): No heartbeat from core client for 30 sec - exiting 09:16:59 (6108): No heartbeat from core client for 30 sec - exiting 09:17:00 (6108): No heartbeat from core client for 30 sec - exiting 09:17:01 (6108): No heartbeat from core client for 30 sec - exiting 09:17:02 (6108): No heartbeat from core client for 30 sec - exiting 09:17:03 (6108): No heartbeat from core client for 30 sec - exiting 09:17:04 (6108): No heartbeat from core client for 30 sec - exiting 09:17:05 (6108): No heartbeat from core client for 30 sec - exiting 09:17:06 (6108): No heartbeat from core client for 30 sec - exiting 09:17:07 (6108): No heartbeat from core client for 30 sec - exiting 09:17:08 (6108): No heartbeat from core client for 30 sec - exiting 09:17:09 (6108): No heartbeat from core client for 30 sec - exiting 09:17:10 (6108): No heartbeat from core client for 30 sec - exiting 09:17:11 (6108): No heartbeat from core client for 30 sec - exiting 09:17:12 (6108): No heartbeat from core client for 30 sec - exiting 09:17:13 (6108): No heartbeat from core client for 30 sec - exiting 09:17:14 (6108): No heartbeat from core client for 30 sec - exiting 09:17:15 (6108): No heartbeat from core client for 30 sec - exiting 09:17:16 (6108): No heartbeat from core client for 30 sec - exiting 09:17:17 (6108): No heartbeat from core client for 30 sec - exiting 09:17:18 (6108): No heartbeat from core client for 30 sec - exiting 09:17:19 (6108): No heartbeat from core client for 30 sec - exiting 09:17:20 (6108): No heartbeat from core client for 30 sec - exiting 09:17:21 (6108): No heartbeat from core client for 30 sec - exiting 09:17:22 (6108): No heartbeat from core client for 30 sec - exiting 09:17:23 (6108): No heartbeat from core client for 30 sec - exiting 09:17:24 (6108): No heartbeat from core client for 30 sec - exiting 09:17:25 (6108): No heartbeat from core client for 30 sec - exiting 09:17:26 (6108): No heartbeat from core client for 30 sec - exiting 09:17:27 (6108): No heartbeat from core client for 30 sec - exiting 09:17:28 (6108): No heartbeat from core client for 30 sec - exiting 09:17:29 (6108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:18:11 (3816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Sep 2011 07:21:59	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	362,880	576,697	1.5892
28 Sep 2011 13:03:21	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	336,960	535,633	1.5896
27 Sep 2011 21:35:53	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	311,040	494,696	1.5905
26 Sep 2011 21:30:07	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	285,120	453,563	1.5908
25 Sep 2011 19:45:06	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	259,200	411,788	1.5887
24 Sep 2011 21:20:48	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	233,280	370,008	1.5861
23 Sep 2011 14:35:22	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	207,360	328,545	1.5844
21 Sep 2011 22:05:54	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	181,440	287,903	1.5868
21 Sep 2011 00:41:30	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	155,520	247,108	1.5889
20 Sep 2011 06:57:12	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	129,600	206,504	1.5934
19 Sep 2011 04:10:09	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	103,680	165,528	1.5965
17 Sep 2011 18:29:52	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	77,760	123,767	1.5917
16 Sep 2011 07:04:09	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	51,840	82,623	1.5938
15 Sep 2011 01:17:26	1168571	13352310	hadcm3n_t68k_1940_40_007444010_2	25,920	41,497	1.6010