Task 15656216

Name	hadcm3n_37u5_1940_40_008264193_3
Workunit	8419317
Created	10 Mar 2013, 22:13:54 UTC
Sent	10 Mar 2013, 22:14:20 UTC
Report deadline	10 Jun 2013, 5:41:31 UTC
Received	21 Mar 2013, 18:00:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1273791
Run time	5 days 23 hours 40 min 22 sec
CPU time	5 days 6 hours 7 min 58 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	3.12 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:02:15 (8309): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:05:43 (10941): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:17 (10984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:18 (10984): No heartbeat from core client for 30 sec - exiting 22:10:19 (10984): No heartbeat from core client for 30 sec - exiting 22:10:20 (10984): No heartbeat from core client for 30 sec - exiting 22:10:21 (10984): No heartbeat from core client for 30 sec - exiting 22:10:22 (10984): No heartbeat from core client for 30 sec - exiting 22:10:23 (10984): No heartbeat from core client for 30 sec - exiting 22:10:24 (10984): No heartbeat from core client for 30 sec - exiting 22:10:25 (10984): No heartbeat from core client for 30 sec - exiting 22:10:26 (10984): No heartbeat from core client for 30 sec - exiting 22:10:27 (10984): No heartbeat from core client for 30 sec - exiting 22:10:28 (10984): No heartbeat from core client for 30 sec - exiting 22:10:29 (10984): No heartbeat from core client for 30 sec - exiting 22:10:30 (10984): No heartbeat from core client for 30 sec - exiting 22:15:04 (11030): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:15:06 (11030): No heartbeat from core client for 30 sec - exiting 22:15:07 (11030): No heartbeat from core client for 30 sec - exiting 22:15:08 (11030): No heartbeat from core client for 30 sec - exiting 22:15:09 (11030): No heartbeat from core client for 30 sec - exiting 22:15:10 (11030): No heartbeat from core client for 30 sec - exiting 22:15:11 (11030): No heartbeat from core client for 30 sec - exiting 22:15:12 (11030): No heartbeat from core client for 30 sec - exiting 22:15:13 (11030): No heartbeat from core client for 30 sec - exiting 22:15:14 (11030): No heartbeat from core client for 30 sec - exiting 22:15:15 (11030): No heartbeat from core client for 30 sec - exiting 22:15:16 (11030): No heartbeat from core client for 30 sec - exiting 22:15:17 (11030): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:29:28 (11087): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:29 (11087): No heartbeat from core client for 30 sec - exiting 22:29:30 (11087): No heartbeat from core client for 30 sec - exiting 22:29:31 (11087): No heartbeat from core client for 30 sec - exiting 22:29:32 (11087): No heartbeat from core client for 30 sec - exiting 22:29:33 (11087): No heartbeat from core client for 30 sec - exiting 22:29:34 (11087): No heartbeat from core client for 30 sec - exiting 22:29:35 (11087): No heartbeat from core client for 30 sec - exiting 22:29:36 (11087): No heartbeat from core client for 30 sec - exiting 22:29:37 (11087): No heartbeat from core client for 30 sec - exiting 22:29:38 (11087): No heartbeat from core client for 30 sec - exiting 22:29:39 (11087): No heartbeat from core client for 30 sec - exiting 22:29:40 (11087): No heartbeat from core client for 30 sec - exiting 22:29:41 (11087): No heartbeat from core client for 30 sec - exiting 22:29:42 (11087): No heartbeat from core client for 30 sec - exiting 22:29:43 (11087): No heartbeat from core client for 30 sec - exiting 22:29:44 (11087): No heartbeat from core client for 30 sec - exiting 22:29:45 (11087): No heartbeat from core client for 30 sec - exiting 22:29:46 (11087): No heartbeat from core client for 30 sec - exiting 22:29:47 (11087): No heartbeat from core client for 30 sec - exiting 22:29:48 (11087): No heartbeat from core client for 30 sec - exiting 22:29:49 (11087): No heartbeat from core client for 30 sec - exiting 22:29:50 (11087): No heartbeat from core client for 30 sec - exiting 22:33:14 (11223): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:33:15 (11223): No heartbeat from core client for 30 sec - exiting 22:33:16 (11223): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 22:42:35 (11271): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:42:37 (11271): No heartbeat from core client for 30 sec - exiting 22:42:38 (11271): No heartbeat from core client for 30 sec - exiting 22:42:39 (11271): No heartbeat from core client for 30 sec - exiting 22:42:40 (11271): No heartbeat from core client for 30 sec - exiting 22:42:41 (11271): No heartbeat from core client for 30 sec - exiting 22:42:42 (11271): No heartbeat from core client for 30 sec - exiting 22:42:43 (11271): No heartbeat from core client for 30 sec - exiting 22:42:44 (11271): No heartbeat from core client for 30 sec - exiting 22:42:45 (11271): No heartbeat from core client for 30 sec - exiting 22:42:46 (11271): No heartbeat from core client for 30 sec - exiting 22:42:47 (11271): No heartbeat from core client for 30 sec - exiting 22:42:48 (11271): No heartbeat from core client for 30 sec - exiting 22:42:49 (11271): No heartbeat from core client for 30 sec - exiting 22:42:50 (11271): No heartbeat from core client for 30 sec - exiting 22:42:51 (11271): No heartbeat from core client for 30 sec - exiting 22:42:52 (11271): No heartbeat from core client for 30 sec - exiting 22:42:53 (11271): No heartbeat from core client for 30 sec - exiting 22:42:54 (11271): No heartbeat from core client for 30 sec - exiting 22:42:55 (11271): No heartbeat from core client for 30 sec - exiting 22:43:46 (11271): No heartbeat from core client for 30 sec - exiting 22:43:47 (11271): No heartbeat from core client for 30 sec - exiting 22:43:48 (11271): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:42:07 (13442): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:42:08 (13442): No heartbeat from core client for 30 sec - exiting 12:42:09 (13442): No heartbeat from core client for 30 sec - exiting 12:42:10 (13442): No heartbeat from core client for 30 sec - exiting 12:42:11 (13442): No heartbeat from core client for 30 sec - exiting 12:42:12 (13442): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:23:49 (7057): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:23:51 (7057): No heartbeat from core client for 30 sec - exiting 23:23:52 (7057): No heartbeat from core client for 30 sec - exiting 23:23:53 (7057): No heartbeat from core client for 30 sec - exiting 23:23:54 (7057): No heartbeat from core client for 30 sec - exiting 23:27:05 (7635): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:27:06 (7635): No heartbeat from core client for 30 sec - exiting 23:27:07 (7635): No heartbeat from core client for 30 sec - exiting 23:27:08 (7635): No heartbeat from core client for 30 sec - exiting 23:27:09 (7635): No heartbeat from core client for 30 sec - exiting 23:27:10 (7635): No heartbeat from core client for 30 sec - exiting 23:27:11 (7635): No heartbeat from core client for 30 sec - exiting 23:27:12 (7635): No heartbeat from core client for 30 sec - exiting 23:27:13 (7635): No heartbeat from core client for 30 sec - exiting 23:27:14 (7635): No heartbeat from core client for 30 sec - exiting 23:27:15 (7635): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:02:21 (15955): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:02:22 (15955): No heartbeat from core client for 30 sec - exiting 15:02:23 (15955): No heartbeat from core client for 30 sec - exiting 15:02:24 (15955): No heartbeat from core client for 30 sec - exiting 15:02:55 (15955): No heartbeat from core client for 30 sec - exiting 15:02:56 (15955): No heartbeat from core client for 30 sec - exiting 15:02:57 (15955): No heartbeat from core client for 30 sec - exiting 15:02:58 (15955): No heartbeat from core client for 30 sec - exiting 15:02:59 (15955): No heartbeat from core client for 30 sec - exiting 15:03:00 (15955): No heartbeat from core client for 30 sec - exiting 15:03:01 (15955): No heartbeat from core client for 30 sec - exiting 15:03:02 (15955): No heartbeat from core client for 30 sec - exiting 15:05:56 (18958): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:05:57 (18958): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:23:57 (18989): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:23:59 (18989): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:47:33 (40985): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:34 (40985): No heartbeat from core client for 30 sec - exiting 08:47:35 (40985): No heartbeat from core client for 30 sec - exiting 08:47:36 (40985): No heartbeat from core client for 30 sec - exiting 08:47:37 (40985): No heartbeat from core client for 30 sec - exiting 08:47:38 (40985): No heartbeat from core client for 30 sec - exiting 08:47:39 (40985): No heartbeat from core client for 30 sec - exiting 08:47:40 (40985): No heartbeat from core client for 30 sec - exiting 08:47:41 (40985): No heartbeat from core client for 30 sec - exiting 08:47:42 (40985): No heartbeat from core client for 30 sec - exiting 08:47:43 (40985): No heartbeat from core client for 30 sec - exiting 08:47:44 (40985): No heartbeat from core client for 30 sec - exiting 08:47:45 (40985): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:07:54 (41572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:07:55 (41572): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Mar 2013 16:05:44	1273791	15656216	hadcm3n_37u5_1940_40_008264193_3	285,120	439,382	1.5410
19 Mar 2013 16:48:10	1273791	15656216	hadcm3n_37u5_1940_40_008264193_3	259,200	398,131	1.5360
18 Mar 2013 00:56:38	1273791	15656216	hadcm3n_37u5_1940_40_008264193_3	233,280	356,807	1.5295
17 Mar 2013 11:59:41	1273791	15656216	hadcm3n_37u5_1940_40_008264193_3	207,360	315,855	1.5232
16 Mar 2013 05:27:08	1273791	15656216	hadcm3n_37u5_1940_40_008264193_3	181,440	274,727	1.5141
15 Mar 2013 05:47:40	1273791	15656216	hadcm3n_37u5_1940_40_008264193_3	155,520	233,612	1.5021
14 Mar 2013 15:40:39	1273791	15656216	hadcm3n_37u5_1940_40_008264193_3	129,600	192,541	1.4857
13 Mar 2013 17:26:20	1273791	15656216	hadcm3n_37u5_1940_40_008264193_3	103,680	151,203	1.4584
12 Mar 2013 19:58:25	1273791	15656216	hadcm3n_37u5_1940_40_008264193_3	77,760	109,937	1.4138
12 Mar 2013 04:30:21	1273791	15656216	hadcm3n_37u5_1940_40_008264193_3	51,840	68,810	1.3274
11 Mar 2013 13:35:18	1273791	15656216	hadcm3n_37u5_1940_40_008264193_3	25,920	27,726	1.0697