Task 16012145

Name	hadcm3n_o2o7_2020_40_008408295_1
Workunit	8559151
Created	11 Sep 2013, 16:09:18 UTC
Sent	11 Sep 2013, 16:30:59 UTC
Report deadline	11 Dec 2013, 23:58:10 UTC
Received	11 Jan 2014, 16:39:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	25 (0x00000019) Unknown error code
Computer ID	1292564
Run time	6 days 1 hours 59 min 31 sec
CPU time	5 days 6 hours 17 min
Validate state	Invalid
Credit	1,555.20
Device peak FLOPS	1.98 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> The drive cannot locate a specific area or track on the disk. (0x19) - exit code 25 (0x19) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:02:08 (4476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:02:09 (4476): No heartbeat from core client for 30 sec - exiting 03:02:10 (4476): No heartbeat from core client for 30 sec - exiting 03:02:11 (4476): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3948, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4664, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4824, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4332, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5112, iMonCtr=1 Model crash detected, will try to restart... 11:11:53 (4660): No heartbeat from core client for 30 sec - exiting 11:11:54 (4660): No heartbeat from core client for 30 sec - exiting 11:11:55 (4660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4904, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:06:58 (4836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:07:01 (4836): No heartbeat from core client for 30 sec - exiting 15:07:02 (4836): No heartbeat from core client for 30 sec - exiting 15:07:03 (4836): No heartbeat from core client for 30 sec - exiting 15:07:04 (4836): No heartbeat from core client for 30 sec - exiting 15:07:05 (4836): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 20:06:10 (3024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:45:36 (1192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:46:02 (3424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:46:06 (3424): No heartbeat from core client for 30 sec - exiting 13:46:07 (3424): No heartbeat from core client for 30 sec - exiting 13:46:08 (3424): No heartbeat from core client for 30 sec - exiting 13:46:09 (3424): No heartbeat from core client for 30 sec - exiting 13:46:10 (3424): No heartbeat from core client for 30 sec - exiting 13:46:13 (3424): No heartbeat from core client for 30 sec - exiting 13:46:14 (3424): No heartbeat from core client for 30 sec - exiting 13:46:16 (3424): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:04:52 (4680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:28:48 (3472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:33:54 (4040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:39:14 (5200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:40:48 (5600): No heartbeat from core client for 30 sec - exiting 10:40:49 (5600): No heartbeat from core client for 30 sec - exiting 10:40:50 (5600): No heartbeat from core client for 30 sec - exiting 10:40:51 (5600): No heartbeat from core client for 30 sec - exiting 10:40:52 (5600): No heartbeat from core client for 30 sec - exiting 10:40:53 (5600): No heartbeat from core client for 30 sec - exiting 10:40:54 (5600): No heartbeat from core client for 30 sec - exiting 10:40:55 (5600): No heartbeat from core client for 30 sec - exiting 10:40:56 (5600): No heartbeat from core client for 30 sec - exiting 10:40:57 (5600): No heartbeat from core client for 30 sec - exiting 10:40:58 (5600): No heartbeat from core client for 30 sec - exiting 10:40:59 (5600): No heartbeat from core client for 30 sec - exiting 10:41:00 (5600): No heartbeat from core client for 30 sec - exiting 10:41:01 (5600): No heartbeat from core client for 30 sec - exiting 10:41:02 (5600): No heartbeat from core client for 30 sec - exiting 10:41:03 (5600): No heartbeat from core client for 30 sec - exiting 10:41:04 (5600): No heartbeat from core client for 30 sec - exiting 10:41:05 (5600): No heartbeat from core client for 30 sec - exiting 10:41:06 (5600): No heartbeat from core client for 30 sec - exiting 10:41:07 (5600): No heartbeat from core client for 30 sec - exiting 10:41:08 (5600): No heartbeat from core client for 30 sec - exiting 10:41:09 (5600): No heartbeat from core client for 30 sec - exiting 10:41:10 (5600): No heartbeat from core client for 30 sec - exiting 10:41:11 (5600): No heartbeat from core client for 30 sec - exiting 10:41:12 (5600): No heartbeat from core client for 30 sec - exiting 10:41:13 (5600): No heartbeat from core client for 30 sec - exiting 10:41:14 (5600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:50:39 (4600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:33:32 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:33:33 (4904): No heartbeat from core client for 30 sec - exiting 10:37:57 (4976): No heartbeat from core client for 30 sec - exiting 10:37:58 (4976): No heartbeat from core client for 30 sec - exiting 10:37:59 (4976): No heartbeat from core client for 30 sec - exiting 10:38:00 (4976): No heartbeat from core client for 30 sec - exiting 10:38:01 (4976): No heartbeat from core client for 30 sec - exiting 10:38:02 (4976): No heartbeat from core client for 30 sec - exiting 10:38:03 (4976): No heartbeat from core client for 30 sec - exiting 10:38:04 (4976): No heartbeat from core client for 30 sec - exiting 10:38:05 (4976): No heartbeat from core client for 30 sec - exiting 10:38:07 (4976): No heartbeat from core client for 30 sec - exiting 10:38:08 (4976): No heartbeat from core client for 30 sec - exiting 10:38:09 (4976): No heartbeat from core client for 30 sec - exiting 10:38:10 (4976): No heartbeat from core client for 30 sec - exiting 10:38:11 (4976): No heartbeat from core client for 30 sec - exiting 10:38:12 (4976): No heartbeat from core client for 30 sec - exiting 10:38:13 (4976): No heartbeat from core client for 30 sec - exiting 10:38:14 (4976): No heartbeat from core client for 30 sec - exiting 10:38:15 (4976): No heartbeat from core client for 30 sec - exiting 10:38:16 (4976): No heartbeat from core client for 30 sec - exiting 10:38:17 (4976): No heartbeat from core client for 30 sec - exiting 10:38:18 (4976): No heartbeat from core client for 30 sec - exiting 10:38:19 (4976): No heartbeat from core client for 30 sec - exiting 10:38:20 (4976): No heartbeat from core client for 30 sec - exiting 10:38:21 (4976): No heartbeat from core client for 30 sec - exiting 10:38:22 (4976): No heartbeat from core client for 30 sec - exiting 10:38:23 (4976): No heartbeat from core client for 30 sec - exiting 10:38:24 (4976): No heartbeat from core client for 30 sec - exiting 10:38:25 (4976): No heartbeat from core client for 30 sec - exiting 10:38:26 (4976): No heartbeat from core client for 30 sec - exiting 10:38:27 (4976): No heartbeat from core client for 30 sec - exiting 10:38:28 (4976): No heartbeat from core client for 30 sec - exiting 10:38:29 (4976): No heartbeat from core client for 30 sec - exiting 10:38:30 (4976): No heartbeat from core client for 30 sec - exiting 10:38:31 (4976): No heartbeat from core client for 30 sec - exiting 10:38:32 (4976): No heartbeat from core client for 30 sec - exiting 10:38:33 (4976): No heartbeat from core client for 30 sec - exiting 10:38:34 (4976): No heartbeat from core client for 30 sec - exiting 10:38:35 (4976): No heartbeat from core client for 30 sec - exiting 10:38:36 (4976): No heartbeat from core client for 30 sec - exiting 10:38:37 (4976): No heartbeat from core client for 30 sec - exiting 10:38:38 (4976): No heartbeat from core client for 30 sec - exiting 10:38:39 (4976): No heartbeat from core client for 30 sec - exiting 10:38:40 (4976): No heartbeat from core client for 30 sec - exiting 10:38:41 (4976): No heartbeat from core client for 30 sec - exiting 10:38:42 (4976): No heartbeat from core client for 30 sec - exiting 10:38:43 (4976): No heartbeat from core client for 30 sec - exiting 10:38:44 (4976): No heartbeat from core client for 30 sec - exiting 10:38:45 (4976): No heartbeat from core client for 30 sec - exiting 10:38:46 (4976): No heartbeat from core client for 30 sec - exiting 10:38:47 (4976): No heartbeat from core client for 30 sec - exiting 10:38:48 (4976): No heartbeat from core client for 30 sec - exiting 10:38:49 (4976): No heartbeat from core client for 30 sec - exiting 10:38:50 (4976): No heartbeat from core client for 30 sec - exiting 10:38:51 (4976): No heartbeat from core client for 30 sec - exiting 10:38:52 (4976): No heartbeat from core client for 30 sec - exiting 10:38:53 (4976): No heartbeat from core client for 30 sec - exiting 10:38:54 (4976): No heartbeat from core client for 30 sec - exiting 10:38:55 (4976): No heartbeat from core client for 30 sec - exiting 10:38:56 (4976): No heartbeat from core client for 30 sec - exiting 10:38:57 (4976): No heartbeat from core client for 30 sec - exiting 10:38:58 (4976): No heartbeat from core client for 30 sec - exiting 10:38:59 (4976): No heartbeat from core client for 30 sec - exiting 10:39:00 (4976): No heartbeat from core client for 30 sec - exiting 10:39:01 (4976): No heartbeat from core client for 30 sec - exiting 10:39:02 (4976): No heartbeat from core client for 30 sec - exiting 10:39:03 (4976): No heartbeat from core client for 30 sec - exiting 10:39:04 (4976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4620, iMonCtr=1 Model crash detected, will try to restart... 09:34:27 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:42:24 (4412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:42:26 (4412): No heartbeat from core client for 30 sec - exiting 09:42:27 (4412): No heartbeat from core client for 30 sec - exiting 09:42:28 (4412): No heartbeat from core client for 30 sec - exiting 09:42:29 (4412): No heartbeat from core client for 30 sec - exiting 09:42:30 (4412): No heartbeat from core client for 30 sec - exiting 09:46:08 (2824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:48:00 (3888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:48:07 (3888): No heartbeat from core client for 30 sec - exiting 12:48:09 (3888): No heartbeat from core client for 30 sec - exiting 12:48:10 (3888): No heartbeat from core client for 30 sec - exiting 12:48:11 (3888): No heartbeat from core client for 30 sec - exiting 12:48:12 (3888): No heartbeat from core client for 30 sec - exiting 12:48:13 (3888): No heartbeat from core client for 30 sec - exiting 12:48:14 (3888): No heartbeat from core client for 30 sec - exiting 12:48:15 (3888): No heartbeat from core client for 30 sec - exiting 12:55:59 (4636): No heartbeat from core client for 30 sec - exiting 12:56:00 (4636): No heartbeat from core client for 30 sec - exiting 12:56:01 (4636): No heartbeat from core client for 30 sec - exiting 12:56:02 (4636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:25:15 (4404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:29:50 (6064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:29:52 (6064): No heartbeat from core client for 30 sec - exiting 16:29:53 (6064): No heartbeat from core client for 30 sec - exiting 16:33:15 (5024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:35:20 (6100): No heartbeat from core client for 30 sec - exiting 16:35:21 (6100): No heartbeat from core client for 30 sec - exiting 16:35:22 (6100): No heartbeat from core client for 30 sec - exiting 16:35:23 (6100): No heartbeat from core client for 30 sec - exiting 16:35:24 (6100): No heartbeat from core client for 30 sec - exiting 16:35:25 (6100): No heartbeat from core client for 30 sec - exiting 16:35:26 (6100): No heartbeat from core client for 30 sec - exiting 16:35:27 (6100): No heartbeat from core client for 30 sec - exiting 16:35:28 (6100): No heartbeat from core client for 30 sec - exiting 16:35:29 (6100): No heartbeat from core client for 30 sec - exiting 16:35:30 (6100): No heartbeat from core client for 30 sec - exiting 16:35:31 (6100): No heartbeat from core client for 30 sec - exiting 16:35:32 (6100): No heartbeat from core client for 30 sec - exiting 16:35:33 (6100): No heartbeat from core client for 30 sec - exiting 16:35:34 (6100): No heartbeat from core client for 30 sec - exiting 16:35:35 (6100): No heartbeat from core client for 30 sec - exiting 16:35:36 (6100): No heartbeat from core client for 30 sec - exiting 16:35:37 (6100): No heartbeat from core client for 30 sec - exiting 16:35:38 (6100): No heartbeat from core client for 30 sec - exiting 16:35:39 (6100): No heartbeat from core client for 30 sec - exiting 16:35:40 (6100): No heartbeat from core client for 30 sec - exiting 16:35:41 (6100): No heartbeat from core client for 30 sec - exiting 16:35:42 (6100): No heartbeat from core client for 30 sec - exiting 16:35:43 (6100): No heartbeat from core client for 30 sec - exiting 16:35:44 (6100): No heartbeat from core client for 30 sec - exiting 16:35:45 (6100): No heartbeat from core client for 30 sec - exiting 16:35:46 (6100): No heartbeat from core client for 30 sec - exiting 16:35:47 (6100): No heartbeat from core client for 30 sec - exiting 16:35:48 (6100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:25:39 (3120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:25:42 (3120): No heartbeat from core client for 30 sec - exiting 16:25:43 (3120): No heartbeat from core client for 30 sec - exiting 16:25:44 (3120): No heartbeat from core client for 30 sec - exiting 16:25:45 (3120): No heartbeat from core client for 30 sec - exiting 16:30:10 (5340): No heartbeat from core client for 30 sec - exiting 16:30:11 (5340): No heartbeat from core client for 30 sec - exiting 16:30:12 (5340): No heartbeat from core client for 30 sec - exiting 16:30:13 (5340): No heartbeat from core client for 30 sec - exiting 16:30:14 (5340): No heartbeat from core client for 30 sec - exiting 16:30:15 (5340): No heartbeat from core client for 30 sec - exiting 16:30:16 (5340): No heartbeat from core client for 30 sec - exiting 16:30:17 (5340): No heartbeat from core client for 30 sec - exiting 16:30:18 (5340): No heartbeat from core client for 30 sec - exiting 16:30:19 (5340): No heartbeat from core client for 30 sec - exiting 16:30:20 (5340): No heartbeat from core client for 30 sec - exiting 16:30:21 (5340): No heartbeat from core client for 30 sec - exiting 16:30:22 (5340): No heartbeat from core client for 30 sec - exiting 16:30:23 (5340): No heartbeat from core client for 30 sec - exiting 16:30:24 (5340): No heartbeat from core client for 30 sec - exiting 16:30:25 (5340): No heartbeat from core client for 30 sec - exiting 16:30:26 (5340): No heartbeat from core client for 30 sec - exiting 16:30:27 (5340): No heartbeat from core client for 30 sec - exiting 16:30:28 (5340): No heartbeat from core client for 30 sec - exiting 16:30:29 (5340): No heartbeat from core client for 30 sec - exiting 16:30:30 (5340): No heartbeat from core client for 30 sec - exiting 16:30:31 (5340): No heartbeat from core client for 30 sec - exiting 16:30:32 (5340): No heartbeat from core client for 30 sec - exiting 16:30:33 (5340): No heartbeat from core client for 30 sec - exiting 16:30:34 (5340): No heartbeat from core client for 30 sec - exiting 16:30:35 (5340): No heartbeat from core client for 30 sec - exiting 16:30:36 (5340): No heartbeat from core client for 30 sec - exiting 16:30:37 (5340): No heartbeat from core client for 30 sec - exiting 16:30:38 (5340): No heartbeat from core client for 30 sec - exiting 16:31:11 (5340): No heartbeat from core client for 30 sec - exiting 16:31:12 (5340): No heartbeat from core client for 30 sec - exiting 16:31:13 (5340): No heartbeat from core client for 30 sec - exiting 16:31:14 (5340): No heartbeat from core client for 30 sec - exiting 16:31:15 (5340): No heartbeat from core client for 30 sec - exiting 16:31:16 (5340): No heartbeat from core client for 30 sec - exiting 16:31:17 (5340): No heartbeat from core client for 30 sec - exiting 16:31:18 (5340): No heartbeat from core client for 30 sec - exiting 16:31:19 (5340): No heartbeat from core client for 30 sec - exiting 16:31:20 (5340): No heartbeat from core client for 30 sec - exiting 16:31:21 (5340): No heartbeat from core client for 30 sec - exiting 16:31:22 (5340): No heartbeat from core client for 30 sec - exiting 16:31:23 (5340): No heartbeat from core client for 30 sec - exiting 16:31:24 (5340): No heartbeat from core client for 30 sec - exiting 16:31:25 (5340): No heartbeat from core client for 30 sec - exiting 16:31:26 (5340): No heartbeat from core client for 30 sec - exiting 16:31:27 (5340): No heartbeat from core client for 30 sec - exiting 16:31:28 (5340): No heartbeat from core client for 30 sec - exiting 16:31:29 (5340): No heartbeat from core client for 30 sec - exiting 16:31:30 (5340): No heartbeat from core client for 30 sec - exiting 16:31:31 (5340): No heartbeat from core client for 30 sec - exiting 16:31:32 (5340): No heartbeat from core client for 30 sec - exiting 16:31:33 (5340): No heartbeat from core client for 30 sec - exiting 16:31:34 (5340): No heartbeat from core client for 30 sec - exiting 16:31:35 (5340): No heartbeat from core client for 30 sec - exiting 16:31:36 (5340): No heartbeat from core client for 30 sec - exiting 16:31:37 (5340): No heartbeat from core client for 30 sec - exiting 16:31:38 (5340): No heartbeat from core client for 30 sec - exiting 16:31:39 (5340): No heartbeat from core client for 30 sec - exiting 16:31:40 (5340): No heartbeat from core client for 30 sec - exiting 16:31:41 (5340): No heartbeat from core client for 30 sec - exiting 16:31:42 (5340): No heartbeat from core client for 30 sec - exiting 16:31:43 (5340): No heartbeat from core client for 30 sec - exiting 16:31:44 (5340): No heartbeat from core client for 30 sec - exiting 16:31:45 (5340): No heartbeat from core client for 30 sec - exiting 16:31:46 (5340): No heartbeat from core client for 30 sec - exiting 16:31:47 (5340): No heartbeat from core client for 30 sec - exiting 16:31:48 (5340): No heartbeat from core client for 30 sec - exiting 16:31:49 (5340): No heartbeat from core client for 30 sec - exiting 16:31:50 (5340): No heartbeat from core client for 30 sec - exiting 16:31:51 (5340): No heartbeat from core client for 30 sec - exiting 16:31:52 (5340): No heartbeat from core client for 30 sec - exiting 16:31:53 (5340): No heartbeat from core client for 30 sec - exiting 16:31:54 (5340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:34:24 (5968): No heartbeat from core client for 30 sec - exiting 16:34:25 (5968): No heartbeat from core client for 30 sec - exiting 16:34:26 (5968): No heartbeat from core client for 30 sec - exiting 16:34:27 (5968): No heartbeat from core client for 30 sec - exiting 16:34:28 (5968): No heartbeat from core client for 30 sec - exiting 16:34:29 (5968): No heartbeat from core client for 30 sec - exiting 16:34:30 (5968): No heartbeat from core client for 30 sec - exiting 16:34:31 (5968): No heartbeat from core client for 30 sec - exiting 16:34:32 (5968): No heartbeat from core client for 30 sec - exiting 16:34:33 (5968): No heartbeat from core client for 30 sec - exiting 16:34:34 (5968): No heartbeat from core client for 30 sec - exiting 16:34:35 (5968): No heartbeat from core client for 30 sec - exiting 16:34:36 (5968): No heartbeat from core client for 30 sec - exiting 16:34:37 (5968): No heartbeat from core client for 30 sec - exiting 16:34:38 (5968): No heartbeat from core client for 30 sec - exiting 16:34:39 (5968): No heartbeat from core client for 30 sec - exiting 16:34:40 (5968): No heartbeat from core client for 30 sec - exiting 16:34:41 (5968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:44:28 (5832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:57:51 (5228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:57:53 (5228): No heartbeat from core client for 30 sec - exiting 09:57:54 (5228): No heartbeat from core client for 30 sec - exiting 09:57:55 (5228): No heartbeat from core client for 30 sec - exiting 09:57:56 (5228): No heartbeat from core client for 30 sec - exiting 09:57:57 (5228): No heartbeat from core client for 30 sec - exiting 10:00:11 (4176): No heartbeat from core client for 30 sec - exiting 10:00:12 (4176): No heartbeat from core client for 30 sec - exiting 10:00:13 (4176): No heartbeat from core client for 30 sec - exiting 10:00:14 (4176): No heartbeat from core client for 30 sec - exiting 10:00:15 (4176): No heartbeat from core client for 30 sec - exiting 10:00:16 (4176): No heartbeat from core client for 30 sec - exiting 10:00:17 (4176): No heartbeat from core client for 30 sec - exiting 10:00:18 (4176): No heartbeat from core client for 30 sec - exiting 10:00:19 (4176): No heartbeat from core client for 30 sec - exiting 10:00:20 (4176): No heartbeat from core client for 30 sec - exiting 10:00:21 (4176): No heartbeat from core client for 30 sec - exiting 10:00:22 (4176): No heartbeat from core client for 30 sec - exiting 10:00:23 (4176): No heartbeat from core client for 30 sec - exiting 10:00:24 (4176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:01:45 (2804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:09:07 (4024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:10:43 (300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:11:40 (3200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:06:58 (5660): No heartbeat from core client for 30 sec - exiting 14:07:00 (5660): No heartbeat from core client for 30 sec - exiting 14:07:01 (5660): No heartbeat from core client for 30 sec - exiting 14:07:02 (5660): No heartbeat from core client for 30 sec - exiting 14:07:03 (5660): No heartbeat from core client for 30 sec - exiting 14:07:04 (5660): No heartbeat from core client for 30 sec - exiting 14:07:05 (5660): No heartbeat from core client for 30 sec - exiting 14:07:06 (5660): No heartbeat from core client for 30 sec - exiting 14:07:07 (5660): No heartbeat from core client for 30 sec - exiting 14:07:08 (5660): No heartbeat from core client for 30 sec - exiting 14:07:09 (5660): No heartbeat from core client for 30 sec - exiting 14:07:10 (5660): No heartbeat from core client for 30 sec - exiting 14:07:11 (5660): No heartbeat from core client for 30 sec - exiting 14:07:12 (5660): No heartbeat from core client for 30 sec - exiting 14:07:13 (5660): No heartbeat from core client for 30 sec - exiting 14:07:14 (5660): No heartbeat from core client for 30 sec - exiting 14:07:15 (5660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5684, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4368, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 15:16:41 (448): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 15:17:36 (3588): No heartbeat from core client for 30 sec - exiting 15:17:37 (3588): No heartbeat from core client for 30 sec - exiting 15:17:38 (3588): No heartbeat from core client for 30 sec - exiting 15:17:40 (3588): No heartbeat from core client for 30 sec - exiting 15:17:41 (3588): No heartbeat from core client for 30 sec - exiting 15:17:42 (3588): No heartbeat from core client for 30 sec - exiting 15:17:43 (3588): No heartbeat from core client for 30 sec - exiting 15:17:44 (3588): No heartbeat from core client for 30 sec - exiting 15:17:45 (3588): No heartbeat from core client for 30 sec - exiting 15:17:46 (3588): No heartbeat from core client for 30 sec - exiting 15:17:48 (3588): No heartbeat from core client for 30 sec - exiting 15:17:49 (3588): No heartbeat from core client for 30 sec - exiting 15:17:50 (3588): No heartbeat from core client for 30 sec - exiting 15:17:51 (3588): No heartbeat from core client for 30 sec - exiting 15:17:52 (3588): No heartbeat from core client for 30 sec - exiting 15:17:53 (3588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:22:05 (4544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:38:29 (4196): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:58:57 (5252): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 10:59:12 (5252): No heartbeat from core client for 30 sec - exiting 10:59:13 (5252): No heartbeat from core client for 30 sec - exiting 10:59:14 (5252): No heartbeat from core client for 30 sec - exiting 10:59:15 (5252): No heartbeat from core client for 30 sec - exiting 10:59:16 (5252): No heartbeat from core client for 30 sec - exiting 10:59:17 (5252): No heartbeat from core client for 30 sec - exiting 10:59:18 (5252): No heartbeat from core client for 30 sec - exiting 10:59:19 (5252): No heartbeat from core client for 30 sec - exiting 10:59:20 (5252): No heartbeat from core client for 30 sec - exiting 10:59:22 (5252): No heartbeat from core client for 30 sec - exiting 10:59:23 (5252): No heartbeat from core client for 30 sec - exiting 10:59:24 (5252): No heartbeat from core client for 30 sec - exiting 11:06:06 (5940): No heartbeat from core client for 30 sec - exiting 11:06:07 (5940): No heartbeat from core client for 30 sec - exiting 11:06:08 (5940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1532, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5116, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Oct 2013 21:03:09	1292564	16012145	hadcm3n_o2o7_2020_40_008408295_1	129,600	392,457	3.0282
07 Oct 2013 18:29:27	1292564	16012145	hadcm3n_o2o7_2020_40_008408295_1	103,680	314,260	3.0311
02 Oct 2013 16:06:20	1292564	16012145	hadcm3n_o2o7_2020_40_008408295_1	77,760	236,776	3.0450
25 Sep 2013 18:11:24	1292564	16012145	hadcm3n_o2o7_2020_40_008408295_1	51,840	159,123	3.0695
18 Sep 2013 19:43:43	1292564	16012145	hadcm3n_o2o7_2020_40_008408295_1	25,920	79,721	3.0757