Task 16008229

Name	hadcm3n_80b5_1980_40_008458340_1
Workunit	8609196
Created	8 Sep 2013, 13:44:10 UTC
Sent	8 Sep 2013, 13:46:25 UTC
Report deadline	8 Dec 2013, 21:13:36 UTC
Received	5 Dec 2013, 17:03:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1213041
Run time	24 days 3 hours 4 min 30 sec
CPU time	19 days 11 hours 15 min 8 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.38 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:49:26 (6012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:39:15 (3364): No heartbeat from core client for 30 sec - exiting 10:39:16 (3364): No heartbeat from core client for 30 sec - exiting 10:39:17 (3364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:39:18 (3364): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:42:15 (5252): No heartbeat from core client for 30 sec - exiting 16:42:16 (5252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:52:43 (5916): No heartbeat from core client for 30 sec - exiting 16:52:44 (5916): No heartbeat from core client for 30 sec - exiting 16:52:45 (5916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:52:46 (5916): No heartbeat from core client for 30 sec - exiting 16:26:16 (612): No heartbeat from core client for 30 sec - exiting 16:26:17 (612): No heartbeat from core client for 30 sec - exiting 16:26:18 (612): No heartbeat from core client for 30 sec - exiting 16:26:19 (612): No heartbeat from core client for 30 sec - exiting 16:26:20 (612): No heartbeat from core client for 30 sec - exiting 16:26:21 (612): No heartbeat from core client for 30 sec - exiting 16:26:22 (612): No heartbeat from core client for 30 sec - exiting 16:26:23 (612): No heartbeat from core client for 30 sec - exiting 16:26:24 (612): No heartbeat from core client for 30 sec - exiting 16:26:25 (612): No heartbeat from core client for 30 sec - exiting 16:26:26 (612): No heartbeat from core client for 30 sec - exiting 16:26:27 (612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:35:27 (5532): No heartbeat from core client for 30 sec - exiting 16:35:28 (5532): No heartbeat from core client for 30 sec - exiting 16:35:29 (5532): No heartbeat from core client for 30 sec - exiting 16:35:30 (5532): No heartbeat from core client for 30 sec - exiting 16:35:31 (5532): No heartbeat from core client for 30 sec - exiting 16:35:32 (5532): No heartbeat from core client for 30 sec - exiting 16:35:33 (5532): No heartbeat from core client for 30 sec - exiting 16:35:34 (5532): No heartbeat from core client for 30 sec - exiting 16:35:35 (5532): No heartbeat from core client for 30 sec - exiting 16:35:36 (5532): No heartbeat from core client for 30 sec - exiting 16:35:37 (5532): No heartbeat from core client for 30 sec - exiting 16:35:38 (5532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:53:01 (5540): No heartbeat from core client for 30 sec - exiting 09:53:02 (5540): No heartbeat from core client for 30 sec - exiting 09:53:03 (5540): No heartbeat from core client for 30 sec - exiting 09:53:04 (5540): No heartbeat from core client for 30 sec - exiting 09:53:05 (5540): No heartbeat from core client for 30 sec - exiting 09:53:07 (5540): No heartbeat from core client for 30 sec - exiting 09:53:08 (5540): No heartbeat from core client for 30 sec - exiting 09:53:09 (5540): No heartbeat from core client for 30 sec - exiting 09:53:10 (5540): No heartbeat from core client for 30 sec - exiting 09:53:11 (5540): No heartbeat from core client for 30 sec - exiting 09:53:12 (5540): No heartbeat from core client for 30 sec - exiting 09:53:13 (5540): No heartbeat from core client for 30 sec - exiting 09:53:14 (5540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:55:16 (5996): No heartbeat from core client for 30 sec - exiting 17:55:17 (5996): No heartbeat from core client for 30 sec - exiting 17:55:18 (5996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:59:57 (5288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:09:30 (5144): No heartbeat from core client for 30 sec - exiting 18:09:31 (5144): No heartbeat from core client for 30 sec - exiting 18:09:32 (5144): No heartbeat from core client for 30 sec - exiting 18:09:33 (5144): No heartbeat from core client for 30 sec - exiting 18:09:34 (5144): No heartbeat from core client for 30 sec - exiting 18:09:35 (5144): No heartbeat from core client for 30 sec - exiting 18:09:36 (5144): No heartbeat from core client for 30 sec - exiting 18:09:37 (5144): No heartbeat from core client for 30 sec - exiting 18:09:38 (5144): No heartbeat from core client for 30 sec - exiting 18:09:40 (5144): No heartbeat from core client for 30 sec - exiting 18:09:41 (5144): No heartbeat from core client for 30 sec - exiting 18:09:42 (5144): No heartbeat from core client for 30 sec - exiting 18:09:43 (5144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4048, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4048, iMonCtr=1 Model crash detected, will try to restart... 12:38:21 (5324): No heartbeat from core client for 30 sec - exiting 12:38:22 (5324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:43:57 (5236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:48:41 (6100): No heartbeat from core client for 30 sec - exiting 17:48:42 (6100): No heartbeat from core client for 30 sec - exiting 17:48:43 (6100): No heartbeat from core client for 30 sec - exiting 17:48:44 (6100): No heartbeat from core client for 30 sec - exiting 17:48:46 (6100): No heartbeat from core client for 30 sec - exiting 17:48:47 (6100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:44:41 (5336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:44:42 (5336): No heartbeat from core client for 30 sec - exiting 16:37:02 (6000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:21:13 (6484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:19:05 (3644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:43:40 (5804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:20:18 (1992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:44:18 (5868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:16:06 (7016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:39:07 (6412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:28:57 (5472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7064, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:56:06 (5800): No heartbeat from core client for 30 sec - exiting 16:56:08 (5800): No heartbeat from core client for 30 sec - exiting 16:56:09 (5800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5256, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7400, iMonCtr=1 Model crash detected, will try to restart... 16:51:41 (5784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:34:04 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Dec 2013 23:34:45	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	777,600	1,682,104	2.1632
30 Nov 2013 16:18:49	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	751,680	1,629,079	2.1673
28 Nov 2013 18:45:25	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	725,760	1,575,125	2.1703
24 Nov 2013 22:10:38	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	699,840	1,520,769	2.1730
23 Nov 2013 19:38:22	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	673,920	1,466,355	2.1759
21 Nov 2013 17:09:18	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	648,000	1,412,137	2.1792
16 Nov 2013 18:10:25	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	622,080	1,357,221	2.1817
13 Nov 2013 22:35:58	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	596,160	1,302,307	2.1845
10 Nov 2013 21:08:44	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	570,240	1,247,141	2.1870
08 Nov 2013 20:53:20	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	544,320	1,192,402	2.1906
07 Nov 2013 11:16:08	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	518,400	1,138,311	2.1958
05 Nov 2013 18:32:38	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	492,480	1,083,257	2.1996
03 Nov 2013 11:48:42	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	466,560	1,027,714	2.2027
28 Oct 2013 20:11:16	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	440,640	971,417	2.2046
26 Oct 2013 16:40:58	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	414,720	914,017	2.2039
22 Oct 2013 19:40:50	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	388,800	857,104	2.2045
19 Oct 2013 15:30:22	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	362,880	799,631	2.2036
15 Oct 2013 21:41:09	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	336,960	743,002	2.2050
13 Oct 2013 12:31:37	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	311,040	685,938	2.2053
09 Oct 2013 16:55:50	1213041	16008229	hadcm3n_80b5_1980_40_008458340_1	285,120	629,032	2.2062