Task 13667875

Name	hadcm3n_t1wo_1940_40_007545939_0
Workunit	7743171
Created	29 Nov 2011, 0:53:45 UTC
Sent	29 Nov 2011, 0:58:10 UTC
Report deadline	28 Feb 2012, 8:25:21 UTC
Received	23 Dec 2011, 19:11:49 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1182094
Run time	4 days 20 hours 8 min 30 sec
CPU time	4 days 17 hours 56 min 1 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.34 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t1woko.pje1c10 Error converting file to netcdf: dataout/t1woko.pie1c10 Error converting file to netcdf: dataout/t1woko.pfe1c10 Error converting file to netcdf: dataout/t1woka.phe1c10 Error converting file to netcdf: dataout/t1woka.pge1c10 Error converting file to netcdf: dataout/t1woka.pee1c10 Error converting file to netcdf: dataout/t1woka.pde1c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3828, iMonCtr=1 Model crash detected, will try to restart... 13:12:45 (3752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:14:58 (3880): No heartbeat from core client for 30 sec - exiting 13:14:59 (3880): No heartbeat from core client for 30 sec - exiting 13:15:00 (3880): No heartbeat from core client for 30 sec - exiting 13:15:01 (3880): No heartbeat from core client for 30 sec - exiting 13:15:02 (3880): No heartbeat from core client for 30 sec - exiting 13:15:03 (3880): No heartbeat from core client for 30 sec - exiting 13:15:04 (3880): No heartbeat from core client for 30 sec - exiting 13:15:05 (3880): No heartbeat from core client for 30 sec - exiting 13:15:06 (3880): No heartbeat from core client for 30 sec - exiting 13:15:07 (3880): No heartbeat from core client for 30 sec - exiting 13:15:08 (3880): No heartbeat from core client for 30 sec - exiting 13:15:09 (3880): No heartbeat from core client for 30 sec - exiting 13:15:10 (3880): No heartbeat from core client for 30 sec - exiting 13:15:11 (3880): No heartbeat from core client for 30 sec - exiting 13:15:12 (3880): No heartbeat from core client for 30 sec - exiting 13:15:13 (3880): No heartbeat from core client for 30 sec - exiting 13:15:14 (3880): No heartbeat from core client for 30 sec - exiting 13:15:15 (3880): No heartbeat from core client for 30 sec - exiting 13:15:16 (3880): No heartbeat from core client for 30 sec - exiting 13:15:17 (3880): No heartbeat from core client for 30 sec - exiting 13:15:18 (3880): No heartbeat from core client for 30 sec - exiting 13:15:19 (3880): No heartbeat from core client for 30 sec - exiting 13:15:20 (3880): No heartbeat from core client for 30 sec - exiting 13:15:21 (3880): No heartbeat from core client for 30 sec - exiting 13:15:22 (3880): No heartbeat from core client for 30 sec - exiting 13:15:23 (3880): No heartbeat from core client for 30 sec - exiting 13:15:24 (3880): No heartbeat from core client for 30 sec - exiting 13:15:25 (3880): No heartbeat from core client for 30 sec - exiting 13:15:26 (3880): No heartbeat from core client for 30 sec - exiting 13:15:27 (3880): No heartbeat from core client for 30 sec - exiting 13:15:28 (3880): No heartbeat from core client for 30 sec - exiting 13:15:29 (3880): No heartbeat from core client for 30 sec - exiting 13:15:30 (3880): No heartbeat from core client for 30 sec - exiting 13:15:31 (3880): No heartbeat from core client for 30 sec - exiting 13:15:32 (3880): No heartbeat from core client for 30 sec - exiting 13:15:33 (3880): No heartbeat from core client for 30 sec - exiting 13:15:34 (3880): No heartbeat from core client for 30 sec - exiting 13:15:35 (3880): No heartbeat from core client for 30 sec - exiting 13:15:36 (3880): No heartbeat from core client for 30 sec - exiting 13:15:37 (3880): No heartbeat from core client for 30 sec - exiting 13:15:38 (3880): No heartbeat from core client for 30 sec - exiting 13:15:39 (3880): No heartbeat from core client for 30 sec - exiting 13:15:40 (3880): No heartbeat from core client for 30 sec - exiting 13:15:41 (3880): No heartbeat from core client for 30 sec - exiting 13:15:42 (3880): No heartbeat from core client for 30 sec - exiting 13:15:43 (3880): No heartbeat from core client for 30 sec - exiting 13:15:44 (3880): No heartbeat from core client for 30 sec - exiting 13:15:45 (3880): No heartbeat from core client for 30 sec - exiting 13:15:46 (3880): No heartbeat from core client for 30 sec - exiting 13:15:47 (3880): No heartbeat from core client for 30 sec - exiting 13:15:48 (3880): No heartbeat from core client for 30 sec - exiting 13:15:49 (3880): No heartbeat from core client for 30 sec - exiting 13:15:50 (3880): No heartbeat from core client for 30 sec - exiting 13:15:51 (3880): No heartbeat from core client for 30 sec - exiting 13:15:52 (3880): No heartbeat from core client for 30 sec - exiting 13:15:53 (3880): No heartbeat from core client for 30 sec - exiting 13:15:54 (3880): No heartbeat from core client for 30 sec - exiting 13:15:55 (3880): No heartbeat from core client for 30 sec - exiting 13:15:56 (3880): No heartbeat from core client for 30 sec - exiting 13:15:57 (3880): No heartbeat from core client for 30 sec - exiting 13:15:58 (3880): No heartbeat from core client for 30 sec - exiting 13:15:59 (3880): No heartbeat from core client for 30 sec - exiting 13:16:00 (3880): No heartbeat from core client for 30 sec - exiting 13:16:01 (3880): No heartbeat from core client for 30 sec - exiting 13:16:02 (3880): No heartbeat from core client for 30 sec - exiting 13:16:03 (3880): No heartbeat from core client for 30 sec - exiting 13:16:04 (3880): No heartbeat from core client for 30 sec - exiting 13:16:05 (3880): No heartbeat from core client for 30 sec - exiting 13:16:06 (3880): No heartbeat from core client for 30 sec - exiting 13:16:07 (3880): No heartbeat from core client for 30 sec - exiting 13:16:08 (3880): No heartbeat from core client for 30 sec - exiting 13:16:09 (3880): No heartbeat from core client for 30 sec - exiting 13:16:10 (3880): No heartbeat from core client for 30 sec - exiting 13:16:11 (3880): No heartbeat from core client for 30 sec - exiting 13:16:12 (3880): No heartbeat from core client for 30 sec - exiting 13:16:13 (3880): No heartbeat from core client for 30 sec - exiting 13:16:14 (3880): No heartbeat from core client for 30 sec - exiting 13:16:15 (3880): No heartbeat from core client for 30 sec - exiting 13:16:16 (3880): No heartbeat from core client for 30 sec - exiting 13:16:17 (3880): No heartbeat from core client for 30 sec - exiting 13:16:18 (3880): No heartbeat from core client for 30 sec - exiting 13:16:19 (3880): No heartbeat from core client for 30 sec - exiting 13:16:20 (3880): No heartbeat from core client for 30 sec - exiting 13:16:21 (3880): No heartbeat from core client for 30 sec - exiting 13:16:22 (3880): No heartbeat from core client for 30 sec - exiting 13:16:23 (3880): No heartbeat from core client for 30 sec - exiting 13:16:24 (3880): No heartbeat from core client for 30 sec - exiting 13:16:25 (3880): No heartbeat from core client for 30 sec - exiting 13:16:26 (3880): No heartbeat from core client for 30 sec - exiting 13:16:27 (3880): No heartbeat from core client for 30 sec - exiting 13:16:28 (3880): No heartbeat from core client for 30 sec - exiting 13:16:29 (3880): No heartbeat from core client for 30 sec - exiting 13:16:30 (3880): No heartbeat from core client for 30 sec - exiting 13:16:31 (3880): No heartbeat from core client for 30 sec - exiting 13:16:32 (3880): No heartbeat from core client for 30 sec - exiting 13:16:33 (3880): No heartbeat from core client for 30 sec - exiting 13:16:34 (3880): No heartbeat from core client for 30 sec - exiting 13:16:35 (3880): No heartbeat from core client for 30 sec - exiting 13:16:36 (3880): No heartbeat from core client for 30 sec - exiting 13:16:37 (3880): No heartbeat from core client for 30 sec - exiting 13:16:38 (3880): No heartbeat from core client for 30 sec - exiting 13:16:39 (3880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:17:13 (1116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:17:14 (1116): No heartbeat from core client for 30 sec - exiting 13:17:15 (1116): No heartbeat from core client for 30 sec - exiting 13:17:16 (1116): No heartbeat from core client for 30 sec - exiting 13:17:17 (1116): No heartbeat from core client for 30 sec - exiting 13:17:18 (1116): No heartbeat from core client for 30 sec - exiting 13:17:19 (1116): No heartbeat from core client for 30 sec - exiting 13:17:20 (1116): No heartbeat from core client for 30 sec - exiting 13:17:21 (1116): No heartbeat from core client for 30 sec - exiting 13:17:22 (1116): No heartbeat from core client for 30 sec - exiting 13:17:23 (1116): No heartbeat from core client for 30 sec - exiting 13:43:49 (3780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:44:26 (3780): No heartbeat from core client for 30 sec - exiting 13:44:27 (3780): No heartbeat from core client for 30 sec - exiting 13:44:28 (3780): No heartbeat from core client for 30 sec - exiting 13:44:29 (3780): No heartbeat from core client for 30 sec - exiting 13:44:30 (3780): No heartbeat from core client for 30 sec - exiting 13:44:31 (3780): No heartbeat from core client for 30 sec - exiting 13:44:32 (3780): No heartbeat from core client for 30 sec - exiting 13:44:33 (3780): No heartbeat from core client for 30 sec - exiting 13:44:34 (3780): No heartbeat from core client for 30 sec - exiting 13:44:35 (3780): No heartbeat from core client for 30 sec - exiting 13:47:18 (664): No heartbeat from core client for 30 sec - exiting 13:47:19 (664): No heartbeat from core client for 30 sec - exiting 13:47:20 (664): No heartbeat from core client for 30 sec - exiting 13:47:21 (664): No heartbeat from core client for 30 sec - exiting 13:47:22 (664): No heartbeat from core client for 30 sec - exiting 13:47:23 (664): No heartbeat from core client for 30 sec - exiting 13:47:24 (664): No heartbeat from core client for 30 sec - exiting 13:47:25 (664): No heartbeat from core client for 30 sec - exiting 13:47:26 (664): No heartbeat from core client for 30 sec - exiting 13:47:27 (664): No heartbeat from core client for 30 sec - exiting 13:47:28 (664): No heartbeat from core client for 30 sec - exiting 13:47:29 (664): No heartbeat from core client for 30 sec - exiting 13:47:30 (664): No heartbeat from core client for 30 sec - exiting 13:47:31 (664): No heartbeat from core client for 30 sec - exiting 13:47:32 (664): No heartbeat from core client for 30 sec - exiting 13:47:33 (664): No heartbeat from core client for 30 sec - exiting 13:47:34 (664): No heartbeat from core client for 30 sec - exiting 13:47:35 (664): No heartbeat from core client for 30 sec - exiting 13:47:36 (664): No heartbeat from core client for 30 sec - exiting 13:47:37 (664): No heartbeat from core client for 30 sec - exiting 13:47:38 (664): No heartbeat from core client for 30 sec - exiting 13:47:39 (664): No heartbeat from core client for 30 sec - exiting 13:47:40 (664): No heartbeat from core client for 30 sec - exiting 13:47:41 (664): No heartbeat from core client for 30 sec - exiting 13:47:42 (664): No heartbeat from core client for 30 sec - exiting 13:47:43 (664): No heartbeat from core client for 30 sec - exiting 13:47:44 (664): No heartbeat from core client for 30 sec - exiting 13:47:45 (664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 14:30:38 (4780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:30:39 (4780): No heartbeat from core client for 30 sec - exiting 14:30:40 (4780): No heartbeat from core client for 30 sec - exiting 14:30:41 (4780): No heartbeat from core client for 30 sec - exiting 14:30:42 (4780): No heartbeat from core client for 30 sec - exiting 14:30:43 (4780): No heartbeat from core client for 30 sec - exiting 14:30:44 (4780): No heartbeat from core client for 30 sec - exiting 14:30:45 (4780): No heartbeat from core client for 30 sec - exiting 14:30:46 (4780): No heartbeat from core client for 30 sec - exiting 14:30:47 (4780): No heartbeat from core client for 30 sec - exiting 14:30:48 (4780): No heartbeat from core client for 30 sec - exiting 14:48:23 (4916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:47 (3872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1476, iMonCtr=1 Model crash detected, will try to restart... 14:51:21 (2188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3752, iMonCtr=1 Model crash detected, will try to restart... 09:56:26 (456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:57:39 (3624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3868, iMonCtr=1 Model crash detected, will try to restart... 09:38:22 (2752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:38:54 (184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:20:00 (2260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Dec 2011 19:15:16	1182094	13667875	hadcm3n_t1wo_1940_40_007545939_0	259,200	410,159	1.5824
10 Dec 2011 21:08:31	1182094	13667875	hadcm3n_t1wo_1940_40_007545939_0	233,280	371,239	1.5914
09 Dec 2011 21:08:01	1182094	13667875	hadcm3n_t1wo_1940_40_007545939_0	207,360	329,576	1.5894
08 Dec 2011 21:59:53	1182094	13667875	hadcm3n_t1wo_1940_40_007545939_0	181,440	287,956	1.5871
07 Dec 2011 19:58:14	1182094	13667875	hadcm3n_t1wo_1940_40_007545939_0	155,520	246,309	1.5838
06 Dec 2011 20:00:34	1182094	13667875	hadcm3n_t1wo_1940_40_007545939_0	129,600	206,469	1.5931
05 Dec 2011 18:36:42	1182094	13667875	hadcm3n_t1wo_1940_40_007545939_0	103,680	164,109	1.5828
03 Dec 2011 17:28:41	1182094	13667875	hadcm3n_t1wo_1940_40_007545939_0	77,760	121,709	1.5652
02 Dec 2011 03:30:31	1182094	13667875	hadcm3n_t1wo_1940_40_007545939_0	51,840	79,395	1.5315
30 Nov 2011 22:16:55	1182094	13667875	hadcm3n_t1wo_1940_40_007545939_0	25,920	39,290	1.5158