Task 13406763

Name	hadcm3n_u2ur_1980_40_007458411_1
Workunit	7655914
Created	22 Sep 2011, 14:38:52 UTC
Sent	22 Sep 2011, 14:41:21 UTC
Report deadline	22 Dec 2011, 22:08:32 UTC
Received	10 Oct 2011, 7:00:32 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1165189
Run time	5 days 4 hours 43 min 44 sec
CPU time	4 days 20 hours 31 min 8 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.44 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/u2urko.pjj0c10 Error converting file to netcdf: dataout/u2urko.pij0c10 Error converting file to netcdf: dataout/u2urko.pfj0c10 Error converting file to netcdf: dataout/u2urko.pcj0c10 Error converting file to netcdf: dataout/u2urko.pbj0c10 Error converting file to netcdf: dataout/u2urko.paj0c10 Error converting file to netcdf: dataout/u2urka.phj0c10 Error converting file to netcdf: dataout/u2urka.pgj0c10 Error converting file to netcdf: dataout/u2urka.pej0c10 Error converting file to netcdf: dataout/u2urka.pdj0c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:10:54 (4876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:10:55 (4876): No heartbeat from core client for 30 sec - exiting 15:10:56 (4876): No heartbeat from core client for 30 sec - exiting 15:10:57 (4876): No heartbeat from core client for 30 sec - exiting 15:10:58 (4876): No heartbeat from core client for 30 sec - exiting 15:10:59 (4876): No heartbeat from core client for 30 sec - exiting 15:11:00 (4876): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 15:12:12 (6008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:12:13 (6008): No heartbeat from core client for 30 sec - exiting 15:12:14 (6008): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:33:21 (3456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:34:01 (4704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:34:02 (4704): No heartbeat from core client for 30 sec - exiting 21:34:03 (4704): No heartbeat from core client for 30 sec - exiting 21:34:04 (4704): No heartbeat from core client for 30 sec - exiting 21:34:05 (4704): No heartbeat from core client for 30 sec - exiting 21:34:06 (4704): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 21:36:10 (5984): No heartbeat from core client for 30 sec - exiting 21:36:11 (5984): No heartbeat from core client for 30 sec - exiting 21:36:12 (5984): No heartbeat from core client for 30 sec - exiting 21:36:13 (5984): No heartbeat from core client for 30 sec - exiting 21:36:14 (5984): No heartbeat from core client for 30 sec - exiting 21:36:15 (5984): No heartbeat from core client for 30 sec - exiting 21:36:17 (5984): No heartbeat from core client for 30 sec - exiting 21:36:18 (5984): No heartbeat from core client for 30 sec - exiting 21:36:19 (5984): No heartbeat from core client for 30 sec - exiting 21:36:20 (5984): No heartbeat from core client for 30 sec - exiting 21:36:21 (5984): No heartbeat from core client for 30 sec - exiting 21:36:22 (5984): No heartbeat from core client for 30 sec - exiting 21:36:23 (5984): No heartbeat from core client for 30 sec - exiting 21:36:24 (5984): No heartbeat from core client for 30 sec - exiting 21:36:25 (5984): No heartbeat from core client for 30 sec - exiting 21:36:26 (5984): No heartbeat from core client for 30 sec - exiting 21:36:27 (5984): No heartbeat from core client for 30 sec - exiting 21:36:29 (5984): No heartbeat from core client for 30 sec - exiting 21:36:30 (5984): No heartbeat from core client for 30 sec - exiting 21:36:31 (5984): No heartbeat from core client for 30 sec - exiting 21:36:32 (5984): No heartbeat from core client for 30 sec - exiting 21:36:33 (5984): No heartbeat from core client for 30 sec - exiting 21:36:34 (5984): No heartbeat from core client for 30 sec - exiting 21:36:35 (5984): No heartbeat from core client for 30 sec - exiting 21:36:36 (5984): No heartbeat from core client for 30 sec - exiting 21:36:37 (5984): No heartbeat from core client for 30 sec - exiting 21:36:38 (5984): No heartbeat from core client for 30 sec - exiting 21:36:39 (5984): No heartbeat from core client for 30 sec - exiting 21:37:15 (5984): No heartbeat from core client for 30 sec - exiting 21:37:17 (5984): No heartbeat from core client for 30 sec - exiting 21:37:18 (5984): No heartbeat from core client for 30 sec - exiting 21:37:19 (5984): No heartbeat from core client for 30 sec - exiting 21:37:20 (5984): No heartbeat from core client for 30 sec - exiting 21:37:21 (5984): No heartbeat from core client for 30 sec - exiting 21:37:22 (5984): No heartbeat from core client for 30 sec - exiting 21:37:23 (5984): No heartbeat from core client for 30 sec - exiting 21:37:24 (5984): No heartbeat from core client for 30 sec - exiting 21:37:25 (5984): No heartbeat from core client for 30 sec - exiting 21:37:26 (5984): No heartbeat from core client for 30 sec - exiting 21:37:27 (5984): No heartbeat from core client for 30 sec - exiting 21:37:29 (5984): No heartbeat from core client for 30 sec - exiting 21:37:30 (5984): No heartbeat from core client for 30 sec - exiting 21:37:31 (5984): No heartbeat from core client for 30 sec - exiting 21:37:32 (5984): No heartbeat from core client for 30 sec - exiting 21:37:33 (5984): No heartbeat from core client for 30 sec - exiting 21:37:34 (5984): No heartbeat from core client for 30 sec - exiting 21:37:35 (5984): No heartbeat from core client for 30 sec - exiting 21:37:36 (5984): No heartbeat from core client for 30 sec - exiting 21:37:37 (5984): No heartbeat from core client for 30 sec - exiting 21:37:38 (5984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:37:39 (5984): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/u2urko.pjk0c10 Error converting file to netcdf: dataout/u2urko.pik0c10 Error converting file to netcdf: dataout/u2urko.pfk0c10 Error converting file to netcdf: dataout/u2urko.pck0c10 Error converting file to netcdf: dataout/u2urko.pbk0c10 Error converting file to netcdf: dataout/u2urko.pak0c10 Error converting file to netcdf: dataout/u2urka.phk0c10 Error converting file to netcdf: dataout/u2urka.pgk0c10 Error converting file to netcdf: dataout/u2urka.pek0c10 Error converting file to netcdf: dataout/u2urka.pdk0c10 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/u2urko.pjk0c10 Error converting file to netcdf: dataout/u2urko.pik0c10 Error converting file to netcdf: dataout/u2urko.pfk0c10 Error converting file to netcdf: dataout/u2urko.pck0c10 Error converting file to netcdf: dataout/u2urko.pbk0c10 Error converting file to netcdf: dataout/u2urko.pak0c10 Error converting file to netcdf: dataout/u2urka.phk0c10 Error converting file to netcdf: dataout/u2urka.pgk0c10 Error converting file to netcdf: dataout/u2urka.pek0c10 Error converting file to netcdf: dataout/u2urka.pdk0c10 Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Oct 2011 06:57:46	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	518,400	419,465	0.8092
08 Oct 2011 15:21:31	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	492,480	398,768	0.8097
07 Oct 2011 20:30:31	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	466,560	377,958	0.8101
07 Oct 2011 14:17:25	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	440,640	357,380	0.8110
06 Oct 2011 23:40:28	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	414,720	336,313	0.8109
06 Oct 2011 14:50:38	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	388,800	315,376	0.8112
05 Oct 2011 20:46:25	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	362,880	294,596	0.8118
05 Oct 2011 13:25:02	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	336,960	273,705	0.8123
05 Oct 2011 07:23:30	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	311,040	252,975	0.8133
05 Oct 2011 00:06:15	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	285,120	232,080	0.8140
01 Oct 2011 03:18:16	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	259,200	210,959	0.8139
30 Sep 2011 12:04:09	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	233,280	189,801	0.8136
30 Sep 2011 01:34:25	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	207,360	168,910	0.8146
29 Sep 2011 14:10:36	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	181,440	147,808	0.8146
29 Sep 2011 05:50:42	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	155,520	126,854	0.8157
28 Sep 2011 20:27:18	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	129,600	105,764	0.8161
28 Sep 2011 06:17:55	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	103,680	85,122	0.8210
26 Sep 2011 20:23:12	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	77,760	63,609	0.8180
26 Sep 2011 07:39:05	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	51,840	42,762	0.8249
23 Sep 2011 02:20:59	1165189	13406763	hadcm3n_u2ur_1980_40_007458411_1	25,920	21,347	0.8236