Task 13363293

Name	hadcm3n_o3eb_1940_40_007449294_1
Workunit	7646797
Created	9 Sep 2011, 23:45:31 UTC
Sent	9 Sep 2011, 23:50:56 UTC
Report deadline	10 Dec 2011, 7:18:07 UTC
Received	25 Nov 2011, 1:26:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1167504
Run time	8 days 10 hours 54 min 41 sec
CPU time	7 days 7 hours 54 min 54 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.83 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4160, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6740, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3948, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o3ebko.pje1c10 Error converting file to netcdf: dataout/o3ebko.pie1c10 Error converting file to netcdf: dataout/o3ebko.pfe1c10 Error converting file to netcdf: dataout/o3ebka.phe1c10 Error converting file to netcdf: dataout/o3ebka.pge1c10 Error converting file to netcdf: dataout/o3ebka.pee1c10 Error converting file to netcdf: dataout/o3ebka.pde1c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4380, iMonCtr=1 Model crash detected, will try to restart... 10:00:55 (4176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:15:56 (4984): No heartbeat from core client for 30 sec - exiting 01:15:57 (4984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:53:01 (4436): No heartbeat from core client for 30 sec - exiting 10:53:02 (4436): No heartbeat from core client for 30 sec - exiting 10:53:03 (4436): No heartbeat from core client for 30 sec - exiting 10:53:04 (4436): No heartbeat from core client for 30 sec - exiting 10:53:05 (4436): No heartbeat from core client for 30 sec - exiting 10:53:06 (4436): No heartbeat from core client for 30 sec - exiting 10:53:07 (4436): No heartbeat from core client for 30 sec - exiting 10:53:08 (4436): No heartbeat from core client for 30 sec - exiting 10:53:09 (4436): No heartbeat from core client for 30 sec - exiting 10:53:10 (4436): No heartbeat from core client for 30 sec - exiting 10:53:11 (4436): No heartbeat from core client for 30 sec - exiting 10:53:12 (4436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:37:02 (5824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1972, iMonCtr=1 Model crash detected, will try to restart... 07:10:26 (3224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5796, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4108, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4232, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1316, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o3ebko.pjf6c10 Error converting file to netcdf: dataout/o3ebko.pif6c10 Error converting file to netcdf: dataout/o3ebko.pff6c10 Error converting file to netcdf: dataout/o3ebka.phf6c10 Error converting file to netcdf: dataout/o3ebka.pgf6c10 Error converting file to netcdf: dataout/o3ebka.pef6c10 Error converting file to netcdf: dataout/o3ebka.pdf6c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1388, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2188, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8104, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x774B3F79 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5884, selfPID=5884, iMonCtr=1 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x774B3A93 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_o3eb_1940_40_007449294/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Nov 2011 16:08:33	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	518,400	633,272	1.2216
22 Nov 2011 14:50:26	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	492,480	598,155	1.2146
17 Nov 2011 22:31:13	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	466,560	566,872	1.2150
17 Nov 2011 07:07:52	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	440,640	536,076	1.2166
16 Nov 2011 02:27:06	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	414,720	505,080	1.2179
16 Nov 2011 02:27:06	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	388,800	474,145	1.2195
16 Nov 2011 02:27:06	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	362,880	442,942	1.2206
16 Nov 2011 02:27:06	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	336,960	411,832	1.2222
16 Nov 2011 02:27:06	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	311,040	380,522	1.2234
08 Nov 2011 15:00:15	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	285,120	349,075	1.2243
07 Nov 2011 13:44:03	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	259,200	318,197	1.2276
06 Nov 2011 19:34:49	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	233,280	286,655	1.2288
04 Nov 2011 03:03:35	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	207,360	255,670	1.2330
02 Nov 2011 19:06:42	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	181,440	225,001	1.2401
02 Nov 2011 03:02:49	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	155,520	193,876	1.2466
01 Nov 2011 18:54:32	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	129,600	163,557	1.2620
31 Oct 2011 22:32:06	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	103,680	132,744	1.2803
31 Oct 2011 19:46:13	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	77,760	101,068	1.2997
23 Sep 2011 10:35:41	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	51,840	67,526	1.3026
21 Sep 2011 20:23:53	1167504	13363293	hadcm3n_o3eb_1940_40_007449294_1	25,920	33,569	1.2951