Task 15602658

Name	hadcm3n_n7ci_1880_40_008284732_1
Workunit	8435867
Created	9 Feb 2013, 14:53:53 UTC
Sent	9 Feb 2013, 15:00:52 UTC
Report deadline	11 May 2013, 22:28:03 UTC
Received	22 Mar 2013, 19:49:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-529697949 (0xE06D7363) Unknown error code
Computer ID	1187110
Run time	7 days 23 hours 55 min 34 sec
CPU time	7 days 9 hours 46 min 4 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.32 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code -529697949 (0xe06d7363) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 15:31:17 (5496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:33:47 (5580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:49:53 (6904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:57:21 (3264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:04:07 (5920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:09:18 (5868): No heartbeat from core client for 30 sec - exiting 21:09:19 (5868): No heartbeat from core client for 30 sec - exiting 21:09:20 (5868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2748, iMonCtr=1 Model crash detected, will try to restart... 19:25:14 (5744): No heartbeat from core client for 30 sec - exiting 19:25:15 (5744): No heartbeat from core client for 30 sec - exiting 19:25:16 (5744): No heartbeat from core client for 30 sec - exiting 19:25:17 (5744): No heartbeat from core client for 30 sec - exiting 19:25:18 (5744): No heartbeat from core client for 30 sec - exiting 19:25:19 (5744): No heartbeat from core client for 30 sec - exiting 19:25:20 (5744): No heartbeat from core client for 30 sec - exiting 19:25:21 (5744): No heartbeat from core client for 30 sec - exiting 19:25:22 (5744): No heartbeat from core client for 30 sec - exiting 19:25:23 (5744): No heartbeat from core client for 30 sec - exiting 19:25:24 (5744): No heartbeat from core client for 30 sec - exiting 19:25:25 (5744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5716, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5704, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5604, iMonCtr=1 Model crash detected, will try to restart... 19:15:42 (5544): No heartbeat from core client for 30 sec - exiting 19:15:43 (5544): No heartbeat from core client for 30 sec - exiting 19:15:44 (5544): No heartbeat from core client for 30 sec - exiting 19:15:45 (5544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6576, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5408, iMonCtr=1 Model crash detected, will try to restart... 19:46:19 (5664): No heartbeat from core client for 30 sec - exiting 19:46:20 (5664): No heartbeat from core client for 30 sec - exiting 19:46:21 (5664): No heartbeat from core client for 30 sec - exiting 19:46:22 (5664): No heartbeat from core client for 30 sec - exiting 19:46:23 (5664): No heartbeat from core client for 30 sec - exiting 19:46:24 (5664): No heartbeat from core client for 30 sec - exiting 19:46:25 (5664): No heartbeat from core client for 30 sec - exiting 19:46:26 (5664): No heartbeat from core client for 30 sec - exiting 19:46:27 (5664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:54:46 (5404): No heartbeat from core client for 30 sec - exiting 21:54:47 (5404): No heartbeat from core client for 30 sec - exiting 21:54:48 (5404): No heartbeat from core client for 30 sec - exiting 21:54:49 (5404): No heartbeat from core client for 30 sec - exiting 21:54:50 (5404): No heartbeat from core client for 30 sec - exiting 21:54:51 (5404): No heartbeat from core client for 30 sec - exiting 21:54:52 (5404): No heartbeat from core client for 30 sec - exiting 21:54:53 (5404): No heartbeat from core client for 30 sec - exiting 21:54:54 (5404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6140, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5584, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5716, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5668, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5620, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5608, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:01:26 (5280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:15:29 (5456): No heartbeat from core client for 30 sec - exiting 22:15:30 (5456): No heartbeat from core client for 30 sec - exiting 22:15:31 (5456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:18:58 (5620): No heartbeat from core client for 30 sec - exiting 21:18:59 (5620): No heartbeat from core client for 30 sec - exiting 21:19:00 (5620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:57:09 (5660): No heartbeat from core client for 30 sec - exiting 21:57:10 (5660): No heartbeat from core client for 30 sec - exiting 21:57:11 (5660): No heartbeat from core client for 30 sec - exiting 21:57:12 (5660): No heartbeat from core client for 30 sec - exiting 21:57:13 (5660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:16:27 (324): No heartbeat from core client for 30 sec - exiting 22:16:28 (324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:19:17 (5544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:27:29 (5884): No heartbeat from core client for 30 sec - exiting 22:27:30 (5884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:41:31 (5220): No heartbeat from core client for 30 sec - exiting 00:41:32 (5220): No heartbeat from core client for 30 sec - exiting 00:41:33 (5220): No heartbeat from core client for 30 sec - exiting 00:41:34 (5220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:49 (6368): No heartbeat from core client for 30 sec - exiting 13:12:50 (6368): No heartbeat from core client for 30 sec - exiting 13:12:51 (6368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2108, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5220, iMonCtr=1 Model crash detected, will try to restart... 20:51:09 (5744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5912, iMonCtr=1 Model crash detected, will try to restart... 19:56:17 (6544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:50:12 (5692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:46:05 (5984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:12:07 (4248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:16:53 (6112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=728, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5720, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 00:37:00 (6960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:05:55 (6500): No heartbeat from core client for 30 sec - exiting 01:05:56 (6500): No heartbeat from core client for 30 sec - exiting 01:05:57 (6500): No heartbeat from core client for 30 sec - exiting 01:05:58 (6500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:27:55 (6596): No heartbeat from core client for 30 sec - exiting 23:27:56 (6596): No heartbeat from core client for 30 sec - exiting 23:27:57 (6596): No heartbeat from core client for 30 sec - exiting 23:27:58 (6596): No heartbeat from core client for 30 sec - exiting 23:27:59 (6596): No heartbeat from core client for 30 sec - exiting 23:28:00 (6596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:39:01 (6168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77BE3AB3 read attempt to address 0x40E79A20 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77973AB3 read attempt to address 0x40E79A20 Engaging BOINC Windows Runtime Debugger... Cannot serialize file E:\BOINC/projects/climateprediction.net/hadcm3n_n7ci_1880_40_008284732/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77973792 read attempt to address 0x40E79A20 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Mar 2013 21:49:49	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	518,400	633,937	1.2229
18 Mar 2013 23:16:39	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	492,480	603,275	1.2250
18 Mar 2013 13:45:09	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	466,560	572,151	1.2263
15 Mar 2013 18:37:43	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	440,640	540,920	1.2276
14 Mar 2013 17:31:19	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	414,720	508,980	1.2273
11 Mar 2013 19:35:02	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	388,800	477,241	1.2275
09 Mar 2013 12:43:10	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	362,880	445,450	1.2275
05 Mar 2013 20:16:09	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	336,960	413,498	1.2271
01 Mar 2013 20:41:56	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	311,040	381,828	1.2276
26 Feb 2013 22:14:00	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	285,120	349,168	1.2246
24 Feb 2013 16:01:27	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	259,200	316,639	1.2216
23 Feb 2013 12:27:54	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	233,280	284,557	1.2198
22 Feb 2013 11:17:29	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	207,360	252,917	1.2197
19 Feb 2013 19:47:48	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	181,440	221,060	1.2184
17 Feb 2013 17:00:57	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	155,520	189,457	1.2182
16 Feb 2013 22:24:43	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	129,600	158,000	1.2191
15 Feb 2013 23:11:49	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	103,680	126,234	1.2175
12 Feb 2013 23:49:25	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	77,760	94,540	1.2158
12 Feb 2013 14:10:48	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	51,840	62,899	1.2133
11 Feb 2013 15:50:51	1187110	15602658	hadcm3n_n7ci_1880_40_008284732_1	25,920	31,474	1.2143