Task 13768464

Name	hadcm3n_t468_1940_40_007614701_1
Workunit	7792831
Created	12 Dec 2011, 15:38:21 UTC
Sent	12 Dec 2011, 15:38:49 UTC
Report deadline	12 Mar 2012, 23:06:00 UTC
Received	31 Jan 2012, 17:38:57 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1021113
Run time	12 days 14 hours 0 min 4 sec
CPU time	11 days 22 hours 37 min 41 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.53 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5796, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6740, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 16:16:56 (5820): No heartbeat from core client for 30 sec - exiting 16:16:57 (5820): No heartbeat from core client for 30 sec - exiting 16:16:58 (5820): No heartbeat from core client for 30 sec - exiting 16:16:59 (5820): No heartbeat from core client for 30 sec - exiting 16:17:00 (5820): No heartbeat from core client for 30 sec - exiting 16:17:01 (5820): No heartbeat from core client for 30 sec - exiting 16:17:02 (5820): No heartbeat from core client for 30 sec - exiting 16:17:03 (5820): No heartbeat from core client for 30 sec - exiting 16:17:04 (5820): No heartbeat from core client for 30 sec - exiting 16:17:05 (5820): No heartbeat from core client for 30 sec - exiting 16:17:06 (5820): No heartbeat from core client for 30 sec - exiting 16:17:07 (5820): No heartbeat from core client for 30 sec - exiting 16:17:08 (5820): No heartbeat from core client for 30 sec - exiting 16:17:09 (5820): No heartbeat from core client for 30 sec - exiting 16:17:10 (5820): No heartbeat from core client for 30 sec - exiting 16:17:11 (5820): No heartbeat from core client for 30 sec - exiting 16:17:12 (5820): No heartbeat from core client for 30 sec - exiting 16:17:13 (5820): No heartbeat from core client for 30 sec - exiting 16:17:14 (5820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6704, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5896, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4200, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4200, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4200, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5960, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 12:33:43 (4212): No heartbeat from core client for 30 sec - exiting 12:33:45 (4212): No heartbeat from core client for 30 sec - exiting 12:33:46 (4212): No heartbeat from core client for 30 sec - exiting 12:33:47 (4212): No heartbeat from core client for 30 sec - exiting 12:33:48 (4212): No heartbeat from core client for 30 sec - exiting 12:33:49 (4212): No heartbeat from core client for 30 sec - exiting 12:33:50 (4212): No heartbeat from core client for 30 sec - exiting 12:33:51 (4212): No heartbeat from core client for 30 sec - exiting 12:33:52 (4212): No heartbeat from core client for 30 sec - exiting 12:33:53 (4212): No heartbeat from core client for 30 sec - exiting 12:33:54 (4212): No heartbeat from core client for 30 sec - exiting 12:33:55 (4212): No heartbeat from core client for 30 sec - exiting 12:33:56 (4212): No heartbeat from core client for 30 sec - exiting 12:33:57 (4212): No heartbeat from core client for 30 sec - exiting 12:33:58 (4212): No heartbeat from core client for 30 sec - exiting 12:33:59 (4212): No heartbeat from core client for 30 sec - exiting 12:34:00 (4212): No heartbeat from core client for 30 sec - exiting 12:34:01 (4212): No heartbeat from core client for 30 sec - exiting 12:34:02 (4212): No heartbeat from core client for 30 sec - exiting 12:34:03 (4212): No heartbeat from core client for 30 sec - exiting 12:34:04 (4212): No heartbeat from core client for 30 sec - exiting 12:34:05 (4212): No heartbeat from core client for 30 sec - exiting 12:34:06 (4212): No heartbeat from core client for 30 sec - exiting 12:34:07 (4212): No heartbeat from core client for 30 sec - exiting 12:34:08 (4212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8048, iMonCtr=1 Model crash detected, will try to restart... 10:40:34 (5772): No heartbeat from core client for 30 sec - exiting 10:40:35 (5772): No heartbeat from core client for 30 sec - exiting 10:40:36 (5772): No heartbeat from core client for 30 sec - exiting 10:40:37 (5772): No heartbeat from core client for 30 sec - exiting 10:40:38 (5772): No heartbeat from core client for 30 sec - exiting 10:40:39 (5772): No heartbeat from core client for 30 sec - exiting 10:40:40 (5772): No heartbeat from core client for 30 sec - exiting 10:40:41 (5772): No heartbeat from core client for 30 sec - exiting 10:40:42 (5772): No heartbeat from core client for 30 sec - exiting 10:40:43 (5772): No heartbeat from core client for 30 sec - exiting 10:40:44 (5772): No heartbeat from core client for 30 sec - exiting 10:40:45 (5772): No heartbeat from core client for 30 sec - exiting 10:40:46 (5772): No heartbeat from core client for 30 sec - exiting 10:40:47 (5772): No heartbeat from core client for 30 sec - exiting 10:40:48 (5772): No heartbeat from core client for 30 sec - exiting 10:40:49 (5772): No heartbeat from core client for 30 sec - exiting 10:40:50 (5772): No heartbeat from core client for 30 sec - exiting 10:40:51 (5772): No heartbeat from core client for 30 sec - exiting 10:40:52 (5772): No heartbeat from core client for 30 sec - exiting 10:40:53 (5772): No heartbeat from core client for 30 sec - exiting 10:40:54 (5772): No heartbeat from core client for 30 sec - exiting 10:40:55 (5772): No heartbeat from core client for 30 sec - exiting 10:40:56 (5772): No heartbeat from core client for 30 sec - exiting 10:40:57 (5772): No heartbeat from core client for 30 sec - exiting 10:40:58 (5772): No heartbeat from core client for 30 sec - exiting 10:40:59 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3476, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1208, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1928, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4300, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5420, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6056, iMonCtr=1 Model crash detected, will try to restart... 09:42:59 (5708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7120, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7052, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5676, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:54:57 (6092): No heartbeat from core client for 30 sec - exiting 16:54:58 (6092): No heartbeat from core client for 30 sec - exiting 16:54:59 (6092): No heartbeat from core client for 30 sec - exiting 16:55:00 (6092): No heartbeat from core client for 30 sec - exiting 16:55:01 (6092): No heartbeat from core client for 30 sec - exiting 16:55:02 (6092): No heartbeat from core client for 30 sec - exiting 16:55:03 (6092): No heartbeat from core client for 30 sec - exiting 16:55:04 (6092): No heartbeat from core client for 30 sec - exiting 16:55:05 (6092): No heartbeat from core client for 30 sec - exiting 16:55:06 (6092): No heartbeat from core client for 30 sec - exiting 16:55:07 (6092): No heartbeat from core client for 30 sec - exiting 16:55:08 (6092): No heartbeat from core client for 30 sec - exiting 16:55:09 (6092): No heartbeat from core client for 30 sec - exiting 16:55:10 (6092): No heartbeat from core client for 30 sec - exiting 16:55:11 (6092): No heartbeat from core client for 30 sec - exiting 16:55:12 (6092): No heartbeat from core client for 30 sec - exiting 16:55:13 (6092): No heartbeat from core client for 30 sec - exiting 16:55:14 (6092): No heartbeat from core client for 30 sec - exiting 16:55:15 (6092): No heartbeat from core client for 30 sec - exiting 16:55:16 (6092): No heartbeat from core client for 30 sec - exiting 16:55:17 (6092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:18 (6092): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4400, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4188, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77373AB3 read attempt to address 0x4036AFD0 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Jan 2012 16:42:09	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	518,400	1,031,850	1.9905
29 Jan 2012 10:09:21	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	492,480	981,599	1.9932
27 Jan 2012 15:16:54	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	466,560	931,147	1.9958
24 Jan 2012 13:33:34	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	440,640	879,040	1.9949
23 Jan 2012 11:50:58	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	414,720	828,067	1.9967
21 Jan 2012 10:46:11	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	388,800	777,582	2.0000
18 Jan 2012 13:55:19	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	362,880	726,963	2.0033
14 Jan 2012 17:29:03	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	336,960	676,426	2.0074
10 Jan 2012 15:45:06	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	311,040	624,708	2.0084
08 Jan 2012 10:55:22	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	285,120	571,867	2.0057
03 Jan 2012 14:57:43	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	259,200	520,476	2.0080
29 Dec 2011 15:46:04	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	233,280	467,929	2.0059
27 Dec 2011 13:43:39	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	207,360	414,866	2.0007
25 Dec 2011 16:25:28	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	181,440	362,419	1.9975
24 Dec 2011 16:21:22	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	155,520	312,566	2.0098
23 Dec 2011 12:42:07	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	129,600	260,683	2.0114
22 Dec 2011 13:09:39	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	103,680	208,106	2.0072
21 Dec 2011 07:02:29	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	77,760	156,773	2.0161
18 Dec 2011 16:29:05	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	51,840	104,841	2.0224
14 Dec 2011 18:49:53	1021113	13768464	hadcm3n_t468_1940_40_007614701_1	25,920	53,262	2.0549