Task 13349046

Name	hadcm3n_t2bf_1940_40_007443088_2
Workunit	7640591
Created	8 Sep 2011, 23:59:32 UTC
Sent	19 Sep 2011, 13:49:52 UTC
Report deadline	19 Dec 2011, 21:17:03 UTC
Received	11 Oct 2011, 14:27:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1021113
Run time	6 days 13 hours 43 min 2 sec
CPU time	6 days 6 hours 52 min 40 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.52 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4340, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3920, iMonCtr=1 Model crash detected, will try to restart... 15:29:16 (4960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6824, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 07:15:48 (6836): No heartbeat from core client for 30 sec - exiting 07:15:49 (6836): No heartbeat from core client for 30 sec - exiting 07:15:50 (6836): No heartbeat from core client for 30 sec - exiting 07:15:51 (6836): No heartbeat from core client for 30 sec - exiting 07:15:52 (6836): No heartbeat from core client for 30 sec - exiting 07:15:53 (6836): No heartbeat from core client for 30 sec - exiting 07:15:54 (6836): No heartbeat from core client for 30 sec - exiting 07:15:55 (6836): No heartbeat from core client for 30 sec - exiting 07:15:56 (6836): No heartbeat from core client for 30 sec - exiting 07:15:57 (6836): No heartbeat from core client for 30 sec - exiting 07:15:58 (6836): No heartbeat from core client for 30 sec - exiting 07:15:59 (6836): No heartbeat from core client for 30 sec - exiting 07:16:00 (6836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1208, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... 08:36:43 (5128): No heartbeat from core client for 30 sec - exiting 08:36:44 (5128): No heartbeat from core client for 30 sec - exiting 08:36:45 (5128): No heartbeat from core client for 30 sec - exiting 08:36:46 (5128): No heartbeat from core client for 30 sec - exiting 08:36:47 (5128): No heartbeat from core client for 30 sec - exiting 08:36:48 (5128): No heartbeat from core client for 30 sec - exiting 08:36:49 (5128): No heartbeat from core client for 30 sec - exiting 08:36:50 (5128): No heartbeat from core client for 30 sec - exiting 08:36:51 (5128): No heartbeat from core client for 30 sec - exiting 08:36:53 (5128): No heartbeat from core client for 30 sec - exiting 08:36:54 (5128): No heartbeat from core client for 30 sec - exiting 08:36:55 (5128): No heartbeat from core client for 30 sec - exiting 08:36:56 (5128): No heartbeat from core client for 30 sec - exiting 08:36:57 (5128): No heartbeat from core client for 30 sec - exiting 08:36:58 (5128): No heartbeat from core client for 30 sec - exiting 08:36:59 (5128): No heartbeat from core client for 30 sec - exiting 08:37:00 (5128): No heartbeat from core client for 30 sec - exiting 08:37:01 (5128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6136, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6820, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=820, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=820, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6000, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4864, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4048, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4852, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4636, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4868, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4868, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4868, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4868, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4868, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4868, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Oct 2011 16:13:41	1021113	13349046	hadcm3n_t2bf_1940_40_007443088_2	259,200	512,196	1.9761
08 Oct 2011 15:26:39	1021113	13349046	hadcm3n_t2bf_1940_40_007443088_2	233,280	461,184	1.9770
07 Oct 2011 15:34:52	1021113	13349046	hadcm3n_t2bf_1940_40_007443088_2	207,360	410,547	1.9799
06 Oct 2011 15:36:33	1021113	13349046	hadcm3n_t2bf_1940_40_007443088_2	181,440	359,815	1.9831
05 Oct 2011 12:13:33	1021113	13349046	hadcm3n_t2bf_1940_40_007443088_2	155,520	308,619	1.9844
02 Oct 2011 07:55:01	1021113	13349046	hadcm3n_t2bf_1940_40_007443088_2	129,600	258,037	1.9910
01 Oct 2011 05:40:42	1021113	13349046	hadcm3n_t2bf_1940_40_007443088_2	103,680	206,616	1.9928
29 Sep 2011 18:06:53	1021113	13349046	hadcm3n_t2bf_1940_40_007443088_2	77,760	154,770	1.9904
27 Sep 2011 15:59:17	1021113	13349046	hadcm3n_t2bf_1940_40_007443088_2	51,840	103,043	1.9877
24 Sep 2011 17:25:29	1021113	13349046	hadcm3n_t2bf_1940_40_007443088_2	25,920	51,213	1.9758