Task 13348580

Name	hadcm3n_o188_1940_40_007442861_2
Workunit	7640364
Created	8 Sep 2011, 23:33:39 UTC
Sent	19 Sep 2011, 16:35:19 UTC
Report deadline	20 Dec 2011, 0:02:30 UTC
Received	11 Oct 2011, 14:27:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1021113
Run time	6 days 13 hours 15 min 59 sec
CPU time	6 days 6 hours 28 min 11 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.52 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6324, iMonCtr=1 Model crash detected, will try to restart... 15:29:16 (4976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6832, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6768, iMonCtr=1 Model crash detected, will try to restart... 07:15:48 (6852): No heartbeat from core client for 30 sec - exiting 07:15:49 (6852): No heartbeat from core client for 30 sec - exiting 07:15:50 (6852): No heartbeat from core client for 30 sec - exiting 07:15:51 (6852): No heartbeat from core client for 30 sec - exiting 07:15:52 (6852): No heartbeat from core client for 30 sec - exiting 07:15:53 (6852): No heartbeat from core client for 30 sec - exiting 07:15:54 (6852): No heartbeat from core client for 30 sec - exiting 07:15:55 (6852): No heartbeat from core client for 30 sec - exiting 07:15:56 (6852): No heartbeat from core client for 30 sec - exiting 07:15:57 (6852): No heartbeat from core client for 30 sec - exiting 07:15:58 (6852): No heartbeat from core client for 30 sec - exiting 07:15:59 (6852): No heartbeat from core client for 30 sec - exiting 07:16:00 (6852): No heartbeat from core client for 30 sec - exiting 07:16:01 (6852): No heartbeat from core client for 30 sec - exiting 07:16:02 (6852): No heartbeat from core client for 30 sec - exiting 07:16:03 (6852): No heartbeat from core client for 30 sec - exiting 07:16:04 (6852): No heartbeat from core client for 30 sec - exiting 07:16:05 (6852): No heartbeat from core client for 30 sec - exiting 07:16:06 (6852): No heartbeat from core client for 30 sec - exiting 07:16:07 (6852): No heartbeat from core client for 30 sec - exiting 07:16:08 (6852): No heartbeat from core client for 30 sec - exiting 07:16:09 (6852): No heartbeat from core client for 30 sec - exiting 07:16:10 (6852): No heartbeat from core client for 30 sec - exiting 07:16:11 (6852): No heartbeat from core client for 30 sec - exiting 07:16:12 (6852): No heartbeat from core client for 30 sec - exiting 07:16:13 (6852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:16:14 (6852): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2692, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1564, iMonCtr=1 Model crash detected, will try to restart... 08:36:43 (5140): No heartbeat from core client for 30 sec - exiting 08:36:44 (5140): No heartbeat from core client for 30 sec - exiting 08:36:45 (5140): No heartbeat from core client for 30 sec - exiting 08:36:46 (5140): No heartbeat from core client for 30 sec - exiting 08:36:47 (5140): No heartbeat from core client for 30 sec - exiting 08:36:48 (5140): No heartbeat from core client for 30 sec - exiting 08:36:49 (5140): No heartbeat from core client for 30 sec - exiting 08:36:50 (5140): No heartbeat from core client for 30 sec - exiting 08:36:51 (5140): No heartbeat from core client for 30 sec - exiting 08:36:53 (5140): No heartbeat from core client for 30 sec - exiting 08:36:54 (5140): No heartbeat from core client for 30 sec - exiting 08:36:55 (5140): No heartbeat from core client for 30 sec - exiting 08:36:56 (5140): No heartbeat from core client for 30 sec - exiting 08:36:57 (5140): No heartbeat from core client for 30 sec - exiting 08:36:58 (5140): No heartbeat from core client for 30 sec - exiting 08:36:59 (5140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3296, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2340, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2340, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4784, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4924, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5280, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4864, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4648, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5160, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4916, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4916, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4916, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4916, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4916, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4916, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Oct 2011 15:12:46	1021113	13348580	hadcm3n_o188_1940_40_007442861_2	259,200	507,642	1.9585
08 Oct 2011 14:25:36	1021113	13348580	hadcm3n_o188_1940_40_007442861_2	233,280	457,001	1.9590
07 Oct 2011 14:33:03	1021113	13348580	hadcm3n_o188_1940_40_007442861_2	207,360	406,866	1.9621
06 Oct 2011 14:35:21	1021113	13348580	hadcm3n_o188_1940_40_007442861_2	181,440	356,707	1.9660
05 Oct 2011 12:13:33	1021113	13348580	hadcm3n_o188_1940_40_007442861_2	155,520	305,698	1.9657
02 Oct 2011 07:55:01	1021113	13348580	hadcm3n_o188_1940_40_007442861_2	129,600	255,720	1.9731
01 Oct 2011 05:40:42	1021113	13348580	hadcm3n_o188_1940_40_007442861_2	103,680	204,817	1.9755
30 Sep 2011 04:22:02	1021113	13348580	hadcm3n_o188_1940_40_007442861_2	77,760	153,321	1.9717
27 Sep 2011 17:00:59	1021113	13348580	hadcm3n_o188_1940_40_007442861_2	51,840	102,018	1.9679
25 Sep 2011 05:48:54	1021113	13348580	hadcm3n_o188_1940_40_007442861_2	25,920	50,712	1.9565