Task 13378461

Name	hadcm3n_o67x_1980_40_007451146_3
Workunit	7648649
Created	12 Sep 2011, 13:21:59 UTC
Sent	12 Sep 2011, 13:41:47 UTC
Report deadline	12 Dec 2011, 21:08:58 UTC
Received	11 Oct 2011, 14:27:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1021113
Run time	7 days 15 hours 45 min 6 sec
CPU time	7 days 7 hours 33 min 51 sec
Validate state	Invalid
Credit	3,732.48
Device peak FLOPS	2.47 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4000, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6864, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1864, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1864, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4216, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4224, iMonCtr=1 Model crash detected, will try to restart... 15:29:15 (6428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2720, iMonCtr=1 Model crash detected, will try to restart... 15:29:16 (4952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6816, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 07:15:48 (6460): No heartbeat from core client for 30 sec - exiting 07:15:49 (6460): No heartbeat from core client for 30 sec - exiting 07:15:50 (6460): No heartbeat from core client for 30 sec - exiting 07:15:51 (6460): No heartbeat from core client for 30 sec - exiting 07:15:52 (6460): No heartbeat from core client for 30 sec - exiting 07:15:53 (6460): No heartbeat from core client for 30 sec - exiting 07:15:54 (6460): No heartbeat from core client for 30 sec - exiting 07:15:55 (6460): No heartbeat from core client for 30 sec - exiting 07:15:56 (6460): No heartbeat from core client for 30 sec - exiting 07:15:57 (6460): No heartbeat from core client for 30 sec - exiting 07:15:58 (6460): No heartbeat from core client for 30 sec - exiting 07:15:59 (6460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1948, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3604, iMonCtr=1 Model crash detected, will try to restart... 08:36:43 (3696): No heartbeat from core client for 30 sec - exiting 08:36:44 (3696): No heartbeat from core client for 30 sec - exiting 08:36:45 (3696): No heartbeat from core client for 30 sec - exiting 08:36:46 (3696): No heartbeat from core client for 30 sec - exiting 08:36:47 (3696): No heartbeat from core client for 30 sec - exiting 08:36:48 (3696): No heartbeat from core client for 30 sec - exiting 08:36:49 (3696): No heartbeat from core client for 30 sec - exiting 08:36:50 (3696): No heartbeat from core client for 30 sec - exiting 08:36:51 (3696): No heartbeat from core client for 30 sec - exiting 08:36:53 (3696): No heartbeat from core client for 30 sec - exiting 08:36:54 (3696): No heartbeat from core client for 30 sec - exiting 08:36:55 (3696): No heartbeat from core client for 30 sec - exiting 08:36:56 (3696): No heartbeat from core client for 30 sec - exiting 08:36:57 (3696): No heartbeat from core client for 30 sec - exiting 08:36:58 (3696): No heartbeat from core client for 30 sec - exiting 08:36:59 (3696): No heartbeat from core client for 30 sec - exiting 08:37:00 (3696): No heartbeat from core client for 30 sec - exiting 08:37:01 (3696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6128, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6812, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1688, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1688, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5992, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4856, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4052, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5256, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4824, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4624, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4848, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4848, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4848, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4848, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4848, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4848, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Oct 2011 18:34:29	1021113	13378461	hadcm3n_o67x_1980_40_007451146_3	311,040	609,341	1.9590
08 Oct 2011 17:36:30	1021113	13378461	hadcm3n_o67x_1980_40_007451146_3	285,120	558,462	1.9587
07 Oct 2011 17:40:05	1021113	13378461	hadcm3n_o67x_1980_40_007451146_3	259,200	508,013	1.9599
06 Oct 2011 17:27:33	1021113	13378461	hadcm3n_o67x_1980_40_007451146_3	233,280	457,578	1.9615
05 Oct 2011 14:10:43	1021113	13378461	hadcm3n_o67x_1980_40_007451146_3	207,360	406,767	1.9616
02 Oct 2011 15:11:24	1021113	13378461	hadcm3n_o67x_1980_40_007451146_3	181,440	356,498	1.9648
01 Oct 2011 07:37:37	1021113	13378461	hadcm3n_o67x_1980_40_007451146_3	155,520	305,896	1.9669
30 Sep 2011 06:44:13	1021113	13378461	hadcm3n_o67x_1980_40_007451146_3	129,600	253,991	1.9598
28 Sep 2011 13:23:56	1021113	13378461	hadcm3n_o67x_1980_40_007451146_3	103,680	202,614	1.9542
25 Sep 2011 17:33:02	1021113	13378461	hadcm3n_o67x_1980_40_007451146_3	77,760	151,034	1.9423
23 Sep 2011 19:51:24	1021113	13378461	hadcm3n_o67x_1980_40_007451146_3	51,840	100,558	1.9398
21 Sep 2011 11:13:06	1021113	13378461	hadcm3n_o67x_1980_40_007451146_3	25,920	50,221	1.9375