Task 11690310

Name	famous_v09a_799_200_006686108_1
Workunit	6889361
Created	26 Aug 2010, 15:39:28 UTC
Sent	28 Aug 2010, 16:51:00 UTC
Report deadline	28 Nov 2010, 0:18:11 UTC
Received	22 Sep 2010, 13:42:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1095731
Run time	8 days 15 hours 24 min 19 sec
CPU time	8 days 7 hours 28 min 27 sec
Validate state	Invalid
Credit	2,841.20
Device peak FLOPS	1.11 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> (12103): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12103): No heartbeat from core client for 30 sec - exiting (12106): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12115): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12119): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12119): No heartbeat from core client for 30 sec - exiting (12119): No heartbeat from core client for 30 sec - exiting (12119): No heartbeat from core client for 30 sec - exiting (12119): No heartbeat from core client for 30 sec - exiting (12119): No heartbeat from core client for 30 sec - exiting (12119): No heartbeat from core client for 30 sec - exiting (12119): No heartbeat from core client for 30 sec - exiting (12125): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12129): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12133): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12137): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12137): No heartbeat from core client for 30 sec - exiting (12140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12140): No heartbeat from core client for 30 sec - exiting (12140): No heartbeat from core client for 30 sec - exiting (12140): No heartbeat from core client for 30 sec - exiting (12143): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12143): No heartbeat from core client for 30 sec - exiting (12143): No heartbeat from core client for 30 sec - exiting (12147): No heartbeat from core client for 30 sec - exiting (12147): No heartbeat from core client for 30 sec - exiting (12147): No heartbeat from core client for 30 sec - exiting (12147): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12150): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12150): No heartbeat from core client for 30 sec - exiting (12150): No heartbeat from core client for 30 sec - exiting (12150): No heartbeat from core client for 30 sec - exiting (12150): No heartbeat from core client for 30 sec - exiting (12150): No heartbeat from core client for 30 sec - exiting (12150): No heartbeat from core client for 30 sec - exiting (12150): No heartbeat from core client for 30 sec - exiting (12150): No heartbeat from core client for 30 sec - exiting (12150): No heartbeat from core client for 30 sec - exiting (12160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12163): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12412, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12412, iMonCtr=1 Model crash detected, will try to restart... (12412): No heartbeat from core client for 30 sec - exiting (12412): No heartbeat from core client for 30 sec - exiting (12412): No heartbeat from core client for 30 sec - exiting (12412): No heartbeat from core client for 30 sec - exiting (12412): No heartbeat from core client for 30 sec - exiting (12412): No heartbeat from core client for 30 sec - exiting (12412): No heartbeat from core client for 30 sec - exiting (12412): No heartbeat from core client for 30 sec - exiting (12412): No heartbeat from core client for 30 sec - exiting (12412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12435): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12435): No heartbeat from core client for 30 sec - exiting (12443): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 15 received, exiting... (1225): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Sorry, too many model crashes! :-( (1300): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Sep 2010 12:14:58	1095731	11690310	famous_v09a_799_200_006686108_1	861,146	712,851	0.8278
22 Sep 2010 10:03:57	1095731	11690310	famous_v09a_799_200_006686108_1	851,786	705,130	0.8278
22 Sep 2010 07:52:07	1095731	11690310	famous_v09a_799_200_006686108_1	842,426	697,388	0.8278
22 Sep 2010 05:26:44	1095731	11690310	famous_v09a_799_200_006686108_1	833,066	689,650	0.8278
22 Sep 2010 03:18:02	1095731	11690310	famous_v09a_799_200_006686108_1	823,706	681,920	0.8279
22 Sep 2010 03:05:24	1095731	11690310	famous_v09a_799_200_006686108_1	814,346	674,208	0.8279
21 Sep 2010 22:57:07	1095731	11690310	famous_v09a_799_200_006686108_1	804,986	666,494	0.8280
21 Sep 2010 20:44:13	1095731	11690310	famous_v09a_799_200_006686108_1	795,626	658,771	0.8280
21 Sep 2010 18:34:48	1095731	11690310	famous_v09a_799_200_006686108_1	786,266	651,052	0.8280
21 Sep 2010 16:26:13	1095731	11690310	famous_v09a_799_200_006686108_1	776,906	643,321	0.8281
21 Sep 2010 14:14:05	1095731	11690310	famous_v09a_799_200_006686108_1	767,546	635,593	0.8281
21 Sep 2010 09:10:39	1095731	11690310	famous_v09a_799_200_006686108_1	758,186	627,858	0.8281
21 Sep 2010 06:52:58	1095731	11690310	famous_v09a_799_200_006686108_1	748,826	620,117	0.8281
21 Sep 2010 04:37:09	1095731	11690310	famous_v09a_799_200_006686108_1	739,466	612,381	0.8281
21 Sep 2010 03:09:34	1095731	11690310	famous_v09a_799_200_006686108_1	730,106	604,652	0.8282
21 Sep 2010 03:09:34	1095731	11690310	famous_v09a_799_200_006686108_1	720,746	596,919	0.8282
20 Sep 2010 21:52:14	1095731	11690310	famous_v09a_799_200_006686108_1	711,386	589,177	0.8282
20 Sep 2010 19:40:41	1095731	11690310	famous_v09a_799_200_006686108_1	702,026	581,427	0.8282
20 Sep 2010 17:30:25	1095731	11690310	famous_v09a_799_200_006686108_1	692,666	573,681	0.8282
20 Sep 2010 15:13:34	1095731	11690310	famous_v09a_799_200_006686108_1	683,306	565,942	0.8282