Task 12815130

Name	hadcm3n_p00l_1900_40_007217605_0
Workunit	7415845
Created	26 Apr 2011, 15:12:47 UTC
Sent	6 May 2011, 6:53:27 UTC
Report deadline	5 Aug 2011, 14:20:38 UTC
Received	24 Jul 2011, 14:32:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1415839
Run time	27 days 5 hours 57 min 18 sec
CPU time	26 days 13 hours 41 min 36 sec
Validate state	Invalid
Credit	11,197.44
Device peak FLOPS	2.13 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:58:45 (11913): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:58:59 (11913): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 20:18:27 (20263): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:18:29 (20263): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:00:39 (1734): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 16:00:53 (1734): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:53:55 (17668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:24:11 (18614): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:16 (966): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:16:17 (998): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:19:13 (19214): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:19:22 (19250): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:19:26 (19250): No heartbeat from core client for 30 sec - exiting 19:24:06 (357): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:24:08 (357): No heartbeat from core client for 30 sec - exiting 19:24:09 (357): No heartbeat from core client for 30 sec - exiting 19:24:10 (357): No heartbeat from core client for 30 sec - exiting 19:24:11 (357): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 22:45:12 (443): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:47:31 (5642): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:49:46 (5663): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:04:43 (5724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:05:36 (5967): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:08:32 (6008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:09:35 (6060): No heartbeat from core client for 30 sec - exiting 23:09:36 (6060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:12:56 (6106): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:17:27 (6123): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:17:29 (6123): No heartbeat from core client for 30 sec - exiting 23:23:49 (6239): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:24:59 (6348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:28:50 (6391): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:28:52 (6391): No heartbeat from core client for 30 sec - exiting 23:28:53 (6391): No heartbeat from core client for 30 sec - exiting 23:30:43 (6437): No heartbeat from core client for 30 sec - exiting 23:30:49 (6437): No heartbeat from core client for 30 sec - exiting 23:30:50 (6437): No heartbeat from core client for 30 sec - exiting 23:30:51 (6437): No heartbeat from core client for 30 sec - exiting 23:30:52 (6437): No heartbeat from core client for 30 sec - exiting 23:30:53 (6437): No heartbeat from core client for 30 sec - exiting 23:30:54 (6437): No heartbeat from core client for 30 sec - exiting 23:30:55 (6437): No heartbeat from core client for 30 sec - exiting 23:30:56 (6437): No heartbeat from core client for 30 sec - exiting 23:30:57 (6437): No heartbeat from core client for 30 sec - exiting 23:30:58 (6437): No heartbeat from core client for 30 sec - exiting 23:30:59 (6437): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:46:59 (1635): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:11:56 (1723): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:48:37 (17290): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 10:48:48 (17290): No heartbeat from core client for 30 sec - exiting 10:48:49 (17290): No heartbeat from core client for 30 sec - exiting 22:28:02 (14648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:01:45 (14703): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:01:46 (14703): No heartbeat from core client for 30 sec - exiting 00:01:47 (14703): No heartbeat from core client for 30 sec - exiting 00:04:40 (14895): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:07:01 (14919): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 08:11:42 (17521): No heartbeat from core client for 30 sec - exiting 08:11:43 (17521): No heartbeat from core client for 30 sec - exiting 08:11:44 (17521): No heartbeat from core client for 30 sec - exiting 08:11:45 (17521): No heartbeat from core client for 30 sec - exiting 08:11:46 (17521): No heartbeat from core client for 30 sec - exiting 08:11:47 (17521): No heartbeat from core client for 30 sec - exiting 08:11:48 (17521): No heartbeat from core client for 30 sec - exiting 08:11:49 (17521): No heartbeat from core client for 30 sec - exiting 08:11:50 (17521): No heartbeat from core client for 30 sec - exiting 08:11:51 (17521): No heartbeat from core client for 30 sec - exiting 08:11:52 (17521): No heartbeat from core client for 30 sec - exiting 08:11:53 (17521): No heartbeat from core client for 30 sec - exiting 08:11:54 (17521): No heartbeat from core client for 30 sec - exiting 08:11:55 (17521): No heartbeat from core client for 30 sec - exiting 08:11:56 (17521): No heartbeat from core client for 30 sec - exiting 08:11:57 (17521): No heartbeat from core client for 30 sec - exiting 08:11:58 (17521): No heartbeat from core client for 30 sec - exiting 08:11:59 (17521): No heartbeat from core client for 30 sec - exiting 08:12:00 (17521): No heartbeat from core client for 30 sec - exiting 08:12:01 (17521): No heartbeat from core client for 30 sec - exiting 08:12:02 (17521): No heartbeat from core client for 30 sec - exiting 08:12:03 (17521): No heartbeat from core client for 30 sec - exiting 08:12:04 (17521): No heartbeat from core client for 30 sec - exiting 08:12:05 (17521): No heartbeat from core client for 30 sec - exiting 08:12:06 (17521): No heartbeat from core client for 30 sec - exiting 08:12:07 (17521): No heartbeat from core client for 30 sec - exiting 08:12:08 (17521): No heartbeat from core client for 30 sec - exiting 08:12:09 (17521): No heartbeat from core client for 30 sec - exiting 08:12:18 (17521): No heartbeat from core client for 30 sec - exiting 08:12:19 (17521): No heartbeat from core client for 30 sec - exiting 08:12:20 (17521): No heartbeat from core client for 30 sec - exiting 08:12:21 (17521): No heartbeat from core client for 30 sec - exiting 08:12:22 (17521): No heartbeat from core client for 30 sec - exiting 08:12:23 (17521): No heartbeat from core client for 30 sec - exiting 08:12:24 (17521): No heartbeat from core client for 30 sec - exiting 08:12:25 (17521): No heartbeat from core client for 30 sec - exiting 08:12:26 (17521): No heartbeat from core client for 30 sec - exiting 08:12:27 (17521): No heartbeat from core client for 30 sec - exiting 08:12:28 (17521): No heartbeat from core client for 30 sec - exiting 08:12:29 (17521): No heartbeat from core client for 30 sec - exiting 08:12:30 (17521): No heartbeat from core client for 30 sec - exiting 08:12:31 (17521): No heartbeat from core client for 30 sec - exiting 08:12:32 (17521): No heartbeat from core client for 30 sec - exiting 08:12:33 (17521): No heartbeat from core client for 30 sec - exiting 08:12:34 (17521): No heartbeat from core client for 30 sec - exiting 08:12:35 (17521): No heartbeat from core client for 30 sec - exiting 08:12:36 (17521): No heartbeat from core client for 30 sec - exiting 08:12:37 (17521): No heartbeat from core client for 30 sec - exiting 08:12:38 (17521): No heartbeat from core client for 30 sec - exiting 08:12:39 (17521): No heartbeat from core client for 30 sec - exiting 08:12:40 (17521): No heartbeat from core client for 30 sec - exiting 08:12:41 (17521): No heartbeat from core client for 30 sec - exiting 08:12:42 (17521): No heartbeat from core client for 30 sec - exiting 08:12:43 (17521): No heartbeat from core client for 30 sec - exiting 08:12:44 (17521): No heartbeat from core client for 30 sec - exiting 08:12:45 (17521): No heartbeat from core client for 30 sec - exiting 08:12:46 (17521): No heartbeat from core client for 30 sec - exiting 08:12:47 (17521): No heartbeat from core client for 30 sec - exiting 08:12:48 (17521): No heartbeat from core client for 30 sec - exiting 08:12:49 (17521): No heartbeat from core client for 30 sec - exiting 08:12:50 (17521): No heartbeat from core client for 30 sec - exiting 08:12:51 (17521): No heartbeat from core client for 30 sec - exiting 08:12:52 (17521): No heartbeat from core client for 30 sec - exiting 08:12:53 (17521): No heartbeat from core client for 30 sec - exiting 08:12:54 (17521): No heartbeat from core client for 30 sec - exiting 08:12:55 (17521): No heartbeat from core client for 30 sec - exiting 08:12:56 (17521): No heartbeat from core client for 30 sec - exiting 08:12:57 (17521): No heartbeat from core client for 30 sec - exiting 08:12:58 (17521): No heartbeat from core client for 30 sec - exiting 08:12:59 (17521): No heartbeat from core client for 30 sec - exiting 08:13:00 (17521): No heartbeat from core client for 30 sec - exiting 08:13:01 (17521): No heartbeat from core client for 30 sec - exiting 08:13:02 (17521): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:19:13 (17550): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:28:55 (17599): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Stack trace (10 frames): /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf771d400] [0xf771d425] /lib32/libc.so.6(gsignal+0x51)[0xf7587ea1] /lib32/libc.so.6(abort+0x17e)[0xf758b2ce] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf7573e37] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17639, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf776c400] [0xf776c425] /lib32/libc.so.6(gsignal+0x51)[0xf75d6ea1] /lib32/libc.so.6(abort+0x17e)[0xf75da2ce] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf75c2e37] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17639, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77a0400] [0xf77a0425] /lib32/libc.so.6(gsignal+0x51)[0xf760aea1] /lib32/libc.so.6(abort+0x17e)[0xf760e2ce] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf75f6e37] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17639, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7752400] [0xf7752425] /lib32/libc.so.6(gsignal+0x51)[0xf75bcea1] /lib32/libc.so.6(abort+0x17e)[0xf75c02ce] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf75a8e37] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17639, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf775c400] [0xf775c425] /lib32/libc.so.6(gsignal+0x51)[0xf75c6ea1] /lib32/libc.so.6(abort+0x17e)[0xf75ca2ce] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf75b2e37] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17639, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7771400] [0xf7771425] /lib32/libc.so.6(gsignal+0x51)[0xf75dbea1] /lib32/libc.so.6(abort+0x17e)[0xf75df2ce] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf75c7e37] /home/m/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17639, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 18:20:35	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	933,120	2,231,898	2.3919
25 Jul 2011 17:27:22	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	907,200	2,151,679	2.3718
25 Jul 2011 15:43:03	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	881,280	2,070,961	2.3499
25 Jul 2011 15:40:53	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	855,360	1,994,905	2.3322
25 Jul 2011 15:40:53	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	829,440	1,915,328	2.3092
25 Jul 2011 15:40:53	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	803,520	1,835,952	2.2849
10 Jul 2011 20:48:48	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	777,600	1,756,522	2.2589
07 Jul 2011 21:09:31	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	751,680	1,677,810	2.2321
07 Jul 2011 15:37:09	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	725,760	1,598,802	2.2029
05 Jul 2011 07:13:15	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	699,840	1,520,862	2.1732
04 Jul 2011 09:43:57	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	673,920	1,449,319	2.1506
01 Jun 2011 06:20:01	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	648,000	1,376,118	2.1236
31 May 2011 11:39:34	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	622,080	1,311,491	2.1082
30 May 2011 17:07:56	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	596,160	1,250,993	2.0984
30 May 2011 00:41:24	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	570,240	1,199,707	2.1039
29 May 2011 06:12:14	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	544,320	1,147,189	2.1076
28 May 2011 15:21:30	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	518,400	1,096,024	2.1142
27 May 2011 20:40:23	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	492,480	1,037,657	2.1070
27 May 2011 05:34:57	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	466,560	983,920	2.1089
26 May 2011 16:12:53	1147525	12815130	hadcm3n_p00l_1900_40_007217605_0	440,640	936,651	2.1257