Task 15007829

Name	hadcm3n_yhl0_1980_40_008049597_4
Workunit	8204711
Created	26 Jul 2012, 23:01:25 UTC
Sent	26 Jul 2012, 23:02:01 UTC
Report deadline	26 Oct 2012, 6:29:12 UTC
Received	3 Nov 2012, 7:50:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1047400
Run time	23 days 12 hours 4 min 53 sec
CPU time	20 days 10 hours 15 min 3 sec
Validate state	Invalid
Credit	7,776.00
Device peak FLOPS	0.94 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:06:17 (1682): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:09:10 (18756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:17:36 (19013): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:26:27 (19102): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:28:23 (19575): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:32:30 (19807): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:46:31 (20092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc/projects/climateprediction.net/hadcm3n_yhl0_1980_40_008049597/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08451069 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0836D10D Unknown Unknown Unknown hadcm3n_um_6.07_i 082EB086 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F66D Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 4D1246B3 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20522, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc/projects/climateprediction.net/hadcm3n_yhl0_1980_40_008049597/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08451069 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0836D10D Unknown Unknown Unknown hadcm3n_um_6.07_i 082EB086 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F66D Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 4D1246B3 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20522, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc/projects/climatepCalled boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Nov 2012 23:48:31	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	648,000	1,748,024	2.6976
31 Oct 2012 06:03:29	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	622,080	1,672,538	2.6886
29 Oct 2012 15:32:30	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	596,160	1,597,094	2.6790
27 Oct 2012 22:42:53	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	570,240	1,521,983	2.6690
26 Oct 2012 23:17:49	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	544,320	1,445,708	2.6560
26 Oct 2012 02:52:29	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	518,400	1,378,238	2.6586
25 Oct 2012 11:55:20	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	492,480	1,325,030	2.6905
24 Oct 2012 21:08:28	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	466,560	1,272,110	2.7266
24 Oct 2012 06:01:09	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	440,640	1,219,086	2.7666
23 Oct 2012 15:08:59	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	414,720	1,166,140	2.8119
23 Oct 2012 00:51:22	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	388,800	1,113,241	2.8633
22 Oct 2012 08:37:43	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	362,880	1,058,226	2.9162
21 Oct 2012 12:55:25	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	336,960	988,325	2.9331
20 Oct 2012 00:25:12	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	311,040	916,590	2.9469
19 Oct 2012 03:10:41	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	285,120	843,034	2.9568
18 Oct 2012 06:16:38	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	259,200	770,363	2.9721
17 Oct 2012 09:24:59	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	233,280	696,582	2.9860
16 Oct 2012 11:25:06	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	207,360	619,162	2.9859
15 Oct 2012 13:26:19	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	181,440	541,642	2.9852
20 Aug 2012 12:55:18	1047400	15007829	hadcm3n_yhl0_1980_40_008049597_4	155,520	464,309	2.9855