Task 16572626

Name	hadam3pm2_e8ft_1991_10_008713980_0
Workunit	8860454
Created	19 Apr 2014, 9:23:24 UTC
Sent	29 Apr 2014, 17:47:33 UTC
Report deadline	11 Apr 2015, 23:07:33 UTC
Received	18 Jul 2014, 17:19:55 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	9 (0x00000009) Unknown error code
Computer ID	1320181
Run time	36 days 22 hours 30 min 21 sec
CPU time	19 days 15 hours 35 min 24 sec
Validate state	Invalid
Credit	8,484.28
Device peak FLOPS	1.83 GFLOPS
Application version	UK Met Office HadAM3P (global only) with MOSES II landsurface scheme v7.03 i686-apple-darwin
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> process exited with code 9 (0x9, -247) </message> <stderr_txt> oa.pc\|1dec.nc oa.pc\|2jan.nc oa.pc\|2feb.nc oa.pc\|2mar.nc oa.pc\|2apr.nc oa.pc\|2may.nc oa.pc\|2jun.nc oa.pc\|2jul.nc oa.pc\|2aug.nc oa.pc\|2sep.nc oa.pc\|2oct.nc oa.pc\|2nov.nc oa.pe\|1dec.nc oa.pe\|2jan.nc oa.pe\|2feb.nc oa.pe\|2mar.nc oa.pe\|2apr.nc oa.pe\|2may.nc oa.pe\|2jun.nc oa.pe\|2jul.nc oa.pe\|2aug.nc oa.pe\|2sep.nc oa.pe\|2oct.nc oa.pe\|2nov.nc oa.pc\|2dec.nc oa.pc\|3jan.nc oa.pc\|3feb.nc oa.pc\|3mar.nc oa.pc\|3apr.nc oa.pc\|3may.nc oa.pc\|3jun.nc oa.pc\|3jul.nc oa.pc\|3aug.nc oa.pc\|3sep.nc oa.pc\|3oct.nc oa.pc\|3nov.nc oa.pe\|2dec.nc oa.pe\|3jan.nc oa.pe\|3feb.nc oa.pe\|3mar.nc oa.pe\|3apr.nc oa.pe\|3may.nc oa.pe\|3jun.nc oa.pe\|3jul.nc oa.pe\|3aug.nc oa.pe\|3sep.nc oa.pe\|3oct.nc oa.pe\|3nov.nc oa.pc\|3dec.nc oa.pc\|4jan.nc oa.pc\|4feb.nc oa.pc\|4mar.nc oa.pc\|4apr.nc oa.pc\|4may.nc oa.pc\|4jun.nc oa.pc\|4jul.nc oa.pc\|4aug.nc oa.pc\|4sep.nc oa.pc\|4oct.nc oa.pc\|4nov.nc oa.pe\|3dec.nc oa.pe\|4jan.nc oa.pe\|4feb.nc oa.pe\|4mar.nc oa.pe\|4apr.nc oa.pe\|4may.nc oa.pe\|4jun.nc oa.pe\|4jul.nc oa.pe\|4aug.nc oa.pe\|4sep.nc oa.pe\|4oct.nc oa.pe\|4nov.nc oa.pc\|4dec.nc oa.pc\|5jan.nc oa.pc\|5feb.nc oa.pc\|5mar.nc oa.pc\|5apr.nc oa.pc\|5may.nc oa.pc\|5jun.nc oa.pc\|5jul.nc oa.pc\|5aug.nc oa.pc\|5sep.nc oa.pc\|5oct.nc oa.pc\|5nov.nc oa.pe\|4dec.nc oa.pe\|5jan.nc oa.pe\|5feb.nc oa.pe\|5mar.nc oa.pe\|5apr.nc oa.pe\|5may.nc oa.pe\|5jun.nc oa.pe\|5jul.nc oa.pe\|5aug.nc oa.pe\|5sep.nc oa.pe\|5oct.nc oa.pe\|5nov.nc Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|5dec.nc oa.pc\|6jan.nc oa.pc\|6feb.nc oa.pc\|6mar.nc oa.pc\|6apr.nc oa.pc\|6may.nc oa.pc\|6jun.nc oa.pc\|6jul.nc oa.pc\|6aug.nc oa.pc\|6sep.nc oa.pc\|6oct.nc oa.pc\|6nov.nc oa.pe\|5dec.nc oa.pe\|6jan.nc oa.pe\|6feb.nc oa.pe\|6mar.nc oa.pe\|6apr.nc oa.pe\|6may.nc oa.pe\|6jun.nc oa.pe\|6jul.nc oa.pe\|6aug.nc oa.pe\|6sep.nc oa.pe\|6oct.nc oa.pe\|6nov.nc Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|6dec.nc oa.pc\|7jan.nc oa.pc\|7feb.nc oa.pc\|7mar.nc oa.pc\|7apr.nc oa.pc\|7may.nc oa.pc\|7jun.nc oa.pc\|7jul.nc oa.pc\|7aug.nc oa.pc\|7sep.nc oa.pc\|7oct.nc oa.pc\|7nov.nc oa.pe\|6dec.nc oa.pe\|7jan.nc oa.pe\|7feb.nc oa.pe\|7mar.nc oa.pe\|7apr.nc oa.pe\|7may.nc oa.pe\|7jun.nc oa.pe\|7jul.nc oa.pe\|7aug.nc oa.pe\|7sep.nc oa.pe\|7oct.nc oa.pe\|7nov.nc CPDN Monitor - Quit request from BOINC... oa.pc\|8dec.nc oa.pc\|9jan.nc oa.pc\|9feb.nc oa.pc\|9mar.nc oa.pc\|9apr.nc oa.pc\|9may.nc oa.pc\|9jun.nc oa.pc\|9jul.nc oa.pc\|9aug.nc oa.pc\|9sep.nc oa.pc\|9oct.nc oa.pc\|9nov.nc oa.pe\|8dec.nc oa.pe\|9jan.nc oa.pe\|9feb.nc oa.pe\|9mar.nc oa.pe\|9apr.nc oa.pe\|9may.nc oa.pe\|9jun.nc oa.pe\|9jul.nc oa.pe\|9aug.nc oa.pe\|9sep.nc oa.pe\|9oct.nc oa.pe\|9nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: Model crashed: Model crashed: Model crashed: Model crashed: Model crashed: Sorry, too many model crashes! :-( oa.pc\|1dec.nc oa.pc\|2jan.nc oa.pc\|2feb.nc oa.pc\|2mar.nc oa.pc\|2apr.nc oa.pc\|2may.nc oa.pc\|2jun.nc oa.pc\|2jul.nc oa.pc\|2aug.nc oa.pc\|2sep.nc oa.pc\|2oct.nc oa.pc\|2nov.nc oa.pe\|1dec.nc oa.pe\|2jan.nc oa.pe\|2feb.nc oa.pe\|2mar.nc oa.pe\|2apr.nc oa.pe\|2may.nc oa.pe\|2jun.nc oa.pe\|2jul.nc oa.pe\|2aug.nc oa.pe\|2sep.nc oa.pe\|2oct.nc oa.pe\|2nov.nc oa.pc\|2dec.nc oa.pc\|3jan.nc oa.pc\|3feb.nc oa.pc\|3mar.nc oa.pc\|3apr.nc oa.pc\|3may.nc oa.pc\|3jun.nc oa.pc\|3jul.nc oa.pc\|3aug.nc oa.pc\|3sep.nc oa.pc\|3oct.nc oa.pc\|3nov.nc oa.pe\|2dec.nc oa.pe\|3jan.nc oa.pe\|3feb.nc oa.pe\|3mar.nc oa.pe\|3apr.nc oa.pe\|3may.nc oa.pe\|3jun.nc oa.pe\|3jul.nc oa.pe\|3aug.nc oa.pe\|3sep.nc oa.pe\|3oct.nc oa.pe\|3nov.nc oa.pc\|3dec.nc oa.pc\|4jan.nc oa.pc\|4feb.nc oa.pc\|4mar.nc oa.pc\|4apr.nc oa.pc\|4may.nc oa.pc\|4jun.nc oa.pc\|4jul.nc oa.pc\|4aug.nc oa.pc\|4sep.nc oa.pc\|4oct.nc oa.pc\|4nov.nc oa.pe\|3dec.nc oa.pe\|4jan.nc oa.pe\|4feb.nc oa.pe\|4mar.nc oa.pe\|4apr.nc oa.pe\|4may.nc oa.pe\|4jun.nc oa.pe\|4jul.nc oa.pe\|4aug.nc oa.pe\|4sep.nc oa.pe\|4oct.nc oa.pe\|4nov.nc oa.pc\|4dec.nc oa.pc\|5jan.nc oa.pc\|5feb.nc oa.pc\|5mar.nc oa.pc\|5apr.nc oa.pc\|5may.nc oa.pc\|5jun.nc oa.pc\|5jul.nc oa.pc\|5aug.nc oa.pc\|5sep.nc oa.pc\|5oct.nc oa.pc\|5nov.nc oa.pe\|4dec.nc oa.pe\|5jan.nc oa.pe\|5feb.nc oa.pe\|5mar.nc oa.pe\|5apr.nc oa.pe\|5may.nc oa.pe\|5jun.nc oa.pe\|5jul.nc oa.pe\|5aug.nc oa.pe\|5sep.nc oa.pe\|5oct.nc oa.pe\|5nov.nc oa.pc\|5dec.nc oa.pc\|6jan.nc oa.pc\|6feb.nc oa.pc\|6mar.nc oa.pc\|6apr.nc oa.pc\|6may.nc oa.pc\|6jun.nc oa.pc\|6jul.nc oa.pc\|6aug.nc oa.pc\|6sep.nc oa.pc\|6oct.nc oa.pc\|6nov.nc oa.pe\|5dec.nc oa.pe\|6jan.nc oa.pe\|6feb.nc oa.pe\|6mar.nc oa.pe\|6apr.nc oa.pe\|6may.nc oa.pe\|6jun.nc oa.pe\|6jul.nc oa.pe\|6aug.nc oa.pe\|6sep.nc oa.pe\|6oct.nc oa.pe\|6nov.nc oa.pc\|6dec.nc oa.pc\|7jan.nc oa.pc\|7feb.nc oa.pc\|7mar.nc oa.pc\|7apr.nc oa.pc\|7may.nc oa.pc\|7jun.nc oa.pc\|7jul.nc oa.pc\|7aug.nc oa.pc\|7sep.nc oa.pc\|7oct.nc oa.pc\|7nov.nc oa.pe\|6dec.nc oa.pe\|7jan.nc oa.pe\|7feb.nc oa.pe\|7mar.nc oa.pe\|7apr.nc oa.pe\|7may.nc oa.pe\|7jun.nc oa.pe\|7jul.nc oa.pe\|7aug.nc oa.pe\|7sep.nc oa.pe\|7oct.nc oa.pe\|7nov.nc BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadam3pm2_e8ft_1991_10_008713980/dataout/stdout_um.txt Image PC Routine Line Source hadam3pm2_um_7.03 00399B82 Unknown Unknown Unknown hadam3pm2_um_7.03 0039864B Unknown Unknown Unknown hadam3pm2_um_7.03 0035FE0D Unknown Unknown Unknown hadam3pm2_um_7.03 003161B5 Unknown Unknown Unknown hadam3pm2_um_7.03 00315917 Unknown Unknown Unknown hadam3pm2_um_7.03 00352527 Unknown Unknown Unknown hadam3pm2_um_7.03 000AA8CE Unknown Unknown Unknown hadam3pm2_um_7.03 00132125 Unknown Unknown Unknown hadam3pm2_um_7.03 0013BD7B Unknown Unknown Unknown hadam3pm2_um_7.03 002B6D17 Unknown Unknown Unknown hadam3pm2_um_7.03 002BFB8B Unknown Unknown Unknown hadam3pm2_um_7.03 002EA01F Unknown Unknown Unknown hadam3pm2_um_7.03 0000285B Unknown Unknown Unknown hadam3pm2_um_7.03 00002789 Unknown Unknown Unknown Unknown 00000008 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=284, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadam3pm2_e8ft_1991_10_008713980/dataout/stdout_um.txt Image PC Routine Line Source hadam3pm2_um_7.03 00399B82 Unknown Unknown Unknown hadam3pm2_um_7.03 0039864B Unknown Unknown Unknown hadam3pm2_um_7.03 0035FE0D Unknown Unknown Unknown hadam3pm2_um_7.03 003161B5 Unknown Unknown Unknown hadam3pm2_um_7.03 00315917 Unknown Unknown Unknown hadam3pm2_um_7.03 00352527 Unknown Unknown Unknown hadam3pm2_um_7.03 000AA8CE Unknown Unknown Unknown hadam3pm2_um_7.03 00132125 Unknown Unknown Unknown hadam3pm2_um_7.03 0013BD7B Unknown Unknown Unknown hadam3pm2_um_7.03 002 </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 May 2014 00:13:24	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	293,828	1,695,482	5.7703
21 May 2014 19:04:31	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	290,948	1,678,906	5.7705
21 May 2014 13:57:49	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	288,068	1,662,454	5.7710
21 May 2014 08:51:06	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	285,188	1,645,948	5.7714
21 May 2014 03:39:48	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	282,308	1,629,560	5.7723
20 May 2014 22:36:33	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	279,428	1,613,167	5.7731
20 May 2014 12:20:19	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	273,668	1,580,224	5.7742
20 May 2014 07:06:30	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	270,788	1,563,445	5.7737
20 May 2014 01:55:14	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	267,908	1,546,589	5.7728
19 May 2014 20:38:48	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	265,028	1,529,829	5.7723
19 May 2014 15:30:00	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	262,148	1,513,189	5.7723
19 May 2014 10:18:51	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	259,268	1,496,567	5.7723
19 May 2014 05:12:37	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	256,388	1,479,960	5.7723
19 May 2014 00:06:17	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	253,508	1,463,442	5.7728
18 May 2014 18:58:57	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	250,628	1,446,860	5.7729
18 May 2014 14:19:59	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	247,748	1,430,379	5.7735
18 May 2014 09:06:39	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	244,868	1,413,906	5.7742
16 May 2014 19:29:13	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	230,468	1,330,256	5.7720
16 May 2014 14:30:01	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	227,588	1,313,568	5.7717
16 May 2014 09:22:03	1320181	16572626	hadam3pm2_e8ft_1991_10_008713980_0	224,708	1,296,901	5.7715