Task 16080069

Name	hadcm3n_808i_1980_40_008458245_3
Workunit	8609101
Created	13 Nov 2013, 23:06:20 UTC
Sent	13 Nov 2013, 23:06:30 UTC
Report deadline	13 Feb 2014, 6:33:41 UTC
Received	31 Dec 2013, 21:07:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1298346
Run time	35 days 11 hours 7 min 18 sec
CPU time	29 days 16 hours 36 min 5 sec
Validate state	Invalid
Credit	11,197.44
Device peak FLOPS	1.82 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.7</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_808i_1980_40_008458245/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 084554BF Unknown Unknown Unknown hadcm3n_um_6.07_i Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1521, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_808i_1980_40_008458245/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 084554BF Unknown Unknown Unknown hadcm3n_um_6.07_i 0832327C Unknown Unknown Unknown hadcm3n_um_6.07_i 08198894 Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F74C8905 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1521, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Dec 2013 22:07:28	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	933,120	2,801,077	3.0018
29 Dec 2013 23:19:39	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	907,200	2,723,671	3.0023
28 Dec 2013 22:25:24	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	881,280	2,644,428	3.0007
27 Dec 2013 23:03:16	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	855,360	2,565,284	2.9991
26 Dec 2013 16:36:33	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	829,440	2,485,585	2.9967
25 Dec 2013 17:24:04	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	803,520	2,406,981	2.9955
24 Dec 2013 18:26:31	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	777,600	2,328,300	2.9942
23 Dec 2013 18:39:36	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	751,680	2,249,035	2.9920
22 Dec 2013 18:01:39	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	725,760	2,169,876	2.9898
21 Dec 2013 16:50:06	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	699,840	2,090,855	2.9876
20 Dec 2013 16:10:32	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	673,920	2,011,885	2.9853
19 Dec 2013 17:08:11	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	648,000	1,933,154	2.9833
18 Dec 2013 17:55:27	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	622,080	1,854,133	2.9805
17 Dec 2013 18:41:55	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	596,160	1,774,524	2.9766
16 Dec 2013 19:11:52	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	570,240	1,694,813	2.9721
15 Dec 2013 19:50:16	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	544,320	1,615,264	2.9675
14 Dec 2013 12:32:14	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	518,400	1,535,148	2.9613
13 Dec 2013 07:48:08	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	492,480	1,454,865	2.9542
12 Dec 2013 08:35:39	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	466,560	1,375,406	2.9480
11 Dec 2013 09:38:29	1298346	16080069	hadcm3n_808i_1980_40_008458245_3	440,640	1,296,085	2.9414