Task 15692076

Name	hadcm3n_3edg_1980_40_008340824_0
Workunit	8491685
Created	29 Mar 2013, 3:24:33 UTC
Sent	29 Mar 2013, 10:38:19 UTC
Report deadline	28 Jun 2013, 18:05:30 UTC
Received	1 May 2013, 6:22:39 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1239468
Run time	13 days 0 hours 14 min 54 sec
CPU time	10 days 17 hours 33 min 13 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.52 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x089f8d88 *** ======= Backtrace: ========= /lib/libc.so.6[0x4c49a1] /lib/libc.so.6[0x4c70e1] /usr/lib/libstdc++.so.6(_ZdlPv+0x22)[0xf7700e32] /usr/lib/libstdc++.so.6(_ZdaPv+0x1e)[0xf7700e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xe6)[0x46ace6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 00432000-00450000 r-xp 00000000 08:06 1442860 /lib/ld-2.12.so 00450000-00451000 r-xp 0001d000 08:06 1442860 /lib/ld-2.12.so 00451000-00452000 rwxp 0001e000 08:06 1442860 /lib/ld-2.12.so 00454000-005e4000 r-xp 00000000 08:06 1442816 /lib/libc-2.12.so 005e4000-005e6000 r-xp 0018f000 08:06 1442816 /lib/libc-2.12.so 005e6000-005e7000 rwxp 00191000 08:06 1442816 /lib/libc-2.12.so 005e7000-005ea000 rwxp 00000000 00:00 0 005ec000-005ef000 r-xp 00000000 08:06 1442862 /lib/libdl-2.12.so 005ef000-005f0000 r-xp 00002000 08:06 1442862 /lib/libdl-2.12.so 005f0000-005f1000 rwxp 00003000 08:06 1442862 /lib/libdl-2.12.so 005f3000-0060a000 r-xp 00000000 08:06 1442868 /lib/libpthread-2.12.so 0060a000-0060b000 r-xp 00016000 08:06 1442868 /lib/libpthread-2.12.so 0060b000-0060c000 rwxp 00017000 08:06 1442868 /lib/libpthread-2.12.so 0060c000-0060e000 rwxp 00000000 00:00 0 08048000-080e3000 r-xp 00000000 08:02 5767283 /mnt/users1/staf/wes/BOINC/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rwxp 0009b000 08:02 5767283 /mnt/users1/staf/wes/BOINC/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rwxp 00000000 00:00 0 089a3000-08a2a000 rwxp 00000000 00:00 0 [heap] f7186000-f7187000 rwxp 00000000 00:00 0 f7187000-f7602000 rwxs 00000000 08:02 51656980 /mnt/users1/staf/wes/BOINC/BOINC/slots/36/135750 f7602000-f7603000 rwxp 00000000 00:00 0 f7603000-f7604000 ---p 00000000 00:00 0 f7604000-f7607000 rwxp 00000000 00:00 0 f7607000-f7609000 rwxs 00000000 08:02 51658074 /mnt/users1/staf/wes/BOINC/BOINC/slots/36/boinc_mmap_file f7609000-f760c000 rwxp 00000000 00:00 0 f760c000-f7629000 r-xp 00000000 08:06 1445356 /lib/libgcc_s-4.4.6-20110824.so.1 f7629000-f762a000 rwxp 0001d000 08:06 1445356 /lib/libgcc_s-4.4.6-20110824.so.1 f762a000-f7652000 r-xp 00000000 08:06 1442863 /lib/libm-2.12.so f7652000-f7653000 r-xp 00027000 08:06 1442863 /lib/libm-2.12.so f7653000-f7654000 rwxp 00028000 08:06 1442863 /lib/libm-2.12.so f7654000-f7733000 r-xp 00000000 08:06 1966939 /usr/lib/libstdc++.so.6.0.13 f7733000-f7737000 r-xp 000de000 08:06 1966939 /usr/lib/libstdc++.so.6.0.13 f7737000-f7738000 rwxp 000e2000 08:06 1966939 /usr/lib/libstdc++.so.6.0.13 f7738000-f7741000 rwxp 00000000 00:00 0 f7741000-f7742000 r-xp 00000000 00:00 0 [vdso] ff9cf000-ffa3f000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (18 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf7741400] [0xf7741430] /lib/libc.so.6(gsignal+0x51)[0x47eb01] /lib/libc.so.6(abort+0x17a)[0x4803da] /lib/libc.so.6[0x4be545] /lib/libc.so.6[0x4c49a1] /lib/libc.so.6[0x4c70e1] /usr/lib/libstdc++.so.6(_ZdlPv+0x22)[0xf7700e32] /usr/lib/libstdc++.so.6(_ZdaPv+0x1e)[0xf7700e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xe6)[0x46ace6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 May 2013 06:23:10	1239468	15692076	hadcm3n_3edg_1980_40_008340824_0	259,200	927,189	3.5771
24 Apr 2013 19:09:58	1239468	15692076	hadcm3n_3edg_1980_40_008340824_0	233,280	836,823	3.5872
16 Apr 2013 13:55:14	1239468	15692076	hadcm3n_3edg_1980_40_008340824_0	207,360	745,361	3.5945
15 Apr 2013 14:32:37	1239468	15692076	hadcm3n_3edg_1980_40_008340824_0	181,440	650,513	3.5853
13 Apr 2013 02:51:31	1239468	15692076	hadcm3n_3edg_1980_40_008340824_0	155,520	558,739	3.5927
11 Apr 2013 11:29:54	1239468	15692076	hadcm3n_3edg_1980_40_008340824_0	129,600	465,214	3.5896
08 Apr 2013 19:28:06	1239468	15692076	hadcm3n_3edg_1980_40_008340824_0	103,680	371,018	3.5785
07 Apr 2013 02:35:53	1239468	15692076	hadcm3n_3edg_1980_40_008340824_0	77,760	275,451	3.5423
05 Apr 2013 23:44:11	1239468	15692076	hadcm3n_3edg_1980_40_008340824_0	51,840	182,780	3.5258
03 Apr 2013 16:11:02	1239468	15692076	hadcm3n_3edg_1980_40_008340824_0	25,920	87,967	3.3938