climateprediction.net home page
Task 13351598

Task 13351598

Name hadcm3n_o7nv_1940_40_007444287_2
Workunit 7641790
Created 9 Sep 2011, 3:04:27 UTC
Sent 9 Sep 2011, 3:11:32 UTC
Report deadline 9 Dec 2011, 10:38:43 UTC
Received 26 Sep 2011, 5:28:43 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1317410
Run time 10 days 21 hours 7 min
CPU time 10 days 10 hours 36 min 35 sec
Validate state Invalid
Credit 3,110.40
Device peak FLOPS 1.54 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr
<core_client_version>6.12.33</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
05:55:14 (13905): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:56:45 (26811): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:00:20 (26883): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:01:05 (26943): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:02:25 (26992): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:03:11 (27045): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:04:44 (27102): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:05:37 (27149): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:06:21 (27199): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:07:09 (27247): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:08:40 (27305): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:09:29 (27352): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:10:58 (27410): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:11:46 (27459): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:13:16 (27518): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:14:16 (27565): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:15:00 (27614): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:17:08 (27681): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:18:38 (27743): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:19:25 (27791): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:20:56 (27848): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:21:44 (27896): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:23:14 (27956): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:24:13 (28003): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:24:57 (28051): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:26:29 (28116): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:28:36 (28182): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:29:23 (28230): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:30:16 (28277): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:31:46 (28337): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:32:33 (28384): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:33:20 (28432): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:34:56 (28490): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:35:39 (28537): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:36:35 (28585): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:37:23 (28633): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:38:10 (28682): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:39:03 (28729): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:39:54 (28777): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:41:37 (28836): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:42:21 (28884): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:43:10 (28932): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:44:41 (28992): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:45:26 (29039): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:46:12 (29086): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:47:57 (29146): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:48:45 (29196): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:49:43 (29243): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:50:26 (29293): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:51:15 (29340): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:52:12 (29387): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:53:42 (29446): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:54:30 (29495): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:56:02 (29557): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:56:47 (29605): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:57:36 (29651): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:59:06 (29709): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:59:54 (29757): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:01:23 (29817): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:02:10 (29864): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:03:43 (29921): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:04:29 (29969): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:05:16 (30017): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:07:01 (30077): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:07:47 (30125): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:08:32 (30172): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:10:12 (30231): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:10:56 (30279): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:11:50 (30328): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:12:34 (30375): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:15:20 (30451): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:16:52 (30511): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:17:39 (30561): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:18:31 (30608): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:20:02 (30666): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:20:48 (30714): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:21:35 (30763): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:24:27 (30840): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:25:16 (30887): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:26:00 (30938): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:27:34 (30998): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:28:21 (31045): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:29:18 (31092): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:30:54 (31156): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:31:39 (31206): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:32:26 (31254): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:33:17 (31301): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:34:48 (31359): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:35:34 (31433): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:37:14 (31506): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:38:01 (31556): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:38:47 (31602): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:39:41 (31649): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:40:24 (31697): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
09:25:30 (6127): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
09:26:42 (6584): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08f9bce8 ***
======= Backtrace: =========
/lib32/libc.so.6(+0x6b511)[0xf7549511]
/lib32/libc.so.6(+0x6ce1b)[0xf754ae1b]
/lib32/libc.so.6(cfree+0x6d)[0xf754df8d]
/usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf7727841]
/usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf772789d]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib32/libc.so.6(__libc_start_main+0xe7)[0xf74f4e37]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]
======= Memory map: ========
08048000-080e3000 r-xp 00000000 08:05 794949                             /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e3000-080e4000 rw-p 0009b000 08:05 794949                             /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e4000-0813b000 rw-p 00000000 00:00 0 
08f46000-08fac000 rw-p 00000000 00:00 0                                  [heap]
f6f00000-f6f21000 rw-p 00000000 00:00 0 
f6f21000-f7000000 ---p 00000000 00:00 0 
f7060000-f74db000 rw-s 00000000 08:05 803528                             /var/lib/boinc-client/slots/9/138095
f74db000-f74de000 rw-p 00000000 00:00 0 
f74de000-f7634000 r-xp 00000000 08:05 1700615                            /lib32/libc-2.13.so
f7634000-f7635000 ---p 00156000 08:05 1700615                            /lib32/libc-2.13.so
f7635000-f7637000 r--p 00156000 08:05 1700615                            /lib32/libc-2.13.so
f7637000-f7638000 rw-p 00158000 08:05 1700615                            /lib32/libc-2.13.so
f7638000-f763b000 rw-p 00000000 00:00 0 
f763b000-f7655000 r-xp 00000000 08:05 277567                             /usr/lib32/libgcc_s.so.1
f7655000-f7656000 r--p 00019000 08:05 277567                             /usr/lib32/libgcc_s.so.1
f7656000-f7657000 rw-p 0001a000 08:05 277567                             /usr/lib32/libgcc_s.so.1
f7657000-f767b000 r-xp 00000000 08:05 1700619                            /lib32/libm-2.13.so
f767b000-f767c000 r--p 00023000 08:05 1700619                            /lib32/libm-2.13.so
f767c000-f767d000 rw-p 00024000 08:05 1700619                            /lib32/libm-2.13.so
f767d000-f775c000 r-xp 00000000 08:05 277570                             /usr/lib32/libstdc++.so.6.0.14
f775c000-f7760000 r--p 000de000 08:05 277570                             /usr/lib32/libstdc++.so.6.0.14
f7760000-f7761000 rw-p 000e2000 08:05 277570                             /usr/lib32/libstdc++.so.6.0.14
f7761000-f7768000 rw-p 00000000 00:00 0 
f7768000-f776a000 r-xp 00000000 08:05 1700618                            /lib32/libdl-2.13.so
f776a000-f776b000 r--p 00001000 08:05 1700618                            /lib32/libdl-2.13.so
f776b000-f776c000 rw-p 00002000 08:05 1700618                            /lib32/libdl-2.13.so
f776c000-f7781000 r-xp 00000000 08:05 1700629                            /lib32/libpthread-2.13.so
f7781000-f7782000 r--p 00014000 08:05 1700629                            /lib32/libpthread-2.13.so
f7782000-f7783000 rw-p 00015000 08:05 1700629                            /lib32/libpthread-2.13.so
f7783000-f7786000 rw-p 00000000 00:00 0 
f77a3000-f77a4000 rw-p 00000000 00:00 0 
f77a4000-f77a5000 ---p 00000000 00:00 0 
f77a5000-f77a8000 rw-p 00000000 00:00 0 
f77a8000-f77aa000 rw-s 00000000 08:05 803076                             /var/lib/boinc-client/slots/9/boinc_mmap_file
f77aa000-f77ab000 rw-p 00000000 00:00 0 
f77ab000-f77ac000 r-xp 00000000 00:00 0                                  [vdso]
f77ac000-f77c8000 r-xp 00000000 08:05 1700612                            /lib32/ld-2.13.so
f77c8000-f77c9000 r--p 0001b000 08:05 1700612                            /lib32/ld-2.13.so
f77c9000-f77ca000 rw-p 0001c000 08:05 1700612                            /lib32/ld-2.13.so
ffa18000-ffa88000 rw-p 00000000 00:00 0                                  [stack]
SIGABRT: abort called
Stack trace (19 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
[0xf77ab400]
[0xf77ab430]
/lib32/libc.so.6(gsignal+0x51)[0xf7508ea1]
/lib32/libc.so.6(abort+0x17e)[0xf750c2ce]
/lib32/libc.so.6(+0x61277)[0xf753f277]
/lib32/libc.so.6(+0x6b511)[0xf7549511]
/lib32/libc.so.6(+0x6ce1b)[0xf754ae1b]
/lib32/libc.so.6(cfree+0x6d)[0xf754df8d]
/usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf7727841]
/usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf772789d]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib32/libc.so.6(__libc_start_main+0xe7)[0xf74f4e37]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
26 Sep 2011 04:27:07 1162363 13351598 hadcm3n_o7nv_1940_40_007444287_2 259,200 902,186 3.4807
25 Sep 2011 02:01:06 1162363 13351598 hadcm3n_o7nv_1940_40_007444287_2 233,280 811,838 3.4801
23 Sep 2011 23:55:38 1162363 13351598 hadcm3n_o7nv_1940_40_007444287_2 207,360 721,848 3.4811
22 Sep 2011 20:23:37 1162363 13351598 hadcm3n_o7nv_1940_40_007444287_2 181,440 632,336 3.4851
21 Sep 2011 18:36:54 1162363 13351598 hadcm3n_o7nv_1940_40_007444287_2 155,520 541,224 3.4801
20 Sep 2011 15:27:25 1162363 13351598 hadcm3n_o7nv_1940_40_007444287_2 129,600 451,045 3.4803
19 Sep 2011 12:15:31 1162363 13351598 hadcm3n_o7nv_1940_40_007444287_2 103,680 361,168 3.4835
18 Sep 2011 09:17:24 1162363 13351598 hadcm3n_o7nv_1940_40_007444287_2 77,760 270,629 3.4803
13 Sep 2011 07:01:18 1162363 13351598 hadcm3n_o7nv_1940_40_007444287_2 51,840 181,084 3.4931
11 Sep 2011 18:34:16 1162363 13351598 hadcm3n_o7nv_1940_40_007444287_2 25,920 91,190 3.5181


©2024 climateprediction.net