climateprediction.net home page
Task 22276623

Task 22276623

Name oifs_43r3_ps_0323_2005050100_123_974_12190967_0
Workunit 12190967
Created 23 Dec 2022, 10:16:26 UTC
Sent 23 Dec 2022, 10:46:15 UTC
Report deadline 22 Jan 2023, 10:46:15 UTC
Received 1 Jan 2023, 19:44:17 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1520317
Run time 20 hours 39 min 10 sec
CPU time 17 hours 32 min 59 sec
Validate state Invalid
Credit 13,184.64
Device peak FLOPS 8.55 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.39 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,743.54 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
00] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:29616:29616] [20230101:173015:1672554615:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:29616:29616] [20230101:173015:1672554615:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:29616:29616] [20230101:173015:1672554615:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:roo:1:1:29616:29616] [20230101:173015:1672554615:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:29616:29616] [20230101:173015:1672554615:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:29616:29616] [20230101:173015:1672554615:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:29616:29616] [20230101:173015:1672554615:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:29616:29616] [20230101:173015:1672554615:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:29616:29616] [20230101:173015:1672554615:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:29616:29616] [20230101:173015:1672554615:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:29616:29616] [20230101:173015:1672554615:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:29616:29616] [20230101:173015:1672554615:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:29616:29616] [20230101:173015:1672554615:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:30:51 STEP 2136 H=2136:00 +CPU= 28.964
  17:31:09 STEP 2137 H=2137:00 +CPU= 16.797
  17:31:28 STEP 2138 H=2138:00 +CPU= 16.746
  17:31:56 STEP 2139 H=2139:00 +CPU= 25.516
  17:32:14 STEP 2140 H=2140:00 +CPU= 16.762
  17:32:32 STEP 2141 H=2141:00 +CPU= 16.302
  17:33:00 STEP 2142 H=2142:00 +CPU= 25.792
  17:33:18 STEP 2143 H=2143:00 +CPU= 16.192
  17:33:36 STEP 2144 H=2144:00 +CPU= 16.637
  17:34:03 STEP 2145 H=2145:00 +CPU= 25.100
  17:34:22 STEP 2146 H=2146:00 +CPU= 16.825
  17:34:40 STEP 2147 H=2147:00 +CPU= 16.379
  17:35:11 STEP 2148 H=2148:00 +CPU= 28.158
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002148
  17:35:29 STEP 2149 H=2149:00 +CPU= 16.424
  17:35:48 STEP 2150 H=2150:00 +CPU= 16.971
  17:36:15 STEP 2151 H=2151:00 +CPU= 25.285
  17:36:34 STEP 2152 H=2152:00 +CPU= 16.658
  17:36:52 STEP 2153 H=2153:00 +CPU= 16.335
  17:37:20 STEP 2154 H=2154:00 +CPU= 25.531
  17:37:37 STEP 2155 H=2155:00 +CPU= 16.228
  17:37:56 STEP 2156 H=2156:00 +CPU= 16.542
  17:38:23 STEP 2157 H=2157:00 +CPU= 24.856
  17:38:41 STEP 2158 H=2158:00 +CPU= 16.497
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:06:30 STEP 2159 H=2159:00 +CPU= 16.956
  22:07:03 STEP 2160 H=2160:00 +CPU= 30.318
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_89.zip
Uploading the intermediate file: upload_file_89.zip
  22:07:22 STEP 2161 H=2161:00 +CPU= 16.705
  22:07:41 STEP 2162 H=2162:00 +CPU= 17.177
  22:08:09 STEP 2163 H=2163:00 +CPU= 25.707
  22:08:28 STEP 2164 H=2164:00 +CPU= 17.109
  22:08:48 STEP 2165 H=2165:00 +CPU= 16.826
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:36:52 STEP 2166 H=2166:00 +CPU= 26.665
  01:37:09 STEP 2167 H=2167:00 +CPU= 15.917
  01:37:26 STEP 2168 H=2168:00 +CPU= 16.482
  01:37:51 STEP 2169 H=2169:00 +CPU= 24.825
  01:38:08 STEP 2170 H=2170:00 +CPU= 16.087
  01:38:24 STEP 2171 H=2171:00 +CPU= 15.496
  01:38:52 STEP 2172 H=2172:00 +CPU= 27.424
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002172
  01:39:09 STEP 2173 H=2173:00 +CPU= 16.326
  01:39:26 STEP 2174 H=2174:00 +CPU= 16.284
  01:39:52 STEP 2175 H=2175:00 +CPU= 24.690
  01:40:08 STEP 2176 H=2176:00 +CPU= 16.288
  01:40:24 STEP 2177 H=2177:00 +CPU= 15.644
  01:40:51 STEP 2178 H=2178:00 +CPU= 25.387
  01:41:08 STEP 2179 H=2179:00 +CPU= 15.980
  01:41:25 STEP 2180 H=2180:00 +CPU= 16.299
  01:41:51 STEP 2181 H=2181:00 +CPU= 24.692
  01:42:07 STEP 2182 H=2182:00 +CPU= 15.923
  01:42:24 STEP 2183 H=2183:00 +CPU= 16.151
  01:42:56 STEP 2184 H=2184:00 +CPU= 29.903
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_90.zip
Uploading the intermediate file: upload_file_90.zip
  01:43:12 STEP 2185 H=2185:00 +CPU= 15.799
  01:43:29 STEP 2186 H=2186:00 +CPU= 16.487
  01:43:55 STEP 2187 H=2187:00 +CPU= 24.840
  01:44:12 STEP 2188 H=2188:00 +CPU= 16.126
  01:44:28 STEP 2189 H=2189:00 +CPU= 15.973
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:57:14 STEP 2190 H=2190:00 +CPU= 25.010
  02:57:31 STEP 2191 H=2191:00 +CPU= 15.805
  02:57:48 STEP 2192 H=2192:00 +CPU= 16.402
  02:58:13 STEP 2193 H=2193:00 +CPU= 24.229
  02:58:29 STEP 2194 H=2194:00 +CPU= 16.275
  02:58:45 STEP 2195 H=2195:00 +CPU= 15.719
  02:59:13 STEP 2196 H=2196:00 +CPU= 27.071
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002196
  02:59:30 STEP 2197 H=2197:00 +CPU= 15.914
  02:59:47 STEP 2198 H=2198:00 +CPU= 16.366
  03:00:12 STEP 2199 H=2199:00 +CPU= 24.417
  03:00:29 STEP 2200 H=2200:00 +CPU= 16.134
  03:00:45 STEP 2201 H=2201:00 +CPU= 15.566
  03:01:11 STEP 2202 H=2202:00 +CPU= 24.995
  03:01:27 STEP 2203 H=2203:00 +CPU= 15.722
  03:01:44 STEP 2204 H=2204:00 +CPU= 16.250
  03:02:10 STEP 2205 H=2205:00 +CPU= 24.636
  03:02:27 STEP 2206 H=2206:00 +CPU= 16.218
  03:02:44 STEP 2207 H=2207:00 +CPU= 16.004
  03:03:15 STEP 2208 H=2208:00 +CPU= 29.371
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_91.zip
Uploading the intermediate file: upload_file_91.zip
  03:03:31 STEP 2209 H=2209:00 +CPU= 15.692
  03:03:48 STEP 2210 H=2210:00 +CPU= 16.477
  03:04:13 STEP 2211 H=2211:00 +CPU= 24.554
  03:04:30 STEP 2212 H=2212:00 +CPU= 16.649
  03:04:47 STEP 2213 H=2213:00 +CPU= 15.656
  03:05:13 STEP 2214 H=2214:00 +CPU= 25.237
  03:05:29 STEP 2215 H=2215:00 +CPU= 15.478
  03:05:46 STEP 2216 H=2216:00 +CPU= 15.905
  03:06:11 STEP 2217 H=2217:00 +CPU= 24.609
  03:06:27 STEP 2218 H=2218:00 +CPU= 15.987
  03:06:44 STEP 2219 H=2219:00 +CPU= 15.512
  03:07:11 STEP 2220 H=2220:00 +CPU= 27.070
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002220
  03:07:27 STEP 2221 H=2221:00 +CPU= 15.358
  03:07:44 STEP 2222 H=2222:00 +CPU= 15.958
  03:08:09 STEP 2223 H=2223:00 +CPU= 24.427
  03:08:26 STEP 2224 H=2224:00 +CPU= 16.344
  03:08:42 STEP 2225 H=2225:00 +CPU= 15.377
  03:09:08 STEP 2226 H=2226:00 +CPU= 24.854
  03:09:24 STEP 2227 H=2227:00 +CPU= 15.573
  03:09:40 STEP 2228 H=2228:00 +CPU= 16.188
  03:10:06 STEP 2229 H=2229:00 +CPU= 24.470
  03:10:23 STEP 2230 H=2230:00 +CPU= 16.061
  03:10:39 STEP 2231 H=2231:00 +CPU= 15.651
  03:11:09 STEP 2232 H=2232:00 +CPU= 29.092
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_92.zip
Uploading the intermediate file: upload_file_92.zip
  03:11:26 STEP 2233 H=2233:00 +CPU= 15.705
  03:11:43 STEP 2234 H=2234:00 +CPU= 15.960
  03:12:08 STEP 2235 H=2235:00 +CPU= 24.503
  03:12:25 STEP 2236 H=2236:00 +CPU= 15.914
  03:12:41 STEP 2237 H=2237:00 +CPU= 15.616
  03:13:07 STEP 2238 H=2238:00 +CPU= 24.859
  03:13:23 STEP 2239 H=2239:00 +CPU= 15.743
  03:13:41 STEP 2240 H=2240:00 +CPU= 16.504
  03:14:07 STEP 2241 H=2241:00 +CPU= 26.100
  03:14:25 STEP 2242 H=2242:00 +CPU= 16.996
  03:14:41 STEP 2243 H=2243:00 +CPU= 15.484
  03:15:09 STEP 2244 H=2244:00 +CPU= 27.475
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002244
  03:15:25 STEP 2245 H=2245:00 +CPU= 15.443
  03:15:42 STEP 2246 H=2246:00 +CPU= 16.028
  03:16:07 STEP 2247 H=2247:00 +CPU= 24.709
  03:16:24 STEP 2248 H=2248:00 +CPU= 15.805
  03:16:40 STEP 2249 H=2249:00 +CPU= 15.404
  03:17:05 STEP 2250 H=2250:00 +CPU= 24.815
  03:17:21 STEP 2251 H=2251:00 +CPU= 15.622
  03:17:38 STEP 2252 H=2252:00 +CPU= 15.770
  03:18:03 STEP 2253 H=2253:00 +CPU= 24.375
  03:18:20 STEP 2254 H=2254:00 +CPU= 16.090
  03:18:37 STEP 2255 H=2255:00 +CPU= 16.362
  03:19:07 STEP 2256 H=2256:00 +CPU= 29.046
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_93.zip
Uploading the intermediate file: upload_file_93.zip
  03:19:24 STEP 2257 H=2257:00 +CPU= 15.631
  03:19:40 STEP 2258 H=2258:00 +CPU= 15.963
  03:20:06 STEP 2259 H=2259:00 +CPU= 24.277
  03:20:22 STEP 2260 H=2260:00 +CPU= 16.070
  03:20:38 STEP 2261 H=2261:00 +CPU= 15.493
  03:21:05 STEP 2262 H=2262:00 +CPU= 25.227
  03:21:21 STEP 2263 H=2263:00 +CPU= 15.892
  03:21:38 STEP 2264 H=2264:00 +CPU= 16.118
  03:22:04 STEP 2265 H=2265:00 +CPU= 24.859
  03:22:20 STEP 2266 H=2266:00 +CPU= 16.041
  03:22:36 STEP 2267 H=2267:00 +CPU= 15.618
  03:23:04 STEP 2268 H=2268:00 +CPU= 27.246
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002268
  03:23:21 STEP 2269 H=2269:00 +CPU= 15.808
  03:23:37 STEP 2270 H=2270:00 +CPU= 16.100
  03:24:02 STEP 2271 H=2271:00 +CPU= 24.441
  03:24:19 STEP 2272 H=2272:00 +CPU= 16.272
  03:24:35 STEP 2273 H=2273:00 +CPU= 15.713
  03:25:01 STEP 2274 H=2274:00 +CPU= 24.908
  03:25:17 STEP 2275 H=2275:00 +CPU= 15.543
  03:25:34 STEP 2276 H=2276:00 +CPU= 15.977
  03:25:59 STEP 2277 H=2277:00 +CPU= 24.349
  03:26:15 STEP 2278 H=2278:00 +CPU= 15.977
  03:26:32 STEP 2279 H=2279:00 +CPU= 16.052
  03:27:01 STEP 2280 H=2280:00 +CPU= 29.031
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_94.zip
Uploading the intermediate file: upload_file_94.zip
  03:27:18 STEP 2281 H=2281:00 +CPU= 15.656
  03:27:34 STEP 2282 H=2282:00 +CPU= 16.020
  03:27:59 STEP 2283 H=2283:00 +CPU= 24.413
  03:28:16 STEP 2284 H=2284:00 +CPU= 16.132
  03:28:32 STEP 2285 H=2285:00 +CPU= 15.500
  03:28:58 STEP 2286 H=2286:00 +CPU= 24.847
  03:29:14 STEP 2287 H=2287:00 +CPU= 15.247
  03:29:31 STEP 2288 H=2288:00 +CPU= 15.835
  03:29:56 STEP 2289 H=2289:00 +CPU= 24.455
  03:30:13 STEP 2290 H=2290:00 +CPU= 15.894
  03:30:29 STEP 2291 H=2291:00 +CPU= 15.704
  03:30:58 STEP 2292 H=2292:00 +CPU= 27.304
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002292
  03:31:14 STEP 2293 H=2293:00 +CPU= 15.845
  03:31:31 STEP 2294 H=2294:00 +CPU= 16.357
  03:31:57 STEP 2295 H=2295:00 +CPU= 24.685
  03:32:14 STEP 2296 H=2296:00 +CPU= 16.439
  03:32:31 STEP 2297 H=2297:00 +CPU= 16.182
  03:32:56 STEP 2298 H=2298:00 +CPU= 24.781
  03:33:13 STEP 2299 H=2299:00 +CPU= 15.785
  03:33:29 STEP 2300 H=2300:00 +CPU= 16.216
  03:33:54 STEP 2301 H=2301:00 +CPU= 24.208
  03:34:11 STEP 2302 H=2302:00 +CPU= 15.929
  03:34:27 STEP 2303 H=2303:00 +CPU= 15.866
  03:34:57 STEP 2304 H=2304:00 +CPU= 29.257
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_95.zip
Uploading the intermediate file: upload_file_95.zip
  03:35:14 STEP 2305 H=2305:00 +CPU= 15.545
  03:35:30 STEP 2306 H=2306:00 +CPU= 16.262
  03:35:56 STEP 2307 H=2307:00 +CPU= 24.567
  03:36:13 STEP 2308 H=2308:00 +CPU= 15.915
  03:36:29 STEP 2309 H=2309:00 +CPU= 15.618
  03:36:55 STEP 2310 H=2310:00 +CPU= 24.940
  03:37:12 STEP 2311 H=2311:00 +CPU= 15.434
  03:37:28 STEP 2312 H=2312:00 +CPU= 16.098
  03:37:53 STEP 2313 H=2313:00 +CPU= 24.501
  03:38:10 STEP 2314 H=2314:00 +CPU= 15.963
  03:38:27 STEP 2315 H=2315:00 +CPU= 15.550
  03:38:55 STEP 2316 H=2316:00 +CPU= 27.234
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002316
  03:39:12 STEP 2317 H=2317:00 +CPU= 15.846
  03:39:30 STEP 2318 H=2318:00 +CPU= 17.099
  03:39:55 STEP 2319 H=2319:00 +CPU= 24.660
  03:40:12 STEP 2320 H=2320:00 +CPU= 15.891
  03:40:28 STEP 2321 H=2321:00 +CPU= 15.603
  03:40:54 STEP 2322 H=2322:00 +CPU= 25.119
  03:41:10 STEP 2323 H=2323:00 +CPU= 15.843
  03:41:27 STEP 2324 H=2324:00 +CPU= 15.996
  03:41:52 STEP 2325 H=2325:00 +CPU= 24.723
  03:42:09 STEP 2326 H=2326:00 +CPU= 16.083
  03:42:26 STEP 2327 H=2327:00 +CPU= 16.247
  03:42:56 STEP 2328 H=2328:00 +CPU= 29.630
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_96.zip
Uploading the intermediate file: upload_file_96.zip
  03:43:13 STEP 2329 H=2329:00 +CPU= 15.627
  03:43:29 STEP 2330 H=2330:00 +CPU= 15.667
  03:43:55 STEP 2331 H=2331:00 +CPU= 24.493
  03:44:11 STEP 2332 H=2332:00 +CPU= 15.972
  03:44:28 STEP 2333 H=2333:00 +CPU= 15.492
  03:44:54 STEP 2334 H=2334:00 +CPU= 25.008
  03:45:10 STEP 2335 H=2335:00 +CPU= 15.197
  03:45:26 STEP 2336 H=2336:00 +CPU= 15.973
  03:45:52 STEP 2337 H=2337:00 +CPU= 24.817
  03:46:09 STEP 2338 H=2338:00 +CPU= 16.316
  03:46:26 STEP 2339 H=2339:00 +CPU= 15.814
  03:46:55 STEP 2340 H=2340:00 +CPU= 28.258
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002340
  03:47:11 STEP 2341 H=2341:00 +CPU= 15.318
  03:47:27 STEP 2342 H=2342:00 +CPU= 16.291
  03:47:52 STEP 2343 H=2343:00 +CPU= 24.450
  03:48:09 STEP 2344 H=2344:00 +CPU= 15.861
  03:48:25 STEP 2345 H=2345:00 +CPU= 15.229
  03:48:51 STEP 2346 H=2346:00 +CPU= 25.106
  03:49:07 STEP 2347 H=2347:00 +CPU= 15.483
  03:49:23 STEP 2348 H=2348:00 +CPU= 16.004
  03:49:49 STEP 2349 H=2349:00 +CPU= 24.431
  03:50:06 STEP 2350 H=2350:00 +CPU= 16.327
  03:50:23 STEP 2351 H=2351:00 +CPU= 16.365
  03:50:53 STEP 2352 H=2352:00 +CPU= 29.326
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_97.zip
Uploading the intermediate file: upload_file_97.zip
  03:51:10 STEP 2353 H=2353:00 +CPU= 15.353
  03:51:26 STEP 2354 H=2354:00 +CPU= 16.027
  03:51:52 STEP 2355 H=2355:00 +CPU= 24.638
  03:52:08 STEP 2356 H=2356:00 +CPU= 16.007
  03:52:24 STEP 2357 H=2357:00 +CPU= 15.279
  03:52:50 STEP 2358 H=2358:00 +CPU= 25.071
  03:53:06 STEP 2359 H=2359:00 +CPU= 15.840
  03:53:23 STEP 2360 H=2360:00 +CPU= 16.040
  03:53:49 STEP 2361 H=2361:00 +CPU= 24.643
  03:54:06 STEP 2362 H=2362:00 +CPU= 16.280
  03:54:23 STEP 2363 H=2363:00 +CPU= 15.771
  03:54:51 STEP 2364 H=2364:00 +CPU= 27.565
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002364
  03:55:08 STEP 2365 H=2365:00 +CPU= 15.699
  03:55:25 STEP 2366 H=2366:00 +CPU= 16.215
  03:55:50 STEP 2367 H=2367:00 +CPU= 24.490
  03:56:07 STEP 2368 H=2368:00 +CPU= 15.959
  03:56:23 STEP 2369 H=2369:00 +CPU= 15.830
  03:56:50 STEP 2370 H=2370:00 +CPU= 25.460
  03:57:06 STEP 2371 H=2371:00 +CPU= 15.786
  03:57:23 STEP 2372 H=2372:00 +CPU= 16.110
  03:57:48 STEP 2373 H=2373:00 +CPU= 24.400
  03:58:04 STEP 2374 H=2374:00 +CPU= 16.025
  03:58:21 STEP 2375 H=2375:00 +CPU= 16.163
  03:58:52 STEP 2376 H=2376:00 +CPU= 29.318
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_98.zip
Uploading the intermediate file: upload_file_98.zip
  03:59:08 STEP 2377 H=2377:00 +CPU= 15.472
Uploading trickle at timestep: 8553600
  03:59:24 STEP 2378 H=2378:00 +CPU= 16.143
  03:59:50 STEP 2379 H=2379:00 +CPU= 24.266
  04:00:06 STEP 2380 H=2380:00 +CPU= 16.014
  04:00:22 STEP 2381 H=2381:00 +CPU= 15.729
  04:00:48 STEP 2382 H=2382:00 +CPU= 25.351
  04:01:04 STEP 2383 H=2383:00 +CPU= 15.623
  04:01:21 STEP 2384 H=2384:00 +CPU= 15.947
  04:01:46 STEP 2385 H=2385:00 +CPU= 24.352
  04:02:03 STEP 2386 H=2386:00 +CPU= 16.167
  04:02:19 STEP 2387 H=2387:00 +CPU= 15.681
  04:02:47 STEP 2388 H=2388:00 +CPU= 27.110
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002388
  04:03:03 STEP 2389 H=2389:00 +CPU= 15.811
  04:03:20 STEP 2390 H=2390:00 +CPU= 16.091
  04:03:46 STEP 2391 H=2391:00 +CPU= 24.499
  04:04:03 STEP 2392 H=2392:00 +CPU= 16.051
  04:04:19 STEP 2393 H=2393:00 +CPU= 15.603
  04:04:45 STEP 2394 H=2394:00 +CPU= 25.117
  04:05:01 STEP 2395 H=2395:00 +CPU= 15.480
  04:05:18 STEP 2396 H=2396:00 +CPU= 16.089
  04:05:43 STEP 2397 H=2397:00 +CPU= 24.167
  04:06:00 STEP 2398 H=2398:00 +CPU= 16.128
  04:06:17 STEP 2399 H=2399:00 +CPU= 16.042
  04:06:48 STEP 2400 H=2400:00 +CPU= 29.766
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_99.zip
Uploading the intermediate file: upload_file_99.zip
  04:07:04 STEP 2401 H=2401:00 +CPU= 15.727
  04:07:21 STEP 2402 H=2402:00 +CPU= 16.080
  04:07:46 STEP 2403 H=2403:00 +CPU= 24.569
  04:08:03 STEP 2404 H=2404:00 +CPU= 15.912
  04:08:19 STEP 2405 H=2405:00 +CPU= 15.563
  04:08:44 STEP 2406 H=2406:00 +CPU= 24.757
  04:09:01 STEP 2407 H=2407:00 +CPU= 15.587
  04:09:17 STEP 2408 H=2408:00 +CPU= 15.786
  04:09:42 STEP 2409 H=2409:00 +CPU= 24.157
  04:09:59 STEP 2410 H=2410:00 +CPU= 16.299
  04:10:15 STEP 2411 H=2411:00 +CPU= 15.340
  04:10:43 STEP 2412 H=2412:00 +CPU= 27.785
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002412
  04:11:01 STEP 2413 H=2413:00 +CPU= 17.252
  04:11:19 STEP 2414 H=2414:00 +CPU= 17.568
  04:11:44 STEP 2415 H=2415:00 +CPU= 25.126
  04:11:59 STEP 2416 H=2416:00 +CPU= 14.763
  04:12:14 STEP 2417 H=2417:00 +CPU= 14.358
  04:12:37 STEP 2418 H=2418:00 +CPU= 23.229
  04:12:54 STEP 2419 H=2419:00 +CPU= 16.407
  04:13:12 STEP 2420 H=2420:00 +CPU= 17.610
  04:13:42 STEP 2421 H=2421:00 +CPU= 27.738
  04:14:02 STEP 2422 H=2422:00 +CPU= 18.614
  04:14:23 STEP 2423 H=2423:00 +CPU= 18.822
  04:14:58 STEP 2424 H=2424:00 +CPU= 33.521
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_100.zip
Uploading the intermediate file: upload_file_100.zip
  04:15:17 STEP 2425 H=2425:00 +CPU= 17.208
  04:15:36 STEP 2426 H=2426:00 +CPU= 17.740
  04:16:04 STEP 2427 H=2427:00 +CPU= 27.146
  04:16:22 STEP 2428 H=2428:00 +CPU= 17.620
  04:16:41 STEP 2429 H=2429:00 +CPU= 17.667
  04:17:09 STEP 2430 H=2430:00 +CPU= 27.282
  04:17:27 STEP 2431 H=2431:00 +CPU= 17.397
  04:17:45 STEP 2432 H=2432:00 +CPU= 17.373
  04:18:12 STEP 2433 H=2433:00 +CPU= 26.244
  04:18:30 STEP 2434 H=2434:00 +CPU= 17.588
  04:18:50 STEP 2435 H=2435:00 +CPU= 18.267
  04:19:21 STEP 2436 H=2436:00 +CPU= 30.031
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002436
  04:19:40 STEP 2437 H=2437:00 +CPU= 17.185
  04:19:59 STEP 2438 H=2438:00 +CPU= 18.150
  04:20:28 STEP 2439 H=2439:00 +CPU= 26.887
  04:20:46 STEP 2440 H=2440:00 +CPU= 17.347
  04:21:03 STEP 2441 H=2441:00 +CPU= 17.305
  04:21:30 STEP 2442 H=2442:00 +CPU= 26.226
  04:21:52 STEP 2443 H=2443:00 +CPU= 20.660
  04:22:12 STEP 2444 H=2444:00 +CPU= 19.689
  04:22:41 STEP 2445 H=2445:00 +CPU= 28.506
  04:23:02 STEP 2446 H=2446:00 +CPU= 19.329
  04:23:23 STEP 2447 H=2447:00 +CPU= 19.775
  04:23:59 STEP 2448 H=2448:00 +CPU= 34.097
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_101.zip
Uploading the intermediate file: upload_file_101.zip
  04:24:18 STEP 2449 H=2449:00 +CPU= 18.478
  04:24:38 STEP 2450 H=2450:00 +CPU= 19.012
  04:25:06 STEP 2451 H=2451:00 +CPU= 27.289
  04:25:23 STEP 2452 H=2452:00 +CPU= 17.159
  04:25:40 STEP 2453 H=2453:00 +CPU= 16.625
  04:26:08 STEP 2454 H=2454:00 +CPU= 27.323
  04:26:24 STEP 2455 H=2455:00 +CPU= 16.326
  04:26:40 STEP 2456 H=2456:00 +CPU= 15.835
  04:27:05 STEP 2457 H=2457:00 +CPU= 24.054
  04:27:21 STEP 2458 H=2458:00 +CPU= 16.392
  04:27:40 STEP 2459 H=2459:00 +CPU= 18.359
  04:28:13 STEP 2460 H=2460:00 +CPU= 31.266
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002460
  04:28:31 STEP 2461 H=2461:00 +CPU= 18.526
  04:28:52 STEP 2462 H=2462:00 +CPU= 19.261
  04:29:21 STEP 2463 H=2463:00 +CPU= 28.129
  04:29:42 STEP 2464 H=2464:00 +CPU= 19.458
  04:30:02 STEP 2465 H=2465:00 +CPU= 18.292
  04:30:33 STEP 2466 H=2466:00 +CPU= 28.595
  04:30:51 STEP 2467 H=2467:00 +CPU= 18.245
  04:31:11 STEP 2468 H=2468:00 +CPU= 19.007
  04:31:39 STEP 2469 H=2469:00 +CPU= 27.198
  04:31:57 STEP 2470 H=2470:00 +CPU= 18.343
  04:32:17 STEP 2471 H=2471:00 +CPU= 19.038
  04:32:51 STEP 2472 H=2472:00 +CPU= 33.739
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_102.zip
Uploading the intermediate file: upload_file_102.zip
  04:33:08 STEP 2473 H=2473:00 +CPU= 16.515
  04:33:25 STEP 2474 H=2474:00 +CPU= 17.286
  04:33:52 STEP 2475 H=2475:00 +CPU= 26.732
  04:34:10 STEP 2476 H=2476:00 +CPU= 17.221
  04:34:26 STEP 2477 H=2477:00 +CPU= 16.155
  04:34:53 STEP 2478 H=2478:00 +CPU= 26.208
  04:35:11 STEP 2479 H=2479:00 +CPU= 17.683
  04:35:29 STEP 2480 H=2480:00 +CPU= 17.994
  04:35:58 STEP 2481 H=2481:00 +CPU= 28.120
  04:36:18 STEP 2482 H=2482:00 +CPU= 19.418
  04:36:39 STEP 2483 H=2483:00 +CPU= 19.312
  04:37:11 STEP 2484 H=2484:00 +CPU= 31.731
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002484
  04:37:32 STEP 2485 H=2485:00 +CPU= 19.120
  04:37:53 STEP 2486 H=2486:00 +CPU= 19.798
  04:38:22 STEP 2487 H=2487:00 +CPU= 27.774
  04:38:40 STEP 2488 H=2488:00 +CPU= 17.468
  04:38:58 STEP 2489 H=2489:00 +CPU= 17.674
  04:39:27 STEP 2490 H=2490:00 +CPU= 27.368
  04:39:45 STEP 2491 H=2491:00 +CPU= 16.953
  04:40:03 STEP 2492 H=2492:00 +CPU= 17.643
  04:40:32 STEP 2493 H=2493:00 +CPU= 26.950
  04:40:51 STEP 2494 H=2494:00 +CPU= 17.882
  04:41:10 STEP 2495 H=2495:00 +CPU= 17.806
  04:41:43 STEP 2496 H=2496:00 +CPU= 32.319
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_103.zip
Uploading the intermediate file: upload_file_103.zip
  04:42:01 STEP 2497 H=2497:00 +CPU= 17.159
  04:42:20 STEP 2498 H=2498:00 +CPU= 17.807
  04:42:48 STEP 2499 H=2499:00 +CPU= 27.988
  04:43:06 STEP 2500 H=2500:00 +CPU= 17.524
  04:43:24 STEP 2501 H=2501:00 +CPU= 17.496
  04:43:54 STEP 2502 H=2502:00 +CPU= 28.511
  04:44:12 STEP 2503 H=2503:00 +CPU= 17.702
  04:44:32 STEP 2504 H=2504:00 +CPU= 19.261
  04:45:02 STEP 2505 H=2505:00 +CPU= 28.958
  04:45:22 STEP 2506 H=2506:00 +CPU= 19.464
  04:45:40 STEP 2507 H=2507:00 +CPU= 18.464
  04:46:17 STEP 2508 H=2508:00 +CPU= 34.163
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002508
  04:46:38 STEP 2509 H=2509:00 +CPU= 20.515
  04:47:00 STEP 2510 H=2510:00 +CPU= 21.423
  04:47:32 STEP 2511 H=2511:00 +CPU= 30.817
  04:47:54 STEP 2512 H=2512:00 +CPU= 20.962
  04:48:16 STEP 2513 H=2513:00 +CPU= 21.312
  04:48:48 STEP 2514 H=2514:00 +CPU= 30.707
  04:49:08 STEP 2515 H=2515:00 +CPU= 18.999
  04:49:28 STEP 2516 H=2516:00 +CPU= 20.281
  04:49:59 STEP 2517 H=2517:00 +CPU= 30.023
  04:50:20 STEP 2518 H=2518:00 +CPU= 19.988
  04:50:41 STEP 2519 H=2519:00 +CPU= 20.800
  04:51:21 STEP 2520 H=2520:00 +CPU= 37.425
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_104.zip
Uploading the intermediate file: upload_file_104.zip
  04:51:41 STEP 2521 H=2521:00 +CPU= 19.163
  04:52:00 STEP 2522 H=2522:00 +CPU= 18.820
  04:52:29 STEP 2523 H=2523:00 +CPU= 28.386
  04:52:49 STEP 2524 H=2524:00 +CPU= 19.819
  04:53:09 STEP 2525 H=2525:00 +CPU= 19.804
  04:53:39 STEP 2526 H=2526:00 +CPU= 29.744
  04:53:58 STEP 2527 H=2527:00 +CPU= 18.442
  04:54:17 STEP 2528 H=2528:00 +CPU= 18.741
  04:54:46 STEP 2529 H=2529:00 +CPU= 29.014
  04:55:06 STEP 2530 H=2530:00 +CPU= 19.474
  04:55:26 STEP 2531 H=2531:00 +CPU= 20.117
  04:56:02 STEP 2532 H=2532:00 +CPU= 35.135
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002532
  04:56:26 STEP 2533 H=2533:00 +CPU= 22.461
  04:56:53 STEP 2534 H=2534:00 +CPU= 25.616
  04:57:30 STEP 2535 H=2535:00 +CPU= 35.952
  04:57:59 STEP 2536 H=2536:00 +CPU= 28.251
  04:58:29 STEP 2537 H=2537:00 +CPU= 27.919
  04:59:12 STEP 2538 H=2538:00 +CPU= 40.389
  04:59:46 STEP 2539 H=2539:00 +CPU= 29.082
  05:00:20 STEP 2540 H=2540:00 +CPU= 31.482
  05:01:04 STEP 2541 H=2541:00 +CPU= 41.444
  05:01:41 STEP 2542 H=2542:00 +CPU= 33.605
  05:02:20 STEP 2543 H=2543:00 +CPU= 34.611
  05:03:15 STEP 2544 H=2544:00 +CPU= 51.049
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_105.zip
Uploading the intermediate file: upload_file_105.zip
  05:03:39 STEP 2545 H=2545:00 +CPU= 22.996
  05:04:09 STEP 2546 H=2546:00 +CPU= 28.416
  05:04:53 STEP 2547 H=2547:00 +CPU= 42.455
  05:05:24 STEP 2548 H=2548:00 +CPU= 30.819
  05:05:56 STEP 2549 H=2549:00 +CPU= 29.712
  05:06:43 STEP 2550 H=2550:00 +CPU= 45.670
  05:07:11 STEP 2551 H=2551:00 +CPU= 24.868
  05:07:41 STEP 2552 H=2552:00 +CPU= 28.930
  05:08:26 STEP 2553 H=2553:00 +CPU= 42.205
  05:08:58 STEP 2554 H=2554:00 +CPU= 29.441
  05:09:26 STEP 2555 H=2555:00 +CPU= 26.229
  05:10:15 STEP 2556 H=2556:00 +CPU= 45.687
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002556
  05:10:46 STEP 2557 H=2557:00 +CPU= 28.516
  05:11:13 STEP 2558 H=2558:00 +CPU= 25.899
  05:11:53 STEP 2559 H=2559:00 +CPU= 38.508
  05:12:21 STEP 2560 H=2560:00 +CPU= 26.485
  05:12:48 STEP 2561 H=2561:00 +CPU= 25.511
  05:13:31 STEP 2562 H=2562:00 +CPU= 40.768
  05:14:01 STEP 2563 H=2563:00 +CPU= 27.856
  05:14:29 STEP 2564 H=2564:00 +CPU= 26.615
  05:15:08 STEP 2565 H=2565:00 +CPU= 37.832
  05:15:37 STEP 2566 H=2566:00 +CPU= 27.233
  05:16:06 STEP 2567 H=2567:00 +CPU= 27.366
  05:16:56 STEP 2568 H=2568:00 +CPU= 46.851
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_106.zip
Uploading the intermediate file: upload_file_106.zip
  05:17:29 STEP 2569 H=2569:00 +CPU= 30.864
  05:18:00 STEP 2570 H=2570:00 +CPU= 29.159
  05:18:46 STEP 2571 H=2571:00 +CPU= 43.616
  05:19:15 STEP 2572 H=2572:00 +CPU= 27.959
  05:19:48 STEP 2573 H=2573:00 +CPU= 30.100
  05:20:32 STEP 2574 H=2574:00 +CPU= 42.314
  05:20:59 STEP 2575 H=2575:00 +CPU= 25.168
  05:21:28 STEP 2576 H=2576:00 +CPU= 26.657
  05:22:12 STEP 2577 H=2577:00 +CPU= 41.795
  05:22:46 STEP 2578 H=2578:00 +CPU= 31.076
  05:23:17 STEP 2579 H=2579:00 +CPU= 29.300
  05:24:09 STEP 2580 H=2580:00 +CPU= 47.815
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002580
  05:24:39 STEP 2581 H=2581:00 +CPU= 28.642
  05:25:12 STEP 2582 H=2582:00 +CPU= 31.217
  05:25:57 STEP 2583 H=2583:00 +CPU= 41.786
  05:26:25 STEP 2584 H=2584:00 +CPU= 26.368
  05:26:51 STEP 2585 H=2585:00 +CPU= 24.656
  05:27:34 STEP 2586 H=2586:00 +CPU= 40.561
  05:28:01 STEP 2587 H=2587:00 +CPU= 25.683
  05:28:30 STEP 2588 H=2588:00 +CPU= 27.536
  05:29:14 STEP 2589 H=2589:00 +CPU= 40.781
  05:29:41 STEP 2590 H=2590:00 +CPU= 26.109
  05:30:11 STEP 2591 H=2591:00 +CPU= 28.043
  05:31:00 STEP 2592 H=2592:00 +CPU= 46.581
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_107.zip
Uploading the intermediate file: upload_file_107.zip
  05:31:26 STEP 2593 H=2593:00 +CPU= 24.782
  05:31:54 STEP 2594 H=2594:00 +CPU= 25.817
  05:32:31 STEP 2595 H=2595:00 +CPU= 35.995
  05:32:59 STEP 2596 H=2596:00 +CPU= 25.407
  05:33:25 STEP 2597 H=2597:00 +CPU= 24.542
  05:34:11 STEP 2598 H=2598:00 +CPU= 40.832
  05:34:38 STEP 2599 H=2599:00 +CPU= 23.516
  05:35:08 STEP 2600 H=2600:00 +CPU= 27.901
  05:35:51 STEP 2601 H=2601:00 +CPU= 40.005
  05:36:21 STEP 2602 H=2602:00 +CPU= 27.451
  05:36:50 STEP 2603 H=2603:00 +CPU= 27.298
  05:37:36 STEP 2604 H=2604:00 +CPU= 42.670
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002604
  05:38:01 STEP 2605 H=2605:00 +CPU= 23.463
  05:38:28 STEP 2606 H=2606:00 +CPU= 25.691
  05:39:08 STEP 2607 H=2607:00 +CPU= 36.704
  05:39:32 STEP 2608 H=2608:00 +CPU= 22.205
  05:39:56 STEP 2609 H=2609:00 +CPU= 22.761
  05:40:33 STEP 2610 H=2610:00 +CPU= 34.981
  05:40:56 STEP 2611 H=2611:00 +CPU= 22.160
  05:41:20 STEP 2612 H=2612:00 +CPU= 22.953
  05:41:55 STEP 2613 H=2613:00 +CPU= 33.518
  05:42:18 STEP 2614 H=2614:00 +CPU= 21.686
  05:42:42 STEP 2615 H=2615:00 +CPU= 22.549
  05:43:24 STEP 2616 H=2616:00 +CPU= 39.951
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_108.zip
Uploading the intermediate file: upload_file_108.zip
  05:43:45 STEP 2617 H=2617:00 +CPU= 20.550
Uploading trickle at timestep: 9417600
  05:44:09 STEP 2618 H=2618:00 +CPU= 22.430
  05:44:46 STEP 2619 H=2619:00 +CPU= 34.912
  05:45:09 STEP 2620 H=2620:00 +CPU= 22.141
  05:45:34 STEP 2621 H=2621:00 +CPU= 22.904
  05:46:12 STEP 2622 H=2622:00 +CPU= 35.905
  05:46:34 STEP 2623 H=2623:00 +CPU= 21.793
  05:46:59 STEP 2624 H=2624:00 +CPU= 23.521
  05:47:37 STEP 2625 H=2625:00 +CPU= 36.003
  05:48:00 STEP 2626 H=2626:00 +CPU= 22.278
  05:48:24 STEP 2627 H=2627:00 +CPU= 22.775
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0323
(argv4) batchid: 974
(argv5) wuid: 12190967
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/29
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/29/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/29/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_11cadc15e9236c82921edd22cad9855a to: /var/lib/boinc-client/slots/29/oifs_43r3_ps_0323_2005050100_123_974_12190967.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/29/oifs_43r3_ps_0323_2005050100_123_974_12190967.zip
ic_ancil_file: ic_ancil_12190967
ifsdata_file: ifsdata_12190967
climate_data_file: clim_data_12190967
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d89aa46b3aaeeec1f630bcc3187a91eb to: /var/lib/boinc-client/slots/29/ic_ancil_12190967.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/29/ic_ancil_12190967.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_57402246f807196adb631dcc16fca0c1 to: /var/lib/boinc-client/slots/29/ifsdata/ifsdata_12190967.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/29/ifsdata/ifsdata_12190967.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/29/159l_2/clim_data_12190967.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/29/159l_2/clim_data_12190967.zip
Checking for progress XML file: /var/lib/boinc-client/slots/29/progress_file_12190967.xml
Opened progress file ok : /var/lib/boinc-client/slots/29/progress_file_12190967.xml
-- Model is restarting --
Adjusting last_iter, 2628, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/29/progress_file_12190967.xml
last_cpu_time: 62258
upload_file_number: 109
last_iter: 2615
last_upload: 9417600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587
The child process has been launched with process id: 61481
Executing the command: /var/lib/boinc-client/slots/29/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:61481:61481] [20230102:055342:1672599222:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:54:27 STEP 2616 H=2616:00 +CPU= 40.451
  05:54:52 STEP 2617 H=2617:00 +CPU= 23.076
  05:55:13 STEP 2618 H=2618:00 +CPU= 21.368
  05:55:50 STEP 2619 H=2619:00 +CPU= 35.305
  05:56:15 STEP 2620 H=2620:00 +CPU= 23.685
  05:56:39 STEP 2621 H=2621:00 +CPU= 22.711
  05:57:17 STEP 2622 H=2622:00 +CPU= 36.202
  05:57:41 STEP 2623 H=2623:00 +CPU= 22.687
  05:58:05 STEP 2624 H=2624:00 +CPU= 22.632
  05:58:40 STEP 2625 H=2625:00 +CPU= 34.250
  05:59:04 STEP 2626 H=2626:00 +CPU= 22.263
  05:59:26 STEP 2627 H=2627:00 +CPU= 22.174
  06:00:05 STEP 2628 H=2628:00 +CPU= 37.582
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002628
  06:00:29 STEP 2629 H=2629:00 +CPU= 22.328
  06:00:53 STEP 2630 H=2630:00 +CPU= 22.824
  06:01:28 STEP 2631 H=2631:00 +CPU= 33.760
  06:01:52 STEP 2632 H=2632:00 +CPU= 23.075
  06:02:16 STEP 2633 H=2633:00 +CPU= 22.943
  06:02:54 STEP 2634 H=2634:00 +CPU= 35.472
  06:03:16 STEP 2635 H=2635:00 +CPU= 21.634
  06:03:42 STEP 2636 H=2636:00 +CPU= 23.674
  06:04:19 STEP 2637 H=2637:00 +CPU= 34.244
  06:04:44 STEP 2638 H=2638:00 +CPU= 23.292
  06:05:06 STEP 2639 H=2639:00 +CPU= 21.564
  06:05:53 STEP 2640 H=2640:00 +CPU= 43.499
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190967/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0323_2005050100_123_974_12190967_0_r292651587_109.zip
Uploading the intermediate file: upload_file_109.zip
  06:06:20 STEP 2641 H=2641:00 +CPU= 24.484
  06:06:48 STEP 2642 H=2642:00 +CPU= 25.785
  06:07:28 STEP 2643 H=2643:00 +CPU= 38.526
  06:07:55 STEP 2644 H=2644:00 +CPU= 25.897
  06:08:25 STEP 2645 H=2645:00 +CPU= 25.797
  06:09:04 STEP 2646 H=2646:00 +CPU= 38.435
  06:09:30 STEP 2647 H=2647:00 +CPU= 25.031
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:05:21 1520317 22276623 oifs_43r3_ps_0323_2005050100_123_974_12190967_0 9,417,600 61,950 0.0066
09 May 2023 16:05:21 1520317 22276623 oifs_43r3_ps_0323_2005050100_123_974_12190967_0 9,417,600 61,950 0.0066
09 May 2023 16:05:18 1520317 22276623 oifs_43r3_ps_0323_2005050100_123_974_12190967_0 8,553,600 55,972 0.0065
09 May 2023 15:53:45 1520317 22276623 oifs_43r3_ps_0323_2005050100_123_974_12190967_0 6,739,200 45,358 0.0067
09 May 2023 15:53:15 1520317 22276623 oifs_43r3_ps_0323_2005050100_123_974_12190967_0 5,875,200 39,676 0.0068
09 May 2023 15:53:01 1520317 22276623 oifs_43r3_ps_0323_2005050100_123_974_12190967_0 5,011,200 34,116 0.0068
09 May 2023 15:52:49 1520317 22276623 oifs_43r3_ps_0323_2005050100_123_974_12190967_0 4,147,200 28,128 0.0068
09 May 2023 15:52:36 1520317 22276623 oifs_43r3_ps_0323_2005050100_123_974_12190967_0 3,283,200 21,581 0.0066
09 May 2023 15:52:14 1520317 22276623 oifs_43r3_ps_0323_2005050100_123_974_12190967_0 2,419,200 15,120 0.0063
09 May 2023 15:52:02 1520317 22276623 oifs_43r3_ps_0323_2005050100_123_974_12190967_0 1,555,200 9,786 0.0063


©2024 climateprediction.net