climateprediction.net home page
Task 22275241

Task 22275241

Name oifs_43r3_ps_0046_2004050100_123_973_12189690_0
Workunit 12189690
Created 23 Dec 2022, 6:56:41 UTC
Sent 23 Dec 2022, 10:46:15 UTC
Report deadline 22 Jan 2023, 10:46:15 UTC
Received 1 Jan 2023, 19:44:17 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1520317
Run time 21 hours 44 min 14 sec
CPU time 17 hours 37 min 20 sec
Validate state Invalid
Credit 13,184.64
Device peak FLOPS 8.55 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.14 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,751.96 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
554616:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:29617:29617] [20230101:173016:1672554616:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:29617:29617] [20230101:173016:1672554616:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:29617:29617] [20230101:173016:1672554616:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:29617:29617] [20230101:173016:1672554616:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:29617:29617] [20230101:173016:1672554616:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:30:51 STEP 2136 H=2136:00 +CPU= 29.184
  17:31:09 STEP 2137 H=2137:00 +CPU= 16.538
  17:31:28 STEP 2138 H=2138:00 +CPU= 16.916
  17:31:56 STEP 2139 H=2139:00 +CPU= 25.452
  17:32:14 STEP 2140 H=2140:00 +CPU= 16.700
  17:32:32 STEP 2141 H=2141:00 +CPU= 16.213
  17:33:00 STEP 2142 H=2142:00 +CPU= 25.938
  17:33:18 STEP 2143 H=2143:00 +CPU= 16.266
  17:33:36 STEP 2144 H=2144:00 +CPU= 16.769
  17:34:04 STEP 2145 H=2145:00 +CPU= 25.288
  17:34:22 STEP 2146 H=2146:00 +CPU= 16.570
  17:34:39 STEP 2147 H=2147:00 +CPU= 16.089
  17:35:11 STEP 2148 H=2148:00 +CPU= 28.320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002148
  17:35:29 STEP 2149 H=2149:00 +CPU= 16.564
  17:35:47 STEP 2150 H=2150:00 +CPU= 16.932
  17:36:15 STEP 2151 H=2151:00 +CPU= 25.253
  17:36:33 STEP 2152 H=2152:00 +CPU= 16.813
  17:36:51 STEP 2153 H=2153:00 +CPU= 16.200
  17:37:19 STEP 2154 H=2154:00 +CPU= 25.724
  17:37:37 STEP 2155 H=2155:00 +CPU= 16.213
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=            7  MAX ETADOT VERTICAL VEL.=   5.664919276836459E-008
  LON  =    176.250000000000       degrees
  LAT  =   -35.3269057618723       degrees
  17:37:55 STEP 2156 H=2156:00 +CPU= 16.447
  17:38:22 STEP 2157 H=2157:00 +CPU= 25.130
  17:38:40 STEP 2158 H=2158:00 +CPU= 16.398
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:06:27 STEP 2159 H=2159:00 +CPU= 16.643
  22:07:02 STEP 2160 H=2160:00 +CPU= 30.621
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_89.zip
Uploading the intermediate file: upload_file_89.zip
  22:07:20 STEP 2161 H=2161:00 +CPU= 16.694
  22:07:39 STEP 2162 H=2162:00 +CPU= 16.755
  22:08:08 STEP 2163 H=2163:00 +CPU= 26.230
  22:08:27 STEP 2164 H=2164:00 +CPU= 16.988
  22:08:46 STEP 2165 H=2165:00 +CPU= 16.612
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:36:51 STEP 2166 H=2166:00 +CPU= 25.948
  01:37:08 STEP 2167 H=2167:00 +CPU= 16.231
  01:37:25 STEP 2168 H=2168:00 +CPU= 16.302
  01:37:51 STEP 2169 H=2169:00 +CPU= 24.716
  01:38:07 STEP 2170 H=2170:00 +CPU= 16.276
  01:38:24 STEP 2171 H=2171:00 +CPU= 15.960
  01:38:52 STEP 2172 H=2172:00 +CPU= 27.476
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002172
  01:39:09 STEP 2173 H=2173:00 +CPU= 16.119
  01:39:26 STEP 2174 H=2174:00 +CPU= 16.352
  01:39:52 STEP 2175 H=2175:00 +CPU= 24.762
  01:40:09 STEP 2176 H=2176:00 +CPU= 16.351
  01:40:26 STEP 2177 H=2177:00 +CPU= 16.183
  01:40:52 STEP 2178 H=2178:00 +CPU= 25.267
  01:41:09 STEP 2179 H=2179:00 +CPU= 16.102
  01:41:26 STEP 2180 H=2180:00 +CPU= 16.613
  01:41:52 STEP 2181 H=2181:00 +CPU= 24.568
  01:42:09 STEP 2182 H=2182:00 +CPU= 16.361
  01:42:26 STEP 2183 H=2183:00 +CPU= 16.320
  01:42:57 STEP 2184 H=2184:00 +CPU= 29.939
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_90.zip
Uploading the intermediate file: upload_file_90.zip
  01:43:14 STEP 2185 H=2185:00 +CPU= 15.709
  01:43:31 STEP 2186 H=2186:00 +CPU= 16.132
  01:43:57 STEP 2187 H=2187:00 +CPU= 24.991
  01:44:14 STEP 2188 H=2188:00 +CPU= 16.268
  01:44:31 STEP 2189 H=2189:00 +CPU= 16.682
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:57:18 STEP 2190 H=2190:00 +CPU= 24.975
  02:57:34 STEP 2191 H=2191:00 +CPU= 15.910
  02:57:51 STEP 2192 H=2192:00 +CPU= 16.063
  02:58:16 STEP 2193 H=2193:00 +CPU= 24.307
  02:58:33 STEP 2194 H=2194:00 +CPU= 16.206
  02:58:49 STEP 2195 H=2195:00 +CPU= 15.708
  02:59:17 STEP 2196 H=2196:00 +CPU= 27.032
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002196
  02:59:33 STEP 2197 H=2197:00 +CPU= 15.783
  02:59:50 STEP 2198 H=2198:00 +CPU= 16.125
  03:00:15 STEP 2199 H=2199:00 +CPU= 24.223
  03:00:32 STEP 2200 H=2200:00 +CPU= 15.977
  03:00:48 STEP 2201 H=2201:00 +CPU= 15.549
  03:01:14 STEP 2202 H=2202:00 +CPU= 24.983
  03:01:30 STEP 2203 H=2203:00 +CPU= 15.408
  03:01:47 STEP 2204 H=2204:00 +CPU= 16.157
  03:02:13 STEP 2205 H=2205:00 +CPU= 24.479
  03:02:29 STEP 2206 H=2206:00 +CPU= 16.053
  03:02:46 STEP 2207 H=2207:00 +CPU= 16.141
  03:03:17 STEP 2208 H=2208:00 +CPU= 29.480
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_91.zip
Uploading the intermediate file: upload_file_91.zip
  03:03:33 STEP 2209 H=2209:00 +CPU= 15.534
  03:03:50 STEP 2210 H=2210:00 +CPU= 16.171
  03:04:16 STEP 2211 H=2211:00 +CPU= 25.672
  03:04:33 STEP 2212 H=2212:00 +CPU= 16.067
  03:04:49 STEP 2213 H=2213:00 +CPU= 15.632
  03:05:15 STEP 2214 H=2214:00 +CPU= 25.268
  03:05:31 STEP 2215 H=2215:00 +CPU= 15.769
  03:05:48 STEP 2216 H=2216:00 +CPU= 16.161
  03:06:13 STEP 2217 H=2217:00 +CPU= 24.734
  03:06:30 STEP 2218 H=2218:00 +CPU= 15.912
  03:06:46 STEP 2219 H=2219:00 +CPU= 15.932
  03:07:14 STEP 2220 H=2220:00 +CPU= 27.024
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002220
  03:07:30 STEP 2221 H=2221:00 +CPU= 15.707
  03:07:47 STEP 2222 H=2222:00 +CPU= 15.976
  03:08:12 STEP 2223 H=2223:00 +CPU= 24.479
  03:08:29 STEP 2224 H=2224:00 +CPU= 16.289
  03:08:45 STEP 2225 H=2225:00 +CPU= 15.711
  03:09:11 STEP 2226 H=2226:00 +CPU= 25.259
  03:09:27 STEP 2227 H=2227:00 +CPU= 15.351
  03:09:43 STEP 2228 H=2228:00 +CPU= 16.077
  03:10:09 STEP 2229 H=2229:00 +CPU= 24.545
  03:10:25 STEP 2230 H=2230:00 +CPU= 15.796
  03:10:42 STEP 2231 H=2231:00 +CPU= 16.244
  03:11:13 STEP 2232 H=2232:00 +CPU= 29.754
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_92.zip
Uploading the intermediate file: upload_file_92.zip
  03:11:29 STEP 2233 H=2233:00 +CPU= 15.644
  03:11:46 STEP 2234 H=2234:00 +CPU= 15.858
  03:12:11 STEP 2235 H=2235:00 +CPU= 24.298
  03:12:28 STEP 2236 H=2236:00 +CPU= 15.818
  03:12:44 STEP 2237 H=2237:00 +CPU= 15.619
  03:13:10 STEP 2238 H=2238:00 +CPU= 24.925
  03:13:27 STEP 2239 H=2239:00 +CPU= 15.917
  03:13:43 STEP 2240 H=2240:00 +CPU= 16.097
  03:14:09 STEP 2241 H=2241:00 +CPU= 24.877
  03:14:26 STEP 2242 H=2242:00 +CPU= 16.285
  03:14:43 STEP 2243 H=2243:00 +CPU= 15.899
  03:15:11 STEP 2244 H=2244:00 +CPU= 27.507
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002244
  03:15:27 STEP 2245 H=2245:00 +CPU= 15.547
  03:15:44 STEP 2246 H=2246:00 +CPU= 16.371
  03:16:10 STEP 2247 H=2247:00 +CPU= 24.508
  03:16:26 STEP 2248 H=2248:00 +CPU= 16.159
  03:16:42 STEP 2249 H=2249:00 +CPU= 15.497
  03:17:08 STEP 2250 H=2250:00 +CPU= 25.047
  03:17:25 STEP 2251 H=2251:00 +CPU= 15.707
  03:17:41 STEP 2252 H=2252:00 +CPU= 16.128
  03:18:07 STEP 2253 H=2253:00 +CPU= 24.386
  03:18:23 STEP 2254 H=2254:00 +CPU= 15.988
  03:18:40 STEP 2255 H=2255:00 +CPU= 15.944
  03:19:11 STEP 2256 H=2256:00 +CPU= 29.537
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_93.zip
Uploading the intermediate file: upload_file_93.zip
  03:19:27 STEP 2257 H=2257:00 +CPU= 15.527
  03:19:44 STEP 2258 H=2258:00 +CPU= 16.104
  03:20:09 STEP 2259 H=2259:00 +CPU= 24.015
  03:20:25 STEP 2260 H=2260:00 +CPU= 15.718
  03:20:41 STEP 2261 H=2261:00 +CPU= 15.371
  03:21:07 STEP 2262 H=2262:00 +CPU= 25.106
  03:21:24 STEP 2263 H=2263:00 +CPU= 15.804
  03:21:41 STEP 2264 H=2264:00 +CPU= 15.998
  03:22:06 STEP 2265 H=2265:00 +CPU= 24.675
  03:22:23 STEP 2266 H=2266:00 +CPU= 16.209
  03:22:39 STEP 2267 H=2267:00 +CPU= 15.716
  03:23:08 STEP 2268 H=2268:00 +CPU= 27.560
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002268
  03:23:24 STEP 2269 H=2269:00 +CPU= 15.569
  03:23:41 STEP 2270 H=2270:00 +CPU= 16.085
  03:24:06 STEP 2271 H=2271:00 +CPU= 24.484
  03:24:23 STEP 2272 H=2272:00 +CPU= 16.108
  03:24:39 STEP 2273 H=2273:00 +CPU= 15.938
  03:25:05 STEP 2274 H=2274:00 +CPU= 24.671
  03:25:21 STEP 2275 H=2275:00 +CPU= 15.357
  03:25:37 STEP 2276 H=2276:00 +CPU= 15.988
  03:26:03 STEP 2277 H=2277:00 +CPU= 24.399
  03:26:19 STEP 2278 H=2278:00 +CPU= 16.004
  03:26:36 STEP 2279 H=2279:00 +CPU= 16.032
  03:27:06 STEP 2280 H=2280:00 +CPU= 28.988
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_94.zip
Uploading the intermediate file: upload_file_94.zip
  03:27:22 STEP 2281 H=2281:00 +CPU= 15.510
  03:27:39 STEP 2282 H=2282:00 +CPU= 16.042
  03:28:04 STEP 2283 H=2283:00 +CPU= 24.376
  03:28:20 STEP 2284 H=2284:00 +CPU= 16.088
  03:28:36 STEP 2285 H=2285:00 +CPU= 15.433
  03:29:02 STEP 2286 H=2286:00 +CPU= 24.794
  03:29:18 STEP 2287 H=2287:00 +CPU= 15.445
  03:29:34 STEP 2288 H=2288:00 +CPU= 15.606
  03:29:59 STEP 2289 H=2289:00 +CPU= 24.183
  03:30:16 STEP 2290 H=2290:00 +CPU= 16.028
  03:30:32 STEP 2291 H=2291:00 +CPU= 15.552
  03:31:00 STEP 2292 H=2292:00 +CPU= 27.273
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002292
  03:31:17 STEP 2293 H=2293:00 +CPU= 15.664
  03:31:33 STEP 2294 H=2294:00 +CPU= 16.164
  03:31:59 STEP 2295 H=2295:00 +CPU= 25.420
  03:32:16 STEP 2296 H=2296:00 +CPU= 16.321
  03:32:34 STEP 2297 H=2297:00 +CPU= 16.801
  03:33:00 STEP 2298 H=2298:00 +CPU= 25.192
  03:33:16 STEP 2299 H=2299:00 +CPU= 15.791
  03:33:32 STEP 2300 H=2300:00 +CPU= 16.034
  03:33:57 STEP 2301 H=2301:00 +CPU= 24.146
  03:34:14 STEP 2302 H=2302:00 +CPU= 15.854
  03:34:30 STEP 2303 H=2303:00 +CPU= 16.003
  03:35:01 STEP 2304 H=2304:00 +CPU= 29.708
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_95.zip
Uploading the intermediate file: upload_file_95.zip
  03:35:17 STEP 2305 H=2305:00 +CPU= 15.579
  03:35:34 STEP 2306 H=2306:00 +CPU= 16.118
  03:35:59 STEP 2307 H=2307:00 +CPU= 24.402
  03:36:16 STEP 2308 H=2308:00 +CPU= 15.906
  03:36:32 STEP 2309 H=2309:00 +CPU= 15.422
  03:36:58 STEP 2310 H=2310:00 +CPU= 24.751
  03:37:14 STEP 2311 H=2311:00 +CPU= 15.469
  03:37:30 STEP 2312 H=2312:00 +CPU= 15.926
  03:37:56 STEP 2313 H=2313:00 +CPU= 24.564
  03:38:13 STEP 2314 H=2314:00 +CPU= 16.032
  03:38:29 STEP 2315 H=2315:00 +CPU= 15.694
  03:38:58 STEP 2316 H=2316:00 +CPU= 27.667
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002316
  03:39:15 STEP 2317 H=2317:00 +CPU= 15.967
  03:39:32 STEP 2318 H=2318:00 +CPU= 16.619
  03:39:57 STEP 2319 H=2319:00 +CPU= 24.772
  03:40:14 STEP 2320 H=2320:00 +CPU= 16.674
  03:40:31 STEP 2321 H=2321:00 +CPU= 15.713
  03:40:57 STEP 2322 H=2322:00 +CPU= 25.274
  03:41:13 STEP 2323 H=2323:00 +CPU= 15.729
  03:41:30 STEP 2324 H=2324:00 +CPU= 16.143
  03:41:55 STEP 2325 H=2325:00 +CPU= 24.590
  03:42:12 STEP 2326 H=2326:00 +CPU= 16.140
  03:42:29 STEP 2327 H=2327:00 +CPU= 16.291
  03:42:59 STEP 2328 H=2328:00 +CPU= 29.457
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_96.zip
Uploading the intermediate file: upload_file_96.zip
  03:43:15 STEP 2329 H=2329:00 +CPU= 15.606
  03:43:32 STEP 2330 H=2330:00 +CPU= 16.072
  03:43:58 STEP 2331 H=2331:00 +CPU= 24.802
  03:44:14 STEP 2332 H=2332:00 +CPU= 15.894
  03:44:31 STEP 2333 H=2333:00 +CPU= 15.744
  03:44:56 STEP 2334 H=2334:00 +CPU= 24.703
  03:45:13 STEP 2335 H=2335:00 +CPU= 15.661
  03:45:29 STEP 2336 H=2336:00 +CPU= 15.941
  03:45:55 STEP 2337 H=2337:00 +CPU= 24.639
  03:46:11 STEP 2338 H=2338:00 +CPU= 15.946
  03:46:28 STEP 2339 H=2339:00 +CPU= 16.167
  03:46:57 STEP 2340 H=2340:00 +CPU= 27.475
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002340
  03:47:13 STEP 2341 H=2341:00 +CPU= 15.889
  03:47:30 STEP 2342 H=2342:00 +CPU= 16.041
  03:47:55 STEP 2343 H=2343:00 +CPU= 24.404
  03:48:12 STEP 2344 H=2344:00 +CPU= 16.159
  03:48:28 STEP 2345 H=2345:00 +CPU= 15.569
  03:48:54 STEP 2346 H=2346:00 +CPU= 25.159
  03:49:10 STEP 2347 H=2347:00 +CPU= 15.579
  03:49:27 STEP 2348 H=2348:00 +CPU= 16.001
  03:49:52 STEP 2349 H=2349:00 +CPU= 24.659
  03:50:09 STEP 2350 H=2350:00 +CPU= 16.270
  03:50:26 STEP 2351 H=2351:00 +CPU= 16.075
  03:50:56 STEP 2352 H=2352:00 +CPU= 29.208
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_97.zip
Uploading the intermediate file: upload_file_97.zip
  03:51:12 STEP 2353 H=2353:00 +CPU= 15.355
  03:51:28 STEP 2354 H=2354:00 +CPU= 15.765
  03:51:54 STEP 2355 H=2355:00 +CPU= 24.779
  03:52:10 STEP 2356 H=2356:00 +CPU= 16.196
  03:52:26 STEP 2357 H=2357:00 +CPU= 15.349
  03:52:52 STEP 2358 H=2358:00 +CPU= 25.075
  03:53:08 STEP 2359 H=2359:00 +CPU= 15.661
  03:53:25 STEP 2360 H=2360:00 +CPU= 15.930
  03:53:50 STEP 2361 H=2361:00 +CPU= 24.286
  03:54:07 STEP 2362 H=2362:00 +CPU= 16.222
  03:54:23 STEP 2363 H=2363:00 +CPU= 15.784
  03:54:52 STEP 2364 H=2364:00 +CPU= 27.172
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002364
  03:55:08 STEP 2365 H=2365:00 +CPU= 15.632
  03:55:24 STEP 2366 H=2366:00 +CPU= 15.911
  03:55:49 STEP 2367 H=2367:00 +CPU= 24.425
  03:56:06 STEP 2368 H=2368:00 +CPU= 16.177
  03:56:23 STEP 2369 H=2369:00 +CPU= 15.654
  03:56:49 STEP 2370 H=2370:00 +CPU= 25.266
  03:57:05 STEP 2371 H=2371:00 +CPU= 15.975
  03:57:22 STEP 2372 H=2372:00 +CPU= 16.280
  03:57:48 STEP 2373 H=2373:00 +CPU= 24.530
  03:58:04 STEP 2374 H=2374:00 +CPU= 16.024
  03:58:21 STEP 2375 H=2375:00 +CPU= 16.287
  03:58:51 STEP 2376 H=2376:00 +CPU= 29.705
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_98.zip
Uploading the intermediate file: upload_file_98.zip
  03:59:08 STEP 2377 H=2377:00 +CPU= 15.547
Uploading trickle at timestep: 8553600
  03:59:24 STEP 2378 H=2378:00 +CPU= 16.293
  03:59:49 STEP 2379 H=2379:00 +CPU= 24.164
  04:00:06 STEP 2380 H=2380:00 +CPU= 15.919
  04:00:22 STEP 2381 H=2381:00 +CPU= 15.519
  04:00:48 STEP 2382 H=2382:00 +CPU= 24.765
  04:01:03 STEP 2383 H=2383:00 +CPU= 15.341
  04:01:20 STEP 2384 H=2384:00 +CPU= 15.895
  04:01:45 STEP 2385 H=2385:00 +CPU= 24.447
  04:02:02 STEP 2386 H=2386:00 +CPU= 16.043
  04:02:18 STEP 2387 H=2387:00 +CPU= 15.606
  04:02:47 STEP 2388 H=2388:00 +CPU= 27.655
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002388
  04:03:03 STEP 2389 H=2389:00 +CPU= 15.477
  04:03:20 STEP 2390 H=2390:00 +CPU= 16.249
  04:03:45 STEP 2391 H=2391:00 +CPU= 24.503
  04:04:02 STEP 2392 H=2392:00 +CPU= 15.997
  04:04:18 STEP 2393 H=2393:00 +CPU= 15.569
  04:04:44 STEP 2394 H=2394:00 +CPU= 25.174
  04:05:00 STEP 2395 H=2395:00 +CPU= 15.254
  04:05:16 STEP 2396 H=2396:00 +CPU= 15.825
  04:05:42 STEP 2397 H=2397:00 +CPU= 24.231
  04:05:58 STEP 2398 H=2398:00 +CPU= 15.745
  04:06:15 STEP 2399 H=2399:00 +CPU= 15.808
  04:06:45 STEP 2400 H=2400:00 +CPU= 29.552
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_99.zip
Uploading the intermediate file: upload_file_99.zip
  04:07:02 STEP 2401 H=2401:00 +CPU= 15.689
  04:07:19 STEP 2402 H=2402:00 +CPU= 16.357
  04:07:45 STEP 2403 H=2403:00 +CPU= 24.822
  04:08:01 STEP 2404 H=2404:00 +CPU= 16.088
  04:08:17 STEP 2405 H=2405:00 +CPU= 15.321
  04:08:43 STEP 2406 H=2406:00 +CPU= 24.830
  04:08:59 STEP 2407 H=2407:00 +CPU= 15.321
  04:09:15 STEP 2408 H=2408:00 +CPU= 16.041
  04:09:40 STEP 2409 H=2409:00 +CPU= 24.403
  04:09:57 STEP 2410 H=2410:00 +CPU= 15.865
  04:10:13 STEP 2411 H=2411:00 +CPU= 15.289
  04:10:42 STEP 2412 H=2412:00 +CPU= 27.933
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002412
  04:11:00 STEP 2413 H=2413:00 +CPU= 17.090
  04:11:18 STEP 2414 H=2414:00 +CPU= 17.552
  04:11:44 STEP 2415 H=2415:00 +CPU= 25.683
  04:12:00 STEP 2416 H=2416:00 +CPU= 15.137
  04:12:15 STEP 2417 H=2417:00 +CPU= 14.743
  04:12:37 STEP 2418 H=2418:00 +CPU= 22.398
  04:12:54 STEP 2419 H=2419:00 +CPU= 16.111
  04:13:14 STEP 2420 H=2420:00 +CPU= 18.600
  04:13:44 STEP 2421 H=2421:00 +CPU= 27.860
  04:14:04 STEP 2422 H=2422:00 +CPU= 18.476
  04:14:24 STEP 2423 H=2423:00 +CPU= 18.794
  04:15:00 STEP 2424 H=2424:00 +CPU= 33.760
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_100.zip
Uploading the intermediate file: upload_file_100.zip
  04:15:20 STEP 2425 H=2425:00 +CPU= 17.614
  04:15:38 STEP 2426 H=2426:00 +CPU= 17.891
  04:16:06 STEP 2427 H=2427:00 +CPU= 26.296
  04:16:25 STEP 2428 H=2428:00 +CPU= 18.009
  04:16:43 STEP 2429 H=2429:00 +CPU= 17.810
  04:17:12 STEP 2430 H=2430:00 +CPU= 27.381
  04:17:29 STEP 2431 H=2431:00 +CPU= 17.058
  04:17:47 STEP 2432 H=2432:00 +CPU= 17.421
  04:18:15 STEP 2433 H=2433:00 +CPU= 26.928
  04:18:35 STEP 2434 H=2434:00 +CPU= 18.551
  04:18:53 STEP 2435 H=2435:00 +CPU= 17.489
  04:19:24 STEP 2436 H=2436:00 +CPU= 29.668
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002436
  04:19:43 STEP 2437 H=2437:00 +CPU= 17.316
  04:20:01 STEP 2438 H=2438:00 +CPU= 17.916
  04:20:29 STEP 2439 H=2439:00 +CPU= 26.715
  04:20:48 STEP 2440 H=2440:00 +CPU= 18.387
  04:21:06 STEP 2441 H=2441:00 +CPU= 17.093
  04:21:32 STEP 2442 H=2442:00 +CPU= 25.867
  04:21:53 STEP 2443 H=2443:00 +CPU= 20.693
  04:22:13 STEP 2444 H=2444:00 +CPU= 20.184
  04:22:43 STEP 2445 H=2445:00 +CPU= 29.489
  04:23:03 STEP 2446 H=2446:00 +CPU= 19.456
  04:23:25 STEP 2447 H=2447:00 +CPU= 19.734
  04:24:01 STEP 2448 H=2448:00 +CPU= 34.296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_101.zip
Uploading the intermediate file: upload_file_101.zip
  04:24:20 STEP 2449 H=2449:00 +CPU= 18.492
  04:24:40 STEP 2450 H=2450:00 +CPU= 19.218
  04:25:08 STEP 2451 H=2451:00 +CPU= 27.393
  04:25:25 STEP 2452 H=2452:00 +CPU= 17.281
  04:25:42 STEP 2453 H=2453:00 +CPU= 16.778
  04:26:10 STEP 2454 H=2454:00 +CPU= 27.307
  04:26:25 STEP 2455 H=2455:00 +CPU= 15.447
  04:26:41 STEP 2456 H=2456:00 +CPU= 15.775
  04:27:06 STEP 2457 H=2457:00 +CPU= 23.936
  04:27:23 STEP 2458 H=2458:00 +CPU= 16.905
  04:27:42 STEP 2459 H=2459:00 +CPU= 17.905
  04:28:15 STEP 2460 H=2460:00 +CPU= 31.681
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002460
  04:28:34 STEP 2461 H=2461:00 +CPU= 18.600
  04:28:54 STEP 2462 H=2462:00 +CPU= 19.296
  04:29:26 STEP 2463 H=2463:00 +CPU= 28.935
  04:29:46 STEP 2464 H=2464:00 +CPU= 19.407
  04:30:05 STEP 2465 H=2465:00 +CPU= 18.670
  04:30:35 STEP 2466 H=2466:00 +CPU= 29.432
  04:30:53 STEP 2467 H=2467:00 +CPU= 18.141
  04:31:13 STEP 2468 H=2468:00 +CPU= 18.383
  04:31:40 STEP 2469 H=2469:00 +CPU= 27.488
  04:31:59 STEP 2470 H=2470:00 +CPU= 18.170
  04:32:18 STEP 2471 H=2471:00 +CPU= 18.661
  04:32:54 STEP 2472 H=2472:00 +CPU= 34.322
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_102.zip
Uploading the intermediate file: upload_file_102.zip
  04:33:11 STEP 2473 H=2473:00 +CPU= 16.354
  04:33:28 STEP 2474 H=2474:00 +CPU= 17.723
  04:33:55 STEP 2475 H=2475:00 +CPU= 26.285
  04:34:13 STEP 2476 H=2476:00 +CPU= 17.363
  04:34:30 STEP 2477 H=2477:00 +CPU= 16.765
  04:34:57 STEP 2478 H=2478:00 +CPU= 27.035
  04:35:16 STEP 2479 H=2479:00 +CPU= 17.925
  04:35:33 STEP 2480 H=2480:00 +CPU= 17.642
  04:36:02 STEP 2481 H=2481:00 +CPU= 27.654
  04:36:23 STEP 2482 H=2482:00 +CPU= 19.537
  04:36:43 STEP 2483 H=2483:00 +CPU= 18.734
  04:37:16 STEP 2484 H=2484:00 +CPU= 32.031
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002484
  04:37:36 STEP 2485 H=2485:00 +CPU= 19.267
  04:37:57 STEP 2486 H=2486:00 +CPU= 19.576
  04:38:25 STEP 2487 H=2487:00 +CPU= 27.108
  04:38:44 STEP 2488 H=2488:00 +CPU= 18.066
  04:39:01 STEP 2489 H=2489:00 +CPU= 17.084
  04:39:30 STEP 2490 H=2490:00 +CPU= 27.426
  04:39:47 STEP 2491 H=2491:00 +CPU= 17.109
  04:40:05 STEP 2492 H=2492:00 +CPU= 16.935
  04:40:31 STEP 2493 H=2493:00 +CPU= 25.911
  04:40:50 STEP 2494 H=2494:00 +CPU= 17.662
  04:41:08 STEP 2495 H=2495:00 +CPU= 18.051
  04:41:42 STEP 2496 H=2496:00 +CPU= 32.276
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_103.zip
Uploading the intermediate file: upload_file_103.zip
  04:42:00 STEP 2497 H=2497:00 +CPU= 16.853
  04:42:18 STEP 2498 H=2498:00 +CPU= 17.890
  04:42:46 STEP 2499 H=2499:00 +CPU= 27.376
  04:43:05 STEP 2500 H=2500:00 +CPU= 17.821
  04:43:23 STEP 2501 H=2501:00 +CPU= 17.445
  04:43:52 STEP 2502 H=2502:00 +CPU= 27.852
  04:44:11 STEP 2503 H=2503:00 +CPU= 17.896
  04:44:30 STEP 2504 H=2504:00 +CPU= 18.630
  04:45:00 STEP 2505 H=2505:00 +CPU= 29.443
  04:45:20 STEP 2506 H=2506:00 +CPU= 19.166
  04:45:39 STEP 2507 H=2507:00 +CPU= 18.613
  04:46:15 STEP 2508 H=2508:00 +CPU= 34.497
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002508
  04:46:36 STEP 2509 H=2509:00 +CPU= 20.411
  04:47:00 STEP 2510 H=2510:00 +CPU= 21.640
  04:47:33 STEP 2511 H=2511:00 +CPU= 30.690
  04:47:55 STEP 2512 H=2512:00 +CPU= 20.600
  04:48:16 STEP 2513 H=2513:00 +CPU= 19.743
  04:48:48 STEP 2514 H=2514:00 +CPU= 31.204
  04:49:08 STEP 2515 H=2515:00 +CPU= 19.480
  04:49:28 STEP 2516 H=2516:00 +CPU= 19.950
  04:49:59 STEP 2517 H=2517:00 +CPU= 29.558
  04:50:20 STEP 2518 H=2518:00 +CPU= 20.600
  04:50:40 STEP 2519 H=2519:00 +CPU= 19.730
  04:51:19 STEP 2520 H=2520:00 +CPU= 37.089
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_104.zip
Uploading the intermediate file: upload_file_104.zip
  04:51:40 STEP 2521 H=2521:00 +CPU= 19.791
  04:51:59 STEP 2522 H=2522:00 +CPU= 19.102
  04:52:29 STEP 2523 H=2523:00 +CPU= 29.527
  04:52:50 STEP 2524 H=2524:00 +CPU= 20.650
  04:53:10 STEP 2525 H=2525:00 +CPU= 19.263
  04:53:41 STEP 2526 H=2526:00 +CPU= 30.225
  04:53:59 STEP 2527 H=2527:00 +CPU= 17.969
  04:54:18 STEP 2528 H=2528:00 +CPU= 18.475
  04:54:48 STEP 2529 H=2529:00 +CPU= 29.461
  04:55:07 STEP 2530 H=2530:00 +CPU= 18.828
  04:55:27 STEP 2531 H=2531:00 +CPU= 20.242
  04:56:02 STEP 2532 H=2532:00 +CPU= 34.304
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002532
  04:56:26 STEP 2533 H=2533:00 +CPU= 23.388
  04:56:53 STEP 2534 H=2534:00 +CPU= 26.749
  04:57:31 STEP 2535 H=2535:00 +CPU= 36.608
  04:58:01 STEP 2536 H=2536:00 +CPU= 28.470
  04:58:31 STEP 2537 H=2537:00 +CPU= 27.847
  04:59:19 STEP 2538 H=2538:00 +CPU= 43.692
  04:59:52 STEP 2539 H=2539:00 +CPU= 29.162
  05:00:25 STEP 2540 H=2540:00 +CPU= 29.409
  05:01:12 STEP 2541 H=2541:00 +CPU= 42.913
  05:01:41 STEP 2542 H=2542:00 +CPU= 28.711
  05:02:13 STEP 2543 H=2543:00 +CPU= 29.746
  05:03:11 STEP 2544 H=2544:00 +CPU= 52.157
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_105.zip
Uploading the intermediate file: upload_file_105.zip
  05:03:44 STEP 2545 H=2545:00 +CPU= 29.461
  05:04:12 STEP 2546 H=2546:00 +CPU= 27.240
  05:05:00 STEP 2547 H=2547:00 +CPU= 43.806
  05:05:30 STEP 2548 H=2548:00 +CPU= 26.696
  05:05:57 STEP 2549 H=2549:00 +CPU= 24.892
  05:06:46 STEP 2550 H=2550:00 +CPU= 48.082
  05:07:17 STEP 2551 H=2551:00 +CPU= 28.607
  05:07:47 STEP 2552 H=2552:00 +CPU= 28.544
  05:08:31 STEP 2553 H=2553:00 +CPU= 42.021
  05:09:02 STEP 2554 H=2554:00 +CPU= 28.733
  05:09:31 STEP 2555 H=2555:00 +CPU= 27.550
  05:10:20 STEP 2556 H=2556:00 +CPU= 45.999
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002556
  05:10:48 STEP 2557 H=2557:00 +CPU= 27.002
  05:11:19 STEP 2558 H=2558:00 +CPU= 28.470
  05:12:01 STEP 2559 H=2559:00 +CPU= 39.193
  05:12:28 STEP 2560 H=2560:00 +CPU= 25.630
  05:12:55 STEP 2561 H=2561:00 +CPU= 26.208
  05:13:38 STEP 2562 H=2562:00 +CPU= 40.857
  05:14:07 STEP 2563 H=2563:00 +CPU= 26.917
  05:14:38 STEP 2564 H=2564:00 +CPU= 29.125
  05:15:20 STEP 2565 H=2565:00 +CPU= 40.560
  05:15:51 STEP 2566 H=2566:00 +CPU= 30.167
  05:16:18 STEP 2567 H=2567:00 +CPU= 25.579
  05:17:10 STEP 2568 H=2568:00 +CPU= 47.281
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_106.zip
Uploading the intermediate file: upload_file_106.zip
  05:17:43 STEP 2569 H=2569:00 +CPU= 30.067
  05:18:14 STEP 2570 H=2570:00 +CPU= 28.490
  05:19:01 STEP 2571 H=2571:00 +CPU= 42.725
  05:19:32 STEP 2572 H=2572:00 +CPU= 28.794
  05:19:59 STEP 2573 H=2573:00 +CPU= 26.846
  05:20:47 STEP 2574 H=2574:00 +CPU= 44.662
  05:21:17 STEP 2575 H=2575:00 +CPU= 28.401
  05:21:44 STEP 2576 H=2576:00 +CPU= 26.688
  05:22:32 STEP 2577 H=2577:00 +CPU= 42.563
  05:23:06 STEP 2578 H=2578:00 +CPU= 31.226
  05:23:37 STEP 2579 H=2579:00 +CPU= 29.929
  05:24:31 STEP 2580 H=2580:00 +CPU= 50.358
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002580
  05:25:02 STEP 2581 H=2581:00 +CPU= 28.096
  05:25:32 STEP 2582 H=2582:00 +CPU= 28.699
  05:26:15 STEP 2583 H=2583:00 +CPU= 40.918
  05:26:43 STEP 2584 H=2584:00 +CPU= 26.646
  05:27:12 STEP 2585 H=2585:00 +CPU= 26.996
  05:27:53 STEP 2586 H=2586:00 +CPU= 38.467
  05:28:23 STEP 2587 H=2587:00 +CPU= 29.228
  05:28:53 STEP 2588 H=2588:00 +CPU= 27.125
  05:29:37 STEP 2589 H=2589:00 +CPU= 41.207
  05:30:05 STEP 2590 H=2590:00 +CPU= 26.437
  05:30:32 STEP 2591 H=2591:00 +CPU= 25.331
  05:31:20 STEP 2592 H=2592:00 +CPU= 45.152
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_107.zip
Uploading the intermediate file: upload_file_107.zip
  05:31:46 STEP 2593 H=2593:00 +CPU= 24.964
  05:32:12 STEP 2594 H=2594:00 +CPU= 24.395
  05:32:51 STEP 2595 H=2595:00 +CPU= 37.693
  05:33:18 STEP 2596 H=2596:00 +CPU= 25.645
  05:33:44 STEP 2597 H=2597:00 +CPU= 24.534
  05:34:20 STEP 2598 H=2598:00 +CPU= 34.697
  05:34:45 STEP 2599 H=2599:00 +CPU= 23.801
  05:35:15 STEP 2600 H=2600:00 +CPU= 27.970
  05:35:57 STEP 2601 H=2601:00 +CPU= 39.624
  05:36:24 STEP 2602 H=2602:00 +CPU= 25.052
  05:36:53 STEP 2603 H=2603:00 +CPU= 27.461
  05:37:38 STEP 2604 H=2604:00 +CPU= 43.254
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002604
  05:38:03 STEP 2605 H=2605:00 +CPU= 23.514
  05:38:29 STEP 2606 H=2606:00 +CPU= 24.921
  05:39:07 STEP 2607 H=2607:00 +CPU= 35.968
  05:39:31 STEP 2608 H=2608:00 +CPU= 22.769
  05:39:55 STEP 2609 H=2609:00 +CPU= 22.625
  05:40:33 STEP 2610 H=2610:00 +CPU= 36.408
  05:40:56 STEP 2611 H=2611:00 +CPU= 21.653
  05:41:20 STEP 2612 H=2612:00 +CPU= 23.194
  05:41:56 STEP 2613 H=2613:00 +CPU= 33.972
  05:42:19 STEP 2614 H=2614:00 +CPU= 22.153
  05:42:44 STEP 2615 H=2615:00 +CPU= 23.501
  05:43:26 STEP 2616 H=2616:00 +CPU= 40.459
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_108.zip
Uploading the intermediate file: upload_file_108.zip
Uploading trickle at timestep: 9417600
  05:43:48 STEP 2617 H=2617:00 +CPU= 21.112
  05:44:12 STEP 2618 H=2618:00 +CPU= 22.977
  05:44:47 STEP 2619 H=2619:00 +CPU= 33.272
  05:45:11 STEP 2620 H=2620:00 +CPU= 22.618
  05:45:35 STEP 2621 H=2621:00 +CPU= 22.702
  05:46:12 STEP 2622 H=2622:00 +CPU= 35.611
  05:46:35 STEP 2623 H=2623:00 +CPU= 21.739
  05:46:59 STEP 2624 H=2624:00 +CPU= 23.127
  05:47:36 STEP 2625 H=2625:00 +CPU= 35.388
  05:48:00 STEP 2626 H=2626:00 +CPU= 22.643
  05:48:24 STEP 2627 H=2627:00 +CPU= 22.952
  05:49:04 STEP 2628 H=2628:00 +CPU= 39.156
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002628
  05:49:29 STEP 2629 H=2629:00 +CPU= 23.216
  05:49:54 STEP 2630 H=2630:00 +CPU= 23.683
  05:50:31 STEP 2631 H=2631:00 +CPU= 34.447
  05:50:57 STEP 2632 H=2632:00 +CPU= 23.389
  05:51:20 STEP 2633 H=2633:00 +CPU= 22.253
  05:51:59 STEP 2634 H=2634:00 +CPU= 36.530
  05:52:22 STEP 2635 H=2635:00 +CPU= 21.395
  05:52:47 STEP 2636 H=2636:00 +CPU= 23.797
  05:53:29 STEP 2637 H=2637:00 +CPU= 41.249
  05:53:54 STEP 2638 H=2638:00 +CPU= 23.772
  05:54:21 STEP 2639 H=2639:00 +CPU= 25.073
  05:55:04 STEP 2640 H=2640:00 +CPU= 41.679
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_109.zip
Uploading the intermediate file: upload_file_109.zip
  05:55:27 STEP 2641 H=2641:00 +CPU= 22.277
  05:55:51 STEP 2642 H=2642:00 +CPU= 22.698
  05:56:29 STEP 2643 H=2643:00 +CPU= 35.220
  05:56:52 STEP 2644 H=2644:00 +CPU= 22.659
  05:57:15 STEP 2645 H=2645:00 +CPU= 22.233
  05:57:54 STEP 2646 H=2646:00 +CPU= 36.605
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0046
(argv4) batchid: 973
(argv5) wuid: 12189690
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_bca2950f0dbe39634e7a52630d019ce8 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0046_2004050100_123_973_12189690.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0046_2004050100_123_973_12189690.zip
ic_ancil_file: ic_ancil_12189690
ifsdata_file: ifsdata_12189690
climate_data_file: clim_data_12189690
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6f47685f1aaa5667d26260bcb220e59b to: /var/lib/boinc-client/slots/3/ic_ancil_12189690.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12189690.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5d00e797830296c1dd6ea4f9e49ea165 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12189690.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12189690.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12189690.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12189690.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12189690.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12189690.xml
-- Model is restarting --
Adjusting last_iter, 2647, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12189690.xml
last_cpu_time: 62592
upload_file_number: 110
last_iter: 2639
last_upload: 9504000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226
The child process has been launched with process id: 61924
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:61924:61924] [20230102:060505:1672599905:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:05:51 STEP 2640 H=2640:00 +CPU= 40.478
  06:06:18 STEP 2641 H=2641:00 +CPU= 25.590
  06:06:43 STEP 2642 H=2642:00 +CPU= 24.778
  06:07:21 STEP 2643 H=2643:00 +CPU= 37.769
  06:07:47 STEP 2644 H=2644:00 +CPU= 25.466
  06:08:15 STEP 2645 H=2645:00 +CPU= 25.828
  06:08:55 STEP 2646 H=2646:00 +CPU= 39.023
  06:09:20 STEP 2647 H=2647:00 +CPU= 24.848
  06:09:47 STEP 2648 H=2648:00 +CPU= 25.929
  06:10:25 STEP 2649 H=2649:00 +CPU= 37.093
  06:10:51 STEP 2650 H=2650:00 +CPU= 25.068
  06:11:19 STEP 2651 H=2651:00 +CPU= 24.907
  06:12:02 STEP 2652 H=2652:00 +CPU= 41.306
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002652
  06:12:28 STEP 2653 H=2653:00 +CPU= 24.971
  06:12:55 STEP 2654 H=2654:00 +CPU= 25.604
  06:13:35 STEP 2655 H=2655:00 +CPU= 37.536
  06:14:03 STEP 2656 H=2656:00 +CPU= 25.597
  06:14:31 STEP 2657 H=2657:00 +CPU= 25.848
  06:15:16 STEP 2658 H=2658:00 +CPU= 42.986
  06:15:47 STEP 2659 H=2659:00 +CPU= 29.161
  06:16:21 STEP 2660 H=2660:00 +CPU= 31.627
  06:17:12 STEP 2661 H=2661:00 +CPU= 48.884
  06:17:50 STEP 2662 H=2662:00 +CPU= 35.462
  06:18:29 STEP 2663 H=2663:00 +CPU= 35.753
  06:19:35 STEP 2664 H=2664:00 +CPU= 61.267
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189690/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0046_2004050100_123_973_12189690_0_r1917950226_110.zip
Uploading the intermediate file: upload_file_110.zip
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:05:21 1520317 22275241 oifs_43r3_ps_0046_2004050100_123_973_12189690_0 9,417,600 61,758 0.0066
09 May 2023 16:05:18 1520317 22275241 oifs_43r3_ps_0046_2004050100_123_973_12189690_0 8,553,600 55,781 0.0065
09 May 2023 16:05:18 1520317 22275241 oifs_43r3_ps_0046_2004050100_123_973_12189690_0 8,553,600 55,781 0.0065
09 May 2023 15:53:45 1520317 22275241 oifs_43r3_ps_0046_2004050100_123_973_12189690_0 6,825,600 46,044 0.0067
09 May 2023 15:53:15 1520317 22275241 oifs_43r3_ps_0046_2004050100_123_973_12189690_0 5,961,600 40,351 0.0068
09 May 2023 15:53:01 1520317 22275241 oifs_43r3_ps_0046_2004050100_123_973_12189690_0 5,097,600 34,769 0.0068
09 May 2023 15:52:55 1520317 22275241 oifs_43r3_ps_0046_2004050100_123_973_12189690_0 4,233,600 28,766 0.0068
09 May 2023 15:52:43 1520317 22275241 oifs_43r3_ps_0046_2004050100_123_973_12189690_0 3,369,600 22,320 0.0066
09 May 2023 15:52:14 1520317 22275241 oifs_43r3_ps_0046_2004050100_123_973_12189690_0 2,419,200 15,204 0.0063
09 May 2023 15:52:02 1520317 22275241 oifs_43r3_ps_0046_2004050100_123_973_12189690_0 1,555,200 9,836 0.0063


©2024 climateprediction.net