climateprediction.net home page
Task 22253008

Task 22253008

Name oifs_43r3_ps_0098_1983050100_123_952_12168742_0
Workunit 12168742
Created 20 Dec 2022, 23:52:34 UTC
Sent 5 Jan 2023, 13:31:10 UTC
Report deadline 4 Feb 2023, 13:31:10 UTC
Received 12 Jan 2023, 0:22:46 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537625
Run time 11 hours 41 min 54 sec
CPU time 11 hours 31 min 14 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,768.05 MB
Peak swap size 4,977.86 MB
Peak disk usage 3,998.33 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
 the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002268
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_94.zip
Uploading the intermediate file: upload_file_94.zip
  11:11:37 STEP 2281 H=2281:00 +CPU=  9.188
  11:11:46 STEP 2282 H=2282:00 +CPU=  9.342
  11:12:01 STEP 2283 H=2283:00 +CPU= 14.680
  11:12:10 STEP 2284 H=2284:00 +CPU=  9.396
  11:12:19 STEP 2285 H=2285:00 +CPU=  9.082
  11:12:35 STEP 2286 H=2286:00 +CPU= 15.070
  11:12:44 STEP 2287 H=2287:00 +CPU=  9.124
  11:12:53 STEP 2288 H=2288:00 +CPU=  9.357
  11:13:08 STEP 2289 H=2289:00 +CPU= 14.854
  11:13:17 STEP 2290 H=2290:00 +CPU=  9.321
  11:13:27 STEP 2291 H=2291:00 +CPU=  9.175
  11:13:43 STEP 2292 H=2292:00 +CPU= 16.466
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002292
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002292
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002292
  11:13:52 STEP 2293 H=2293:00 +CPU=  9.155
  11:14:02 STEP 2294 H=2294:00 +CPU=  9.472
  11:14:16 STEP 2295 H=2295:00 +CPU= 14.803
  11:14:26 STEP 2296 H=2296:00 +CPU=  9.351
  11:14:35 STEP 2297 H=2297:00 +CPU=  9.208
  11:14:50 STEP 2298 H=2298:00 +CPU= 15.074
  11:14:59 STEP 2299 H=2299:00 +CPU=  8.992
  11:15:08 STEP 2300 H=2300:00 +CPU=  9.373
  11:15:23 STEP 2301 H=2301:00 +CPU= 14.716
  11:15:32 STEP 2302 H=2302:00 +CPU=  9.304
  11:15:42 STEP 2303 H=2303:00 +CPU=  9.371
  11:15:59 STEP 2304 H=2304:00 +CPU= 17.285
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002304
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002304
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002292
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_95.zip
Uploading the intermediate file: upload_file_95.zip
  11:16:09 STEP 2305 H=2305:00 +CPU=  9.255
  11:16:18 STEP 2306 H=2306:00 +CPU=  9.451
  11:16:33 STEP 2307 H=2307:00 +CPU= 14.747
  11:16:42 STEP 2308 H=2308:00 +CPU=  9.415
  11:16:51 STEP 2309 H=2309:00 +CPU=  9.032
  11:17:06 STEP 2310 H=2310:00 +CPU= 15.053
  11:17:15 STEP 2311 H=2311:00 +CPU=  9.050
  11:17:25 STEP 2312 H=2312:00 +CPU=  9.450
  11:17:39 STEP 2313 H=2313:00 +CPU= 14.664
  11:17:49 STEP 2314 H=2314:00 +CPU=  9.325
  11:17:58 STEP 2315 H=2315:00 +CPU=  9.112
  11:18:14 STEP 2316 H=2316:00 +CPU= 16.354
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002316
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002316
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002316
  11:18:23 STEP 2317 H=2317:00 +CPU=  9.111
  11:18:35 STEP 2318 H=2318:00 +CPU= 11.367
  11:18:50 STEP 2319 H=2319:00 +CPU= 15.309
  11:18:59 STEP 2320 H=2320:00 +CPU=  9.385
  11:19:09 STEP 2321 H=2321:00 +CPU=  9.126
  11:19:24 STEP 2322 H=2322:00 +CPU= 14.989
  11:19:32 STEP 2323 H=2323:00 +CPU=  8.953
  11:19:42 STEP 2324 H=2324:00 +CPU=  9.319
  11:19:57 STEP 2325 H=2325:00 +CPU= 14.761
  11:20:06 STEP 2326 H=2326:00 +CPU=  9.344
  11:20:15 STEP 2327 H=2327:00 +CPU=  9.362
  11:20:34 STEP 2328 H=2328:00 +CPU= 17.243
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002328
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002328
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002316
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_96.zip
Uploading the intermediate file: upload_file_96.zip
  11:20:43 STEP 2329 H=2329:00 +CPU=  9.048
  11:20:52 STEP 2330 H=2330:00 +CPU=  9.326
  11:21:07 STEP 2331 H=2331:00 +CPU= 14.706
  11:21:16 STEP 2332 H=2332:00 +CPU=  9.349
  11:21:25 STEP 2333 H=2333:00 +CPU=  9.120
  11:21:40 STEP 2334 H=2334:00 +CPU= 15.041
  11:21:49 STEP 2335 H=2335:00 +CPU=  9.101
  11:21:59 STEP 2336 H=2336:00 +CPU=  9.357
  11:22:14 STEP 2337 H=2337:00 +CPU= 14.784
  11:22:23 STEP 2338 H=2338:00 +CPU=  9.282
  11:22:32 STEP 2339 H=2339:00 +CPU=  9.000
  11:22:48 STEP 2340 H=2340:00 +CPU= 16.394
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002340
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002340
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002340
  11:22:57 STEP 2341 H=2341:00 +CPU=  9.052
  11:23:07 STEP 2342 H=2342:00 +CPU=  9.403
  11:23:22 STEP 2343 H=2343:00 +CPU= 14.863
  11:23:31 STEP 2344 H=2344:00 +CPU=  9.346
  11:23:40 STEP 2345 H=2345:00 +CPU=  9.072
  11:23:55 STEP 2346 H=2346:00 +CPU= 14.895
  11:24:04 STEP 2347 H=2347:00 +CPU=  8.961
  11:24:13 STEP 2348 H=2348:00 +CPU=  9.222
  11:24:28 STEP 2349 H=2349:00 +CPU= 14.481
  11:24:37 STEP 2350 H=2350:00 +CPU=  9.290
  11:24:46 STEP 2351 H=2351:00 +CPU=  9.273
  11:25:04 STEP 2352 H=2352:00 +CPU= 17.070
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002352
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002352
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002340
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_97.zip
Uploading the intermediate file: upload_file_97.zip
  11:25:13 STEP 2353 H=2353:00 +CPU=  8.972
  11:25:23 STEP 2354 H=2354:00 +CPU=  9.421
  11:25:37 STEP 2355 H=2355:00 +CPU= 14.429
  11:25:47 STEP 2356 H=2356:00 +CPU=  9.280
  11:25:58 STEP 2357 H=2357:00 +CPU= 11.308
  11:26:13 STEP 2358 H=2358:00 +CPU= 15.281
  11:26:22 STEP 2359 H=2359:00 +CPU=  9.077
  11:26:31 STEP 2360 H=2360:00 +CPU=  9.243
  11:26:46 STEP 2361 H=2361:00 +CPU= 14.691
  11:26:55 STEP 2362 H=2362:00 +CPU=  9.232
  11:27:04 STEP 2363 H=2363:00 +CPU=  8.970
  11:27:21 STEP 2364 H=2364:00 +CPU= 16.317
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002364
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002364
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002364
  11:27:30 STEP 2365 H=2365:00 +CPU=  8.971
  11:27:39 STEP 2366 H=2366:00 +CPU=  9.254
  11:27:54 STEP 2367 H=2367:00 +CPU= 14.647
  11:28:03 STEP 2368 H=2368:00 +CPU=  9.240
  11:28:32 STEP 2369 H=2369:00 +CPU=  8.914
  11:28:47 STEP 2370 H=2370:00 +CPU= 14.924
  11:28:56 STEP 2371 H=2371:00 +CPU=  9.003
  11:29:05 STEP 2372 H=2372:00 +CPU=  9.234
  11:29:20 STEP 2373 H=2373:00 +CPU= 14.545
  11:29:29 STEP 2374 H=2374:00 +CPU=  9.281
  11:29:39 STEP 2375 H=2375:00 +CPU=  9.211
  11:29:58 STEP 2376 H=2376:00 +CPU= 16.909
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002376
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002376
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002364
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_98.zip
Uploading the intermediate file: upload_file_98.zip
  11:30:06 STEP 2377 H=2377:00 +CPU=  8.924
  11:30:16 STEP 2378 H=2378:00 +CPU=  9.264
  11:30:30 STEP 2379 H=2379:00 +CPU= 14.549
  11:30:40 STEP 2380 H=2380:00 +CPU=  9.253
  11:30:48 STEP 2381 H=2381:00 +CPU=  8.879
  11:31:03 STEP 2382 H=2382:00 +CPU= 14.822
  11:31:12 STEP 2383 H=2383:00 +CPU=  8.990
  11:31:21 STEP 2384 H=2384:00 +CPU=  9.111
  11:31:36 STEP 2385 H=2385:00 +CPU= 14.627
  11:31:45 STEP 2386 H=2386:00 +CPU=  9.163
  11:31:54 STEP 2387 H=2387:00 +CPU=  8.905
  11:32:10 STEP 2388 H=2388:00 +CPU= 16.346
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002388
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002388
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002388
  11:32:19 STEP 2389 H=2389:00 +CPU=  9.018
  11:32:29 STEP 2390 H=2390:00 +CPU=  9.281
  11:32:43 STEP 2391 H=2391:00 +CPU= 14.790
  11:32:53 STEP 2392 H=2392:00 +CPU=  9.407
  11:33:02 STEP 2393 H=2393:00 +CPU=  8.999
  11:33:17 STEP 2394 H=2394:00 +CPU= 15.157
  11:33:26 STEP 2395 H=2395:00 +CPU=  9.156
  11:33:35 STEP 2396 H=2396:00 +CPU=  9.138
  11:33:50 STEP 2397 H=2397:00 +CPU= 14.620
  11:33:59 STEP 2398 H=2398:00 +CPU=  9.374
  11:34:09 STEP 2399 H=2399:00 +CPU=  9.147
  11:34:30 STEP 2400 H=2400:00 +CPU= 19.968
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002400
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002400
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002388
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_99.zip
Uploading the intermediate file: upload_file_99.zip
  11:34:39 STEP 2401 H=2401:00 +CPU=  9.065
  11:34:48 STEP 2402 H=2402:00 +CPU=  9.336
  11:35:03 STEP 2403 H=2403:00 +CPU= 14.527
  11:35:12 STEP 2404 H=2404:00 +CPU=  9.517
  11:35:21 STEP 2405 H=2405:00 +CPU=  8.990
  11:35:36 STEP 2406 H=2406:00 +CPU= 15.016
  11:35:45 STEP 2407 H=2407:00 +CPU=  9.018
  11:35:55 STEP 2408 H=2408:00 +CPU=  9.147
  11:36:09 STEP 2409 H=2409:00 +CPU= 14.909
  11:36:19 STEP 2410 H=2410:00 +CPU=  9.376
  11:36:28 STEP 2411 H=2411:00 +CPU=  8.888
  11:36:44 STEP 2412 H=2412:00 +CPU= 16.425
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002412
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002412
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002412
  11:36:53 STEP 2413 H=2413:00 +CPU=  9.144
  11:37:03 STEP 2414 H=2414:00 +CPU=  9.409
  11:37:17 STEP 2415 H=2415:00 +CPU= 14.713
  11:37:27 STEP 2416 H=2416:00 +CPU=  9.396
  11:37:36 STEP 2417 H=2417:00 +CPU=  9.169
  11:37:51 STEP 2418 H=2418:00 +CPU= 15.013
  11:38:00 STEP 2419 H=2419:00 +CPU=  8.962
  11:38:09 STEP 2420 H=2420:00 +CPU=  9.211
  11:38:24 STEP 2421 H=2421:00 +CPU= 14.746
  11:38:33 STEP 2422 H=2422:00 +CPU=  9.500
  11:38:43 STEP 2423 H=2423:00 +CPU=  9.275
  11:39:03 STEP 2424 H=2424:00 +CPU= 17.298
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002424
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002424
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002412
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_100.zip
  11:39:12 STEP 2425 H=2425:00 +CPU=  9.083
Uploading the intermediate file: upload_file_100.zip
  11:39:21 STEP 2426 H=2426:00 +CPU=  9.403
Uploading trickle at timestep: 8726400
  11:39:36 STEP 2427 H=2427:00 +CPU= 14.912
  11:39:45 STEP 2428 H=2428:00 +CPU=  9.430
  11:39:56 STEP 2429 H=2429:00 +CPU= 10.489
  11:40:11 STEP 2430 H=2430:00 +CPU= 15.222
  11:40:20 STEP 2431 H=2431:00 +CPU=  9.174
  11:40:30 STEP 2432 H=2432:00 +CPU=  9.336
  11:40:44 STEP 2433 H=2433:00 +CPU= 14.666
  11:40:54 STEP 2434 H=2434:00 +CPU=  9.321
  11:41:03 STEP 2435 H=2435:00 +CPU=  9.016
  11:41:19 STEP 2436 H=2436:00 +CPU= 16.549
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002436
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002436
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002436
  11:41:28 STEP 2437 H=2437:00 +CPU=  9.076
  11:41:38 STEP 2438 H=2438:00 +CPU=  9.393
  11:41:52 STEP 2439 H=2439:00 +CPU= 14.615
  11:42:02 STEP 2440 H=2440:00 +CPU=  9.381
  11:42:11 STEP 2441 H=2441:00 +CPU=  9.152
  11:42:26 STEP 2442 H=2442:00 +CPU= 14.939
  11:42:35 STEP 2443 H=2443:00 +CPU=  9.197
  11:42:45 STEP 2444 H=2444:00 +CPU=  9.381
  11:42:59 STEP 2445 H=2445:00 +CPU= 14.514
  11:43:09 STEP 2446 H=2446:00 +CPU=  9.356
  11:43:18 STEP 2447 H=2447:00 +CPU=  9.354
  11:43:39 STEP 2448 H=2448:00 +CPU= 17.134
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002448
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002448
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002436
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_101.zip
Uploading the intermediate file: upload_file_101.zip
  11:43:48 STEP 2449 H=2449:00 +CPU=  9.183
  11:43:58 STEP 2450 H=2450:00 +CPU=  9.485
  11:44:13 STEP 2451 H=2451:00 +CPU= 14.684
  11:44:22 STEP 2452 H=2452:00 +CPU=  9.246
  11:44:31 STEP 2453 H=2453:00 +CPU=  9.016
  11:44:47 STEP 2454 H=2454:00 +CPU= 16.550
  11:44:56 STEP 2455 H=2455:00 +CPU=  8.967
  11:45:06 STEP 2456 H=2456:00 +CPU=  9.333
  11:45:20 STEP 2457 H=2457:00 +CPU= 14.729
  11:45:30 STEP 2458 H=2458:00 +CPU=  9.327
  11:45:39 STEP 2459 H=2459:00 +CPU=  9.037
  11:45:55 STEP 2460 H=2460:00 +CPU= 16.325
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002460
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002460
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002460
  11:46:04 STEP 2461 H=2461:00 +CPU=  9.131
  11:46:14 STEP 2462 H=2462:00 +CPU=  9.424
  11:46:28 STEP 2463 H=2463:00 +CPU= 14.605
  11:46:38 STEP 2464 H=2464:00 +CPU=  9.313
  11:46:47 STEP 2465 H=2465:00 +CPU=  9.143
  11:47:02 STEP 2466 H=2466:00 +CPU= 14.934
  11:47:11 STEP 2467 H=2467:00 +CPU=  8.915
  11:47:20 STEP 2468 H=2468:00 +CPU=  9.171
  11:47:34 STEP 2469 H=2469:00 +CPU= 14.515
  11:47:44 STEP 2470 H=2470:00 +CPU=  9.582
  11:47:53 STEP 2471 H=2471:00 +CPU=  9.282
  11:48:13 STEP 2472 H=2472:00 +CPU= 17.150
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002472
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002472
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002460
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_102.zip
Uploading the intermediate file: upload_file_102.zip
  11:48:22 STEP 2473 H=2473:00 +CPU=  9.015
  11:48:31 STEP 2474 H=2474:00 +CPU=  9.166
  11:48:45 STEP 2475 H=2475:00 +CPU= 14.638
  11:48:55 STEP 2476 H=2476:00 +CPU=  9.255
  11:49:04 STEP 2477 H=2477:00 +CPU=  9.111
  11:49:19 STEP 2478 H=2478:00 +CPU= 15.084
  11:49:28 STEP 2479 H=2479:00 +CPU=  9.108
  11:49:37 STEP 2480 H=2480:00 +CPU=  9.260
  11:49:54 STEP 2481 H=2481:00 +CPU= 16.768
  11:50:03 STEP 2482 H=2482:00 +CPU=  9.357
  11:50:12 STEP 2483 H=2483:00 +CPU=  9.176
  11:50:29 STEP 2484 H=2484:00 +CPU= 16.576
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002484
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002484
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002484
  11:50:38 STEP 2485 H=2485:00 +CPU=  9.162
  11:50:48 STEP 2486 H=2486:00 +CPU=  9.292
  11:51:02 STEP 2487 H=2487:00 +CPU= 14.612
  11:51:11 STEP 2488 H=2488:00 +CPU=  9.202
  11:51:20 STEP 2489 H=2489:00 +CPU=  9.101
  11:51:35 STEP 2490 H=2490:00 +CPU= 14.767
  11:51:44 STEP 2491 H=2491:00 +CPU=  9.018
  11:51:54 STEP 2492 H=2492:00 +CPU=  9.333
  11:52:08 STEP 2493 H=2493:00 +CPU= 14.618
  11:52:18 STEP 2494 H=2494:00 +CPU=  9.293
  11:52:27 STEP 2495 H=2495:00 +CPU=  9.368
  11:52:47 STEP 2496 H=2496:00 +CPU= 17.196
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002496
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002496
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002484
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_103.zip
Uploading the intermediate file: upload_file_103.zip
  11:52:56 STEP 2497 H=2497:00 +CPU=  9.015
  11:53:05 STEP 2498 H=2498:00 +CPU=  9.172
  11:53:20 STEP 2499 H=2499:00 +CPU= 14.568
  11:53:29 STEP 2500 H=2500:00 +CPU=  9.283
  11:53:38 STEP 2501 H=2501:00 +CPU=  8.979
  11:53:53 STEP 2502 H=2502:00 +CPU= 15.006
  11:54:02 STEP 2503 H=2503:00 +CPU=  9.109
  11:54:12 STEP 2504 H=2504:00 +CPU=  9.212
  11:54:26 STEP 2505 H=2505:00 +CPU= 14.644
  11:54:35 STEP 2506 H=2506:00 +CPU=  9.305
  11:54:44 STEP 2507 H=2507:00 +CPU=  8.913
  11:55:01 STEP 2508 H=2508:00 +CPU= 16.208
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002508
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002508
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002508
  11:55:10 STEP 2509 H=2509:00 +CPU=  9.679
  11:55:21 STEP 2510 H=2510:00 +CPU= 10.506
  11:55:36 STEP 2511 H=2511:00 +CPU= 14.804
  11:55:45 STEP 2512 H=2512:00 +CPU=  9.317
  11:55:54 STEP 2513 H=2513:00 +CPU=  9.342
  11:56:09 STEP 2514 H=2514:00 +CPU= 15.110
  11:56:18 STEP 2515 H=2515:00 +CPU=  9.061
  11:56:28 STEP 2516 H=2516:00 +CPU=  9.365
  11:56:43 STEP 2517 H=2517:00 +CPU= 14.782
  11:56:52 STEP 2518 H=2518:00 +CPU=  9.526
  11:57:02 STEP 2519 H=2519:00 +CPU=  9.396
  11:57:21 STEP 2520 H=2520:00 +CPU= 17.068
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002520
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002520
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002508
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_104.zip
Uploading the intermediate file: upload_file_104.zip
  11:57:30 STEP 2521 H=2521:00 +CPU=  8.967
  11:57:39 STEP 2522 H=2522:00 +CPU=  9.157
  11:57:53 STEP 2523 H=2523:00 +CPU= 14.647
  11:58:03 STEP 2524 H=2524:00 +CPU=  9.448
  11:58:12 STEP 2525 H=2525:00 +CPU=  8.984
  11:58:27 STEP 2526 H=2526:00 +CPU= 15.069
  11:58:36 STEP 2527 H=2527:00 +CPU=  9.053
  11:58:45 STEP 2528 H=2528:00 +CPU=  9.288
  11:59:00 STEP 2529 H=2529:00 +CPU= 14.734
  11:59:10 STEP 2530 H=2530:00 +CPU=  9.714
  11:59:21 STEP 2531 H=2531:00 +CPU= 11.093
  11:59:37 STEP 2532 H=2532:00 +CPU= 16.633
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002532
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002532
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002532
  11:59:46 STEP 2533 H=2533:00 +CPU=  9.069
  11:59:56 STEP 2534 H=2534:00 +CPU=  9.259
  12:00:10 STEP 2535 H=2535:00 +CPU= 14.538
  12:00:20 STEP 2536 H=2536:00 +CPU=  9.274
  12:00:29 STEP 2537 H=2537:00 +CPU=  9.184
  12:00:44 STEP 2538 H=2538:00 +CPU= 14.933
  12:00:53 STEP 2539 H=2539:00 +CPU=  9.164
  12:01:02 STEP 2540 H=2540:00 +CPU=  9.320
  12:01:17 STEP 2541 H=2541:00 +CPU= 14.740
  12:01:26 STEP 2542 H=2542:00 +CPU=  9.546
  12:01:36 STEP 2543 H=2543:00 +CPU=  9.269
  12:01:54 STEP 2544 H=2544:00 +CPU= 17.021
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002544
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002544
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002532
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_105.zip
Uploading the intermediate file: upload_file_105.zip
  12:02:03 STEP 2545 H=2545:00 +CPU=  8.906
  12:02:13 STEP 2546 H=2546:00 +CPU=  9.220
  12:02:27 STEP 2547 H=2547:00 +CPU= 14.550
  12:02:37 STEP 2548 H=2548:00 +CPU=  9.396
  12:02:46 STEP 2549 H=2549:00 +CPU=  9.002
  12:03:00 STEP 2550 H=2550:00 +CPU= 14.880
  12:03:10 STEP 2551 H=2551:00 +CPU=  9.167
  12:03:19 STEP 2552 H=2552:00 +CPU=  9.541
  12:03:34 STEP 2553 H=2553:00 +CPU= 14.868
  12:03:45 STEP 2554 H=2554:00 +CPU= 10.734
  12:03:54 STEP 2555 H=2555:00 +CPU=  9.336
  12:04:10 STEP 2556 H=2556:00 +CPU= 16.194
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002556
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002556
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002556
  12:04:19 STEP 2557 H=2557:00 +CPU=  9.166
  12:04:29 STEP 2558 H=2558:00 +CPU=  9.256
  12:04:43 STEP 2559 H=2559:00 +CPU= 14.505
  12:04:53 STEP 2560 H=2560:00 +CPU=  9.411
  12:05:02 STEP 2561 H=2561:00 +CPU=  9.144
  12:05:17 STEP 2562 H=2562:00 +CPU= 15.015
  12:05:26 STEP 2563 H=2563:00 +CPU=  9.021
  12:05:35 STEP 2564 H=2564:00 +CPU=  9.256
  12:05:50 STEP 2565 H=2565:00 +CPU= 14.780
  12:06:00 STEP 2566 H=2566:00 +CPU=  9.678
  12:06:09 STEP 2567 H=2567:00 +CPU=  9.297
  12:06:28 STEP 2568 H=2568:00 +CPU= 17.308
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002568
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002568
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002556
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_106.zip
Uploading the intermediate file: upload_file_106.zip
  12:06:37 STEP 2569 H=2569:00 +CPU=  9.298
  12:06:46 STEP 2570 H=2570:00 +CPU=  9.318
  12:07:01 STEP 2571 H=2571:00 +CPU= 14.739
  12:07:10 STEP 2572 H=2572:00 +CPU=  9.418
  12:07:19 STEP 2573 H=2573:00 +CPU=  9.088
  12:07:35 STEP 2574 H=2574:00 +CPU= 15.182
  12:07:45 STEP 2575 H=2575:00 +CPU=  9.941
  12:07:55 STEP 2576 H=2576:00 +CPU= 10.162
  12:08:09 STEP 2577 H=2577:00 +CPU= 14.592
  12:08:19 STEP 2578 H=2578:00 +CPU=  9.335
  12:08:28 STEP 2579 H=2579:00 +CPU=  9.217
  12:08:44 STEP 2580 H=2580:00 +CPU= 16.534
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002580
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002580
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002580
  12:08:54 STEP 2581 H=2581:00 +CPU=  9.135
  12:09:03 STEP 2582 H=2582:00 +CPU=  9.438
  12:09:17 STEP 2583 H=2583:00 +CPU= 14.506
  12:09:27 STEP 2584 H=2584:00 +CPU=  9.242
  12:09:36 STEP 2585 H=2585:00 +CPU=  9.092
  12:09:51 STEP 2586 H=2586:00 +CPU= 14.797
  12:10:00 STEP 2587 H=2587:00 +CPU=  9.016
  12:10:09 STEP 2588 H=2588:00 +CPU=  9.149
  12:10:23 STEP 2589 H=2589:00 +CPU= 14.482
  12:10:33 STEP 2590 H=2590:00 +CPU=  9.291
  12:10:42 STEP 2591 H=2591:00 +CPU=  9.070
  12:11:00 STEP 2592 H=2592:00 +CPU= 17.067
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002592
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002592
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002580
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_107.zip
Uploading the intermediate file: upload_file_107.zip
  12:11:09 STEP 2593 H=2593:00 +CPU=  8.940
  12:11:18 STEP 2594 H=2594:00 +CPU=  9.079
  12:11:33 STEP 2595 H=2595:00 +CPU= 14.632
  12:11:42 STEP 2596 H=2596:00 +CPU=  9.156
  12:11:51 STEP 2597 H=2597:00 +CPU=  8.922
  12:12:06 STEP 2598 H=2598:00 +CPU= 14.920
  12:12:15 STEP 2599 H=2599:00 +CPU=  9.024
  12:12:24 STEP 2600 H=2600:00 +CPU=  9.142
  12:12:39 STEP 2601 H=2601:00 +CPU= 14.500
  12:12:50 STEP 2602 H=2602:00 +CPU= 11.062
  12:12:59 STEP 2603 H=2603:00 +CPU=  9.327
  12:13:15 STEP 2604 H=2604:00 +CPU= 16.243
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002604
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002604
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002604
  12:13:24 STEP 2605 H=2605:00 +CPU=  9.013
  12:13:33 STEP 2606 H=2606:00 +CPU=  9.249
  12:13:48 STEP 2607 H=2607:00 +CPU= 14.437
  12:13:57 STEP 2608 H=2608:00 +CPU=  9.322
  12:14:06 STEP 2609 H=2609:00 +CPU=  9.109
  12:14:21 STEP 2610 H=2610:00 +CPU= 14.918
  12:14:30 STEP 2611 H=2611:00 +CPU=  9.034
  12:14:39 STEP 2612 H=2612:00 +CPU=  9.135
  12:14:54 STEP 2613 H=2613:00 +CPU= 14.494
  12:15:03 STEP 2614 H=2614:00 +CPU=  9.267
  12:15:12 STEP 2615 H=2615:00 +CPU=  9.191
  12:15:31 STEP 2616 H=2616:00 +CPU= 17.011
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002616
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002616
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002604
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_108.zip
Uploading the intermediate file: upload_file_108.zip
  12:15:40 STEP 2617 H=2617:00 +CPU=  8.968
  12:15:49 STEP 2618 H=2618:00 +CPU=  9.122
  12:16:03 STEP 2619 H=2619:00 +CPU= 14.510
  12:16:12 STEP 2620 H=2620:00 +CPU=  9.182
  12:16:21 STEP 2621 H=2621:00 +CPU=  8.920
  12:16:36 STEP 2622 H=2622:00 +CPU= 14.881
  12:16:45 STEP 2623 H=2623:00 +CPU=  9.066
  12:16:55 STEP 2624 H=2624:00 +CPU=  9.221
  12:17:09 STEP 2625 H=2625:00 +CPU= 14.463
  12:17:18 STEP 2626 H=2626:00 +CPU=  9.206
  12:17:27 STEP 2627 H=2627:00 +CPU=  8.998
  12:17:43 STEP 2628 H=2628:00 +CPU= 16.149
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002628
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002628
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002628
  12:17:54 STEP 2629 H=2629:00 +CPU= 10.312
  12:18:03 STEP 2630 H=2630:00 +CPU=  9.594
  12:18:18 STEP 2631 H=2631:00 +CPU= 14.532
  12:18:27 STEP 2632 H=2632:00 +CPU=  9.268
  12:18:36 STEP 2633 H=2633:00 +CPU=  8.945
  12:18:51 STEP 2634 H=2634:00 +CPU= 14.690
  12:19:00 STEP 2635 H=2635:00 +CPU=  9.001
  12:19:09 STEP 2636 H=2636:00 +CPU=  9.307
  12:19:24 STEP 2637 H=2637:00 +CPU= 14.433
  12:19:33 STEP 2638 H=2638:00 +CPU=  9.431
  12:19:42 STEP 2639 H=2639:00 +CPU=  9.265
  12:20:01 STEP 2640 H=2640:00 +CPU= 16.859
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002640
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002640
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002628
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_109.zip
Uploading the intermediate file: upload_file_109.zip
  12:20:10 STEP 2641 H=2641:00 +CPU=  8.994
  12:20:20 STEP 2642 H=2642:00 +CPU=  9.256
  12:20:34 STEP 2643 H=2643:00 +CPU= 14.477
  12:20:43 STEP 2644 H=2644:00 +CPU=  9.184
  12:20:52 STEP 2645 H=2645:00 +CPU=  8.977
  12:21:08 STEP 2646 H=2646:00 +CPU= 14.719
  12:21:17 STEP 2647 H=2647:00 +CPU=  8.993
  12:21:26 STEP 2648 H=2648:00 +CPU=  9.287
  12:21:40 STEP 2649 H=2649:00 +CPU= 14.610
  12:21:50 STEP 2650 H=2650:00 +CPU=  9.275
  12:21:59 STEP 2651 H=2651:00 +CPU=  9.082
  12:22:15 STEP 2652 H=2652:00 +CPU= 16.002
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002652
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002652
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002652
  12:22:24 STEP 2653 H=2653:00 +CPU=  9.025
  12:22:33 STEP 2654 H=2654:00 +CPU=  9.210
  12:22:48 STEP 2655 H=2655:00 +CPU= 14.462
  12:22:57 STEP 2656 H=2656:00 +CPU=  9.242
  12:23:06 STEP 2657 H=2657:00 +CPU=  9.024
  12:23:20 STEP 2658 H=2658:00 +CPU= 14.559
  12:23:30 STEP 2659 H=2659:00 +CPU=  9.123
  12:23:41 STEP 2660 H=2660:00 +CPU= 11.291
  12:23:55 STEP 2661 H=2661:00 +CPU= 14.607
  12:24:05 STEP 2662 H=2662:00 +CPU=  9.366
  12:24:14 STEP 2663 H=2663:00 +CPU=  9.251
  12:24:33 STEP 2664 H=2664:00 +CPU= 16.957
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002664
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002664
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002652
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_110.zip
Uploading the intermediate file: upload_file_110.zip
  12:24:42 STEP 2665 H=2665:00 +CPU=  9.021
  12:24:51 STEP 2666 H=2666:00 +CPU=  9.163
Uploading trickle at timestep: 9590400
  12:25:05 STEP 2667 H=2667:00 +CPU= 14.468
  12:25:15 STEP 2668 H=2668:00 +CPU=  9.234
  12:25:24 STEP 2669 H=2669:00 +CPU=  9.058
  12:25:38 STEP 2670 H=2670:00 +CPU= 14.697
  12:25:47 STEP 2671 H=2671:00 +CPU=  9.019
  12:25:57 STEP 2672 H=2672:00 +CPU=  9.197
  12:26:11 STEP 2673 H=2673:00 +CPU= 14.519
  12:26:20 STEP 2674 H=2674:00 +CPU=  9.226
  12:26:29 STEP 2675 H=2675:00 +CPU=  8.992
  12:26:45 STEP 2676 H=2676:00 +CPU= 16.052
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002676
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002676
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002676
  12:26:54 STEP 2677 H=2677:00 +CPU=  8.990
  12:27:04 STEP 2678 H=2678:00 +CPU=  9.185
  12:27:18 STEP 2679 H=2679:00 +CPU= 14.431
  12:27:27 STEP 2680 H=2680:00 +CPU=  9.289
  12:27:37 STEP 2681 H=2681:00 +CPU=  9.189
  12:27:51 STEP 2682 H=2682:00 +CPU= 14.854
  12:28:00 STEP 2683 H=2683:00 +CPU=  8.994
  12:28:10 STEP 2684 H=2684:00 +CPU=  9.302
  12:28:24 STEP 2685 H=2685:00 +CPU= 14.568
  12:28:34 STEP 2686 H=2686:00 +CPU=  9.289
  12:28:43 STEP 2687 H=2687:00 +CPU=  9.359
  12:29:03 STEP 2688 H=2688:00 +CPU= 17.179
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002688
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002688
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002676
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_111.zip
Uploading the intermediate file: upload_file_111.zip
  12:29:12 STEP 2689 H=2689:00 +CPU=  9.070
  12:29:21 STEP 2690 H=2690:00 +CPU=  9.301
  12:29:38 STEP 2691 H=2691:00 +CPU= 16.492
  12:29:47 STEP 2692 H=2692:00 +CPU=  9.274
  12:29:56 STEP 2693 H=2693:00 +CPU=  9.127
  12:30:11 STEP 2694 H=2694:00 +CPU= 14.795
  12:30:20 STEP 2695 H=2695:00 +CPU=  9.100
  12:30:29 STEP 2696 H=2696:00 +CPU=  9.368
  12:30:44 STEP 2697 H=2697:00 +CPU= 14.830
  12:30:54 STEP 2698 H=2698:00 +CPU=  9.328
  12:31:03 STEP 2699 H=2699:00 +CPU=  9.269
  12:31:19 STEP 2700 H=2700:00 +CPU= 16.666
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002700
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002700
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002700
  12:31:29 STEP 2701 H=2701:00 +CPU=  9.065
  12:31:38 STEP 2702 H=2702:00 +CPU=  9.458
  12:31:53 STEP 2703 H=2703:00 +CPU= 14.654
  12:32:02 STEP 2704 H=2704:00 +CPU=  9.403
  12:32:11 STEP 2705 H=2705:00 +CPU=  9.208
  12:32:27 STEP 2706 H=2706:00 +CPU= 15.772
  12:32:36 STEP 2707 H=2707:00 +CPU=  9.299
  12:32:46 STEP 2708 H=2708:00 +CPU=  9.425
  12:33:00 STEP 2709 H=2709:00 +CPU= 14.637
  12:33:10 STEP 2710 H=2710:00 +CPU=  9.399
  12:33:19 STEP 2711 H=2711:00 +CPU=  9.295
  12:33:37 STEP 2712 H=2712:00 +CPU= 17.277
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002712
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002712
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002700
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_112.zip
Uploading the intermediate file: upload_file_112.zip
  12:33:46 STEP 2713 H=2713:00 +CPU=  9.108
  12:33:55 STEP 2714 H=2714:00 +CPU=  9.282
  12:34:10 STEP 2715 H=2715:00 +CPU= 14.680
  12:34:19 STEP 2716 H=2716:00 +CPU=  9.437
  12:34:28 STEP 2717 H=2717:00 +CPU=  9.059
  12:34:43 STEP 2718 H=2718:00 +CPU= 14.911
  12:34:52 STEP 2719 H=2719:00 +CPU=  8.941
  12:35:01 STEP 2720 H=2720:00 +CPU=  9.261
  12:35:16 STEP 2721 H=2721:00 +CPU= 14.606
  12:35:25 STEP 2722 H=2722:00 +CPU=  9.233
  12:35:34 STEP 2723 H=2723:00 +CPU=  8.976
  12:35:51 STEP 2724 H=2724:00 +CPU= 16.311
  12:36:00 STEP 2725 H=2725:00 +CPU=  8.996
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002724
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002724
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002724
  12:36:10 STEP 2726 H=2726:00 +CPU= 10.567
  12:36:25 STEP 2727 H=2727:00 +CPU= 14.810
  12:36:34 STEP 2728 H=2728:00 +CPU=  9.215
  12:36:43 STEP 2729 H=2729:00 +CPU=  9.001
  12:36:58 STEP 2730 H=2730:00 +CPU= 14.913
  12:37:07 STEP 2731 H=2731:00 +CPU=  8.995
  12:37:17 STEP 2732 H=2732:00 +CPU=  9.356
  12:37:31 STEP 2733 H=2733:00 +CPU= 14.428
  12:37:40 STEP 2734 H=2734:00 +CPU=  9.218
  12:37:49 STEP 2735 H=2735:00 +CPU=  9.222
  12:38:08 STEP 2736 H=2736:00 +CPU= 17.073
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002736
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002736
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002724
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_113.zip
Uploading the intermediate file: upload_file_113.zip
  12:38:17 STEP 2737 H=2737:00 +CPU=  9.051
  12:38:26 STEP 2738 H=2738:00 +CPU=  9.203
  12:38:40 STEP 2739 H=2739:00 +CPU= 14.525
  12:38:50 STEP 2740 H=2740:00 +CPU=  9.231
  12:38:59 STEP 2741 H=2741:00 +CPU=  9.037
  12:39:13 STEP 2742 H=2742:00 +CPU= 14.790
  12:39:22 STEP 2743 H=2743:00 +CPU=  8.979
  12:39:33 STEP 2744 H=2744:00 +CPU= 10.588
  12:39:48 STEP 2745 H=2745:00 +CPU= 14.923
  12:39:57 STEP 2746 H=2746:00 +CPU=  9.184
  12:40:06 STEP 2747 H=2747:00 +CPU=  8.943
  12:40:22 STEP 2748 H=2748:00 +CPU= 16.215
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002748
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002748
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002748
  12:40:31 STEP 2749 H=2749:00 +CPU=  8.992
  12:40:41 STEP 2750 H=2750:00 +CPU=  9.206
  12:40:55 STEP 2751 H=2751:00 +CPU= 14.686
  12:41:04 STEP 2752 H=2752:00 +CPU=  9.191
  12:41:13 STEP 2753 H=2753:00 +CPU=  9.009
  12:41:29 STEP 2754 H=2754:00 +CPU= 15.058
  12:41:38 STEP 2755 H=2755:00 +CPU=  9.013
  12:41:47 STEP 2756 H=2756:00 +CPU=  9.324
  12:42:01 STEP 2757 H=2757:00 +CPU= 14.426
  12:42:11 STEP 2758 H=2758:00 +CPU=  9.199
  12:42:20 STEP 2759 H=2759:00 +CPU=  9.299
  12:42:38 STEP 2760 H=2760:00 +CPU= 17.231
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002760
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002760
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002748
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_114.zip
Uploading the intermediate file: upload_file_114.zip
  12:42:47 STEP 2761 H=2761:00 +CPU=  9.078
  12:42:57 STEP 2762 H=2762:00 +CPU=  9.251
  12:43:11 STEP 2763 H=2763:00 +CPU= 14.747
  12:43:21 STEP 2764 H=2764:00 +CPU=  9.467
  12:43:32 STEP 2765 H=2765:00 +CPU= 10.817
  12:43:47 STEP 2766 H=2766:00 +CPU= 15.173
  12:43:56 STEP 2767 H=2767:00 +CPU=  8.971
  12:44:05 STEP 2768 H=2768:00 +CPU=  9.170
  12:44:19 STEP 2769 H=2769:00 +CPU= 14.563
  12:44:29 STEP 2770 H=2770:00 +CPU=  9.248
  12:44:38 STEP 2771 H=2771:00 +CPU=  9.020
  12:44:54 STEP 2772 H=2772:00 +CPU= 16.267
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002772
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002772
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002772
  12:45:03 STEP 2773 H=2773:00 +CPU=  9.103
  12:45:12 STEP 2774 H=2774:00 +CPU=  9.243
  12:45:27 STEP 2775 H=2775:00 +CPU= 14.727
  12:45:37 STEP 2776 H=2776:00 +CPU=  9.439
  12:45:46 STEP 2777 H=2777:00 +CPU=  9.227
  12:46:01 STEP 2778 H=2778:00 +CPU= 15.234
  12:46:10 STEP 2779 H=2779:00 +CPU=  8.977
  12:46:20 STEP 2780 H=2780:00 +CPU=  9.522
  12:46:35 STEP 2781 H=2781:00 +CPU= 15.830
  12:46:45 STEP 2782 H=2782:00 +CPU=  9.324
  12:46:54 STEP 2783 H=2783:00 +CPU=  9.476
  12:47:13 STEP 2784 H=2784:00 +CPU= 17.347
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002784
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002784
  12:47:22 STEP 2785 H=2785:00 +CPU=  9.204
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002772
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_115.zip
Uploading the intermediate file: upload_file_115.zip
  12:47:31 STEP 2786 H=2786:00 +CPU=  9.434
  12:47:46 STEP 2787 H=2787:00 +CPU= 14.865
  12:47:56 STEP 2788 H=2788:00 +CPU=  9.493
  12:48:05 STEP 2789 H=2789:00 +CPU=  9.177
  12:48:20 STEP 2790 H=2790:00 +CPU= 15.151
  12:48:29 STEP 2791 H=2791:00 +CPU=  9.001
  12:48:38 STEP 2792 H=2792:00 +CPU=  9.265
  12:48:53 STEP 2793 H=2793:00 +CPU= 14.724
  12:49:03 STEP 2794 H=2794:00 +CPU=  9.811
  12:49:13 STEP 2795 H=2795:00 +CPU= 10.168
  12:49:29 STEP 2796 H=2796:00 +CPU= 16.414
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002796
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002796
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002796
  12:49:39 STEP 2797 H=2797:00 +CPU=  9.117
  12:49:48 STEP 2798 H=2798:00 +CPU=  9.403
  12:50:03 STEP 2799 H=2799:00 +CPU= 14.586
  12:50:12 STEP 2800 H=2800:00 +CPU=  9.322
  12:50:21 STEP 2801 H=2801:00 +CPU=  9.100
  12:50:36 STEP 2802 H=2802:00 +CPU= 14.860
  12:50:45 STEP 2803 H=2803:00 +CPU=  8.821
  12:50:54 STEP 2804 H=2804:00 +CPU=  9.193
  12:51:08 STEP 2805 H=2805:00 +CPU= 14.550
  12:51:18 STEP 2806 H=2806:00 +CPU=  9.176
  12:51:27 STEP 2807 H=2807:00 +CPU=  9.200
  12:51:45 STEP 2808 H=2808:00 +CPU= 17.045
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002808
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002808
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002796
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_116.zip
Uploading the intermediate file: upload_file_116.zip
  12:51:54 STEP 2809 H=2809:00 +CPU=  8.967
  12:52:03 STEP 2810 H=2810:00 +CPU=  9.221
  12:52:18 STEP 2811 H=2811:00 +CPU= 14.848
  12:52:29 STEP 2812 H=2812:00 +CPU= 10.876
  12:52:38 STEP 2813 H=2813:00 +CPU=  9.121
  12:52:53 STEP 2814 H=2814:00 +CPU= 14.881
  12:53:02 STEP 2815 H=2815:00 +CPU=  8.900
  12:53:11 STEP 2816 H=2816:00 +CPU=  9.183
  12:53:25 STEP 2817 H=2817:00 +CPU= 14.511
  12:53:35 STEP 2818 H=2818:00 +CPU=  9.296
  12:53:44 STEP 2819 H=2819:00 +CPU=  8.910
  12:54:00 STEP 2820 H=2820:00 +CPU= 16.240
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002820
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002820
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002820
  12:54:09 STEP 2821 H=2821:00 +CPU=  8.963
  12:54:18 STEP 2822 H=2822:00 +CPU=  9.222
  12:54:33 STEP 2823 H=2823:00 +CPU= 14.662
  12:54:42 STEP 2824 H=2824:00 +CPU=  9.232
  12:54:51 STEP 2825 H=2825:00 +CPU=  9.014
  12:55:06 STEP 2826 H=2826:00 +CPU= 14.993
  12:55:15 STEP 2827 H=2827:00 +CPU=  9.000
  12:55:24 STEP 2828 H=2828:00 +CPU=  9.338
  12:55:39 STEP 2829 H=2829:00 +CPU= 14.473
  12:55:49 STEP 2830 H=2830:00 +CPU= 10.471
  12:55:59 STEP 2831 H=2831:00 +CPU=  9.495
  12:56:18 STEP 2832 H=2832:00 +CPU= 17.128
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002832
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002832
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002820
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_117.zip
Uploading the intermediate file: upload_file_117.zip
  12:56:27 STEP 2833 H=2833:00 +CPU=  8.980
  12:56:36 STEP 2834 H=2834:00 +CPU=  9.281
  12:56:51 STEP 2835 H=2835:00 +CPU= 14.665
  12:57:00 STEP 2836 H=2836:00 +CPU=  9.250
  12:57:09 STEP 2837 H=2837:00 +CPU=  9.018
  12:57:24 STEP 2838 H=2838:00 +CPU= 14.975
  12:57:33 STEP 2839 H=2839:00 +CPU=  8.892
  12:57:42 STEP 2840 H=2840:00 +CPU=  9.184
  12:57:57 STEP 2841 H=2841:00 +CPU= 14.510
  12:58:06 STEP 2842 H=2842:00 +CPU=  9.225
  12:58:15 STEP 2843 H=2843:00 +CPU=  8.980
  12:58:31 STEP 2844 H=2844:00 +CPU= 16.158
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002844
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002844
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002844
  12:58:40 STEP 2845 H=2845:00 +CPU=  8.874
  12:58:49 STEP 2846 H=2846:00 +CPU=  9.196
  12:59:04 STEP 2847 H=2847:00 +CPU= 14.665
  12:59:13 STEP 2848 H=2848:00 +CPU=  9.190
  12:59:24 STEP 2849 H=2849:00 +CPU= 10.607
  12:59:39 STEP 2850 H=2850:00 +CPU= 14.939
  12:59:48 STEP 2851 H=2851:00 +CPU=  8.942
  12:59:57 STEP 2852 H=2852:00 +CPU=  9.330
  13:00:11 STEP 2853 H=2853:00 +CPU= 14.427
  13:00:21 STEP 2854 H=2854:00 +CPU=  9.200
  13:00:30 STEP 2855 H=2855:00 +CPU=  9.359
  13:00:48 STEP 2856 H=2856:00 +CPU= 16.911
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002856
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002856
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002844
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_118.zip
Uploading the intermediate file: upload_file_118.zip
  13:00:57 STEP 2857 H=2857:00 +CPU=  9.028
  13:01:07 STEP 2858 H=2858:00 +CPU=  9.264
  13:01:21 STEP 2859 H=2859:00 +CPU= 14.458
  13:01:30 STEP 2860 H=2860:00 +CPU=  9.230
  13:01:39 STEP 2861 H=2861:00 +CPU=  8.972
  13:01:55 STEP 2862 H=2862:00 +CPU= 16.026
  13:02:05 STEP 2863 H=2863:00 +CPU=  9.082
  13:02:14 STEP 2864 H=2864:00 +CPU=  9.197
  13:02:28 STEP 2865 H=2865:00 +CPU= 14.485
  13:02:37 STEP 2866 H=2866:00 +CPU=  9.209
  13:02:46 STEP 2867 H=2867:00 +CPU=  8.959
  13:03:02 STEP 2868 H=2868:00 +CPU= 16.089
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002868
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002868
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002868
  13:03:11 STEP 2869 H=2869:00 +CPU=  8.981
  13:03:21 STEP 2870 H=2870:00 +CPU=  9.247
  13:03:35 STEP 2871 H=2871:00 +CPU= 14.453
  13:03:44 STEP 2872 H=2872:00 +CPU=  9.198
  13:03:53 STEP 2873 H=2873:00 +CPU=  8.963
  13:04:08 STEP 2874 H=2874:00 +CPU= 14.712
  13:04:17 STEP 2875 H=2875:00 +CPU=  8.934
  13:04:28 STEP 2876 H=2876:00 +CPU= 10.619
  13:04:42 STEP 2877 H=2877:00 +CPU= 14.497
  13:04:51 STEP 2878 H=2878:00 +CPU=  9.081
  13:05:00 STEP 2879 H=2879:00 +CPU=  9.182
  13:05:18 STEP 2880 H=2880:00 +CPU= 17.037
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002880
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002880
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002868
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_119.zip
Uploading the intermediate file: upload_file_119.zip
  13:05:28 STEP 2881 H=2881:00 +CPU=  9.085
  13:05:37 STEP 2882 H=2882:00 +CPU=  9.186
  13:05:51 STEP 2883 H=2883:00 +CPU= 14.524
  13:06:01 STEP 2884 H=2884:00 +CPU=  9.248
  13:06:09 STEP 2885 H=2885:00 +CPU=  8.925
  13:06:24 STEP 2886 H=2886:00 +CPU= 14.866
  13:06:33 STEP 2887 H=2887:00 +CPU=  8.931
  13:06:42 STEP 2888 H=2888:00 +CPU=  9.228
  13:06:57 STEP 2889 H=2889:00 +CPU= 14.575
  13:07:06 STEP 2890 H=2890:00 +CPU=  9.109
  13:07:15 STEP 2891 H=2891:00 +CPU=  8.977
  13:07:31 STEP 2892 H=2892:00 +CPU= 16.181
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002892
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002892
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002892
  13:07:40 STEP 2893 H=2893:00 +CPU=  8.939
  13:07:49 STEP 2894 H=2894:00 +CPU=  9.160
  13:08:06 STEP 2895 H=2895:00 +CPU= 16.096
  13:08:15 STEP 2896 H=2896:00 +CPU=  9.178
  13:08:24 STEP 2897 H=2897:00 +CPU=  8.943
  13:08:39 STEP 2898 H=2898:00 +CPU= 14.925
  13:08:48 STEP 2899 H=2899:00 +CPU=  8.962
  13:08:57 STEP 2900 H=2900:00 +CPU=  9.138
  13:09:11 STEP 2901 H=2901:00 +CPU= 14.521
  13:09:20 STEP 2902 H=2902:00 +CPU=  9.088
  13:09:30 STEP 2903 H=2903:00 +CPU=  9.229
  13:09:48 STEP 2904 H=2904:00 +CPU= 16.907
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002904
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002904
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002892
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_120.zip
Uploading the intermediate file: upload_file_120.zip
  13:09:57 STEP 2905 H=2905:00 +CPU=  8.985
  13:10:06 STEP 2906 H=2906:00 +CPU=  9.269
Uploading trickle at timestep: 10454400
  13:10:20 STEP 2907 H=2907:00 +CPU= 14.492
  13:10:30 STEP 2908 H=2908:00 +CPU=  9.269
  13:10:39 STEP 2909 H=2909:00 +CPU=  8.987
  13:10:53 STEP 2910 H=2910:00 +CPU= 14.787
  13:11:02 STEP 2911 H=2911:00 +CPU=  8.914
  13:11:12 STEP 2912 H=2912:00 +CPU=  9.193
  13:11:28 STEP 2913 H=2913:00 +CPU= 15.981
  13:11:37 STEP 2914 H=2914:00 +CPU=  9.175
  13:11:46 STEP 2915 H=2915:00 +CPU=  8.935
  13:12:02 STEP 2916 H=2916:00 +CPU= 16.113
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002916
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002916
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002916
  13:12:11 STEP 2917 H=2917:00 +CPU=  8.891
  13:12:20 STEP 2918 H=2918:00 +CPU=  9.231
  13:12:34 STEP 2919 H=2919:00 +CPU= 14.522
  13:12:44 STEP 2920 H=2920:00 +CPU=  9.115
  13:12:52 STEP 2921 H=2921:00 +CPU=  8.917
  13:13:07 STEP 2922 H=2922:00 +CPU= 14.790
  13:13:16 STEP 2923 H=2923:00 +CPU=  8.957
  13:13:25 STEP 2924 H=2924:00 +CPU=  9.217
  13:13:40 STEP 2925 H=2925:00 +CPU= 14.325
  13:13:49 STEP 2926 H=2926:00 +CPU=  9.307
  13:13:58 STEP 2927 H=2927:00 +CPU=  9.238
  13:14:19 STEP 2928 H=2928:00 +CPU= 16.955
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002928
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002928
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002916
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_121.zip
Uploading the intermediate file: upload_file_121.zip
  13:14:45 STEP 2929 H=2929:00 +CPU= 10.224
  13:14:55 STEP 2930 H=2930:00 +CPU=  9.491
  13:15:09 STEP 2931 H=2931:00 +CPU= 14.536
  13:15:19 STEP 2932 H=2932:00 +CPU=  9.150
  13:15:28 STEP 2933 H=2933:00 +CPU=  9.012
  13:15:43 STEP 2934 H=2934:00 +CPU= 14.913
  13:15:52 STEP 2935 H=2935:00 +CPU=  9.012
  13:16:01 STEP 2936 H=2936:00 +CPU=  9.250
  13:16:16 STEP 2937 H=2937:00 +CPU= 14.575
  13:16:25 STEP 2938 H=2938:00 +CPU=  9.097
  13:16:34 STEP 2939 H=2939:00 +CPU=  8.983
  13:16:50 STEP 2940 H=2940:00 +CPU= 16.047
  13:16:59 STEP 2941 H=2941:00 +CPU=  8.841
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002940
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002940
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002940
  13:17:08 STEP 2942 H=2942:00 +CPU=  9.245
  13:17:22 STEP 2943 H=2943:00 +CPU= 14.492
  13:17:31 STEP 2944 H=2944:00 +CPU=  9.098
  13:17:40 STEP 2945 H=2945:00 +CPU=  8.987
  13:17:57 STEP 2946 H=2946:00 +CPU= 16.156
  13:18:06 STEP 2947 H=2947:00 +CPU=  8.974
  13:18:15 STEP 2948 H=2948:00 +CPU=  9.016
  13:18:29 STEP 2949 H=2949:00 +CPU= 14.365
  13:18:38 STEP 2950 H=2950:00 +CPU=  9.038
  13:18:47 STEP 2951 H=2951:00 +CPU=  8.966
  13:19:04 STEP 2952 H=2952:00 +CPU= 16.005
The child process terminated with status: 0
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002952
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002952
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMGGhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMSHhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12168742/ICMUAhq0f+002928
Zipping up the final file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0098_1983050100_123_952_12168742_0_r1944436727_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
13:22:27 (80): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:09:49 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 10,623,600 41,474 0.0039
09 May 2023 16:09:49 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 10,454,400 40,946 0.0039
09 May 2023 16:09:49 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 10,454,400 40,946 0.0039
09 May 2023 16:09:49 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 9,590,400 38,243 0.0040
09 May 2023 16:09:45 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 8,726,400 35,539 0.0041
09 May 2023 16:09:42 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 7,862,400 32,824 0.0042
09 May 2023 16:09:42 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 7,862,400 32,824 0.0042
09 May 2023 16:09:38 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 6,998,400 30,116 0.0043
09 May 2023 16:09:38 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 6,998,400 30,116 0.0043
09 May 2023 16:09:34 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 6,048,000 27,097 0.0045
09 May 2023 16:09:32 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 5,184,000 24,403 0.0047
09 May 2023 16:09:32 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 4,320,000 21,715 0.0050
09 May 2023 16:09:32 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 4,320,000 21,715 0.0050
09 May 2023 16:09:28 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 3,456,000 19,023 0.0055
09 May 2023 16:09:24 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 2,592,000 15,703 0.0061
09 May 2023 16:09:17 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 1,728,000 10,179 0.0059
09 May 2023 16:09:13 1537625 22253008 oifs_43r3_ps_0098_1983050100_123_952_12168742_0 864,000 4,965 0.0057


©2024 climateprediction.net