climateprediction.net home page
Task 22296071

Task 22296071

Name oifs_43r3_ps_0002_1997050100_123_966_12182646_1
Workunit 12182646
Created 6 Jan 2023, 15:15:09 UTC
Sent 6 Jan 2023, 15:47:02 UTC
Report deadline 5 Feb 2023, 15:47:02 UTC
Received 26 Jan 2023, 3:47:03 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1545360
Run time 18 hours 18 min 30 sec
CPU time 17 hours 43 min 49 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 7.16 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.26 MB
Peak swap size 4,978.07 MB
Peak disk usage 6,290.14 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
+CPU= 20.889
  07:01:35 STEP 2529 H=2529:00 +CPU= 30.979
  07:01:56 STEP 2530 H=2530:00 +CPU= 20.204
  07:02:17 STEP 2531 H=2531:00 +CPU= 20.560
  07:02:53 STEP 2532 H=2532:00 +CPU= 34.022
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002532
  07:03:13 STEP 2533 H=2533:00 +CPU= 19.976
  07:03:35 STEP 2534 H=2534:00 +CPU= 21.057
  07:04:07 STEP 2535 H=2535:00 +CPU= 30.842
  07:04:28 STEP 2536 H=2536:00 +CPU= 21.003
  07:04:49 STEP 2537 H=2537:00 +CPU= 20.473
  07:05:21 STEP 2538 H=2538:00 +CPU= 30.947
  07:05:42 STEP 2539 H=2539:00 +CPU= 19.966
  07:06:03 STEP 2540 H=2540:00 +CPU= 20.939
  07:06:35 STEP 2541 H=2541:00 +CPU= 30.665
  07:06:56 STEP 2542 H=2542:00 +CPU= 20.317
  07:07:18 STEP 2543 H=2543:00 +CPU= 21.357
  07:07:56 STEP 2544 H=2544:00 +CPU= 36.684
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_105.zip
Uploading the intermediate file: upload_file_105.zip
  07:08:18 STEP 2545 H=2545:00 +CPU= 20.659
  07:08:39 STEP 2546 H=2546:00 +CPU= 20.337
  07:09:10 STEP 2547 H=2547:00 +CPU= 30.448
  07:09:31 STEP 2548 H=2548:00 +CPU= 20.303
  07:09:52 STEP 2549 H=2549:00 +CPU= 19.959
  07:10:25 STEP 2550 H=2550:00 +CPU= 31.387
  07:10:45 STEP 2551 H=2551:00 +CPU= 20.146
  07:11:07 STEP 2552 H=2552:00 +CPU= 20.858
  07:11:38 STEP 2553 H=2553:00 +CPU= 30.621
  07:12:00 STEP 2554 H=2554:00 +CPU= 20.570
  07:12:20 STEP 2555 H=2555:00 +CPU= 19.691
  07:12:54 STEP 2556 H=2556:00 +CPU= 33.207
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002556
  07:13:15 STEP 2557 H=2557:00 +CPU= 20.051
  07:13:36 STEP 2558 H=2558:00 +CPU= 20.319
  07:14:08 STEP 2559 H=2559:00 +CPU= 31.056
  07:14:30 STEP 2560 H=2560:00 +CPU= 21.081
  07:14:51 STEP 2561 H=2561:00 +CPU= 20.460
  07:15:24 STEP 2562 H=2562:00 +CPU= 31.767
  07:15:45 STEP 2563 H=2563:00 +CPU= 20.106
  07:16:06 STEP 2564 H=2564:00 +CPU= 20.580
  07:16:38 STEP 2565 H=2565:00 +CPU= 31.095
  07:16:59 STEP 2566 H=2566:00 +CPU= 20.636
  07:17:21 STEP 2567 H=2567:00 +CPU= 20.892
  07:17:58 STEP 2568 H=2568:00 +CPU= 36.539
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_106.zip
Uploading the intermediate file: upload_file_106.zip
  07:18:19 STEP 2569 H=2569:00 +CPU= 19.867
  07:18:40 STEP 2570 H=2570:00 +CPU= 20.540
  07:19:12 STEP 2571 H=2571:00 +CPU= 30.768
  07:19:33 STEP 2572 H=2572:00 +CPU= 20.634
  07:19:54 STEP 2573 H=2573:00 +CPU= 20.189
  07:20:27 STEP 2574 H=2574:00 +CPU= 31.774
  07:20:47 STEP 2575 H=2575:00 +CPU= 20.281
  07:21:08 STEP 2576 H=2576:00 +CPU= 20.507
  07:21:41 STEP 2577 H=2577:00 +CPU= 31.276
  07:22:02 STEP 2578 H=2578:00 +CPU= 21.180
  07:22:23 STEP 2579 H=2579:00 +CPU= 19.866
  07:22:58 STEP 2580 H=2580:00 +CPU= 33.640
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002580
  07:23:18 STEP 2581 H=2581:00 +CPU= 20.370
  07:23:39 STEP 2582 H=2582:00 +CPU= 20.385
  07:24:11 STEP 2583 H=2583:00 +CPU= 31.348
  07:24:33 STEP 2584 H=2584:00 +CPU= 20.715
  07:24:53 STEP 2585 H=2585:00 +CPU= 19.423
  07:25:26 STEP 2586 H=2586:00 +CPU= 31.434
  07:25:46 STEP 2587 H=2587:00 +CPU= 20.223
  07:26:08 STEP 2588 H=2588:00 +CPU= 20.480
  07:26:41 STEP 2589 H=2589:00 +CPU= 31.499
  07:27:02 STEP 2590 H=2590:00 +CPU= 20.964
  07:27:24 STEP 2591 H=2591:00 +CPU= 20.840
  07:28:02 STEP 2592 H=2592:00 +CPU= 37.025
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_107.zip
Uploading the intermediate file: upload_file_107.zip
  07:28:23 STEP 2593 H=2593:00 +CPU= 20.121
  07:28:44 STEP 2594 H=2594:00 +CPU= 20.941
  07:29:16 STEP 2595 H=2595:00 +CPU= 31.060
  07:29:38 STEP 2596 H=2596:00 +CPU= 21.380
  07:29:59 STEP 2597 H=2597:00 +CPU= 20.018
  07:30:31 STEP 2598 H=2598:00 +CPU= 31.756
  07:30:53 STEP 2599 H=2599:00 +CPU= 20.547
  07:31:14 STEP 2600 H=2600:00 +CPU= 20.641
  07:31:46 STEP 2601 H=2601:00 +CPU= 30.540
  07:32:07 STEP 2602 H=2602:00 +CPU= 20.562
  07:32:27 STEP 2603 H=2603:00 +CPU= 19.631
  07:33:03 STEP 2604 H=2604:00 +CPU= 34.041
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002604
  07:33:23 STEP 2605 H=2605:00 +CPU= 20.013
  07:33:45 STEP 2606 H=2606:00 +CPU= 20.693
  07:34:17 STEP 2607 H=2607:00 +CPU= 30.689
  07:34:38 STEP 2608 H=2608:00 +CPU= 20.998
  07:34:59 STEP 2609 H=2609:00 +CPU= 20.226
  07:35:32 STEP 2610 H=2610:00 +CPU= 31.671
  07:35:53 STEP 2611 H=2611:00 +CPU= 20.974
  07:36:14 STEP 2612 H=2612:00 +CPU= 20.510
  07:36:47 STEP 2613 H=2613:00 +CPU= 30.457
  07:37:08 STEP 2614 H=2614:00 +CPU= 20.805
  07:37:30 STEP 2615 H=2615:00 +CPU= 20.679
  07:38:07 STEP 2616 H=2616:00 +CPU= 36.035
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_108.zip
Uploading the intermediate file: upload_file_108.zip
  07:38:28 STEP 2617 H=2617:00 +CPU= 19.599
Uploading trickle at timestep: 9417600
  07:38:49 STEP 2618 H=2618:00 +CPU= 20.476
  07:39:21 STEP 2619 H=2619:00 +CPU= 30.742
  07:39:42 STEP 2620 H=2620:00 +CPU= 20.098
  07:40:03 STEP 2621 H=2621:00 +CPU= 20.598
  07:40:36 STEP 2622 H=2622:00 +CPU= 31.636
  07:40:57 STEP 2623 H=2623:00 +CPU= 20.374
  07:41:18 STEP 2624 H=2624:00 +CPU= 20.826
  07:41:50 STEP 2625 H=2625:00 +CPU= 30.655
  07:42:12 STEP 2626 H=2626:00 +CPU= 21.490
  07:42:32 STEP 2627 H=2627:00 +CPU= 19.396
  07:43:06 STEP 2628 H=2628:00 +CPU= 33.685
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002628
  07:43:27 STEP 2629 H=2629:00 +CPU= 20.142
  07:43:49 STEP 2630 H=2630:00 +CPU= 20.727
  07:44:20 STEP 2631 H=2631:00 +CPU= 30.695
  07:44:42 STEP 2632 H=2632:00 +CPU= 20.742
  07:45:03 STEP 2633 H=2633:00 +CPU= 20.175
  07:45:35 STEP 2634 H=2634:00 +CPU= 31.445
  07:45:57 STEP 2635 H=2635:00 +CPU= 20.682
  07:46:17 STEP 2636 H=2636:00 +CPU= 20.002
  07:46:50 STEP 2637 H=2637:00 +CPU= 31.551
  07:47:11 STEP 2638 H=2638:00 +CPU= 21.284
  07:47:32 STEP 2639 H=2639:00 +CPU= 20.424
  07:48:09 STEP 2640 H=2640:00 +CPU= 36.064
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_109.zip
Uploading the intermediate file: upload_file_109.zip
  07:48:30 STEP 2641 H=2641:00 +CPU= 19.970
  07:48:52 STEP 2642 H=2642:00 +CPU= 20.401
  07:49:23 STEP 2643 H=2643:00 +CPU= 30.755
  07:49:45 STEP 2644 H=2644:00 +CPU= 20.996
  07:50:06 STEP 2645 H=2645:00 +CPU= 20.040
  07:50:38 STEP 2646 H=2646:00 +CPU= 31.124
  07:50:59 STEP 2647 H=2647:00 +CPU= 20.106
  07:51:20 STEP 2648 H=2648:00 +CPU= 20.990
  07:51:52 STEP 2649 H=2649:00 +CPU= 30.588
  07:52:13 STEP 2650 H=2650:00 +CPU= 20.274
  07:52:34 STEP 2651 H=2651:00 +CPU= 20.378
  07:53:09 STEP 2652 H=2652:00 +CPU= 33.168
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002652
  07:53:29 STEP 2653 H=2653:00 +CPU= 19.598
  07:53:50 STEP 2654 H=2654:00 +CPU= 20.318
  07:54:22 STEP 2655 H=2655:00 +CPU= 30.074
  07:54:43 STEP 2656 H=2656:00 +CPU= 21.295
  07:55:04 STEP 2657 H=2657:00 +CPU= 20.133
  07:55:36 STEP 2658 H=2658:00 +CPU= 31.008
  07:55:57 STEP 2659 H=2659:00 +CPU= 19.961
  07:56:18 STEP 2660 H=2660:00 +CPU= 20.137
  07:56:49 STEP 2661 H=2661:00 +CPU= 30.665
  07:57:11 STEP 2662 H=2662:00 +CPU= 20.889
  07:57:32 STEP 2663 H=2663:00 +CPU= 20.771
  07:58:09 STEP 2664 H=2664:00 +CPU= 36.224
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_110.zip
Uploading the intermediate file: upload_file_110.zip
  07:58:30 STEP 2665 H=2665:00 +CPU= 20.036
  07:58:53 STEP 2666 H=2666:00 +CPU= 21.896
  07:59:24 STEP 2667 H=2667:00 +CPU= 30.112
  07:59:45 STEP 2668 H=2668:00 +CPU= 21.140
  08:00:06 STEP 2669 H=2669:00 +CPU= 20.496
  08:00:38 STEP 2670 H=2670:00 +CPU= 30.754
  08:00:59 STEP 2671 H=2671:00 +CPU= 20.254
  08:01:21 STEP 2672 H=2672:00 +CPU= 20.917
  08:01:52 STEP 2673 H=2673:00 +CPU= 30.692
  08:02:14 STEP 2674 H=2674:00 +CPU= 20.758
  08:02:35 STEP 2675 H=2675:00 +CPU= 20.598
  08:03:10 STEP 2676 H=2676:00 +CPU= 34.023
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002676
  08:03:31 STEP 2677 H=2677:00 +CPU= 20.174
  08:03:53 STEP 2678 H=2678:00 +CPU= 20.830
  08:04:24 STEP 2679 H=2679:00 +CPU= 30.453
  08:04:46 STEP 2680 H=2680:00 +CPU= 21.097
  08:05:06 STEP 2681 H=2681:00 +CPU= 19.651
  08:05:39 STEP 2682 H=2682:00 +CPU= 31.294
  08:06:00 STEP 2683 H=2683:00 +CPU= 19.881
  08:06:21 STEP 2684 H=2684:00 +CPU= 21.068
  08:06:52 STEP 2685 H=2685:00 +CPU= 29.886
  08:07:14 STEP 2686 H=2686:00 +CPU= 20.848
  08:07:35 STEP 2687 H=2687:00 +CPU= 20.237
  08:08:12 STEP 2688 H=2688:00 +CPU= 36.081
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_111.zip
Uploading the intermediate file: upload_file_111.zip
  08:08:33 STEP 2689 H=2689:00 +CPU= 20.590
  08:08:55 STEP 2690 H=2690:00 +CPU= 20.517
  08:09:26 STEP 2691 H=2691:00 +CPU= 30.295
  08:09:48 STEP 2692 H=2692:00 +CPU= 21.536
  08:10:09 STEP 2693 H=2693:00 +CPU= 20.372
  08:10:41 STEP 2694 H=2694:00 +CPU= 30.869
  08:11:01 STEP 2695 H=2695:00 +CPU= 20.150
  08:11:23 STEP 2696 H=2696:00 +CPU= 20.821
  08:11:54 STEP 2697 H=2697:00 +CPU= 30.303
  08:12:15 STEP 2698 H=2698:00 +CPU= 20.705
  08:12:36 STEP 2699 H=2699:00 +CPU= 19.815
  08:13:10 STEP 2700 H=2700:00 +CPU= 33.646
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002700
  08:13:31 STEP 2701 H=2701:00 +CPU= 19.702
  08:13:53 STEP 2702 H=2702:00 +CPU= 21.228
  08:14:25 STEP 2703 H=2703:00 +CPU= 30.837
  08:14:47 STEP 2704 H=2704:00 +CPU= 21.241
  08:15:08 STEP 2705 H=2705:00 +CPU= 20.346
  08:15:40 STEP 2706 H=2706:00 +CPU= 30.661
  08:16:01 STEP 2707 H=2707:00 +CPU= 20.466
  08:16:22 STEP 2708 H=2708:00 +CPU= 20.311
  08:16:53 STEP 2709 H=2709:00 +CPU= 30.501
  08:17:16 STEP 2710 H=2710:00 +CPU= 21.910
  08:17:37 STEP 2711 H=2711:00 +CPU= 20.519
  08:18:15 STEP 2712 H=2712:00 +CPU= 36.206
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_112.zip
Uploading the intermediate file: upload_file_112.zip
  08:18:36 STEP 2713 H=2713:00 +CPU= 20.312
  08:18:57 STEP 2714 H=2714:00 +CPU= 20.787
  08:19:29 STEP 2715 H=2715:00 +CPU= 30.715
  08:19:50 STEP 2716 H=2716:00 +CPU= 20.637
  08:20:11 STEP 2717 H=2717:00 +CPU= 20.340
  08:20:43 STEP 2718 H=2718:00 +CPU= 31.416
  08:21:04 STEP 2719 H=2719:00 +CPU= 20.075
  08:21:25 STEP 2720 H=2720:00 +CPU= 20.676
  08:21:57 STEP 2721 H=2721:00 +CPU= 30.335
  08:22:18 STEP 2722 H=2722:00 +CPU= 20.365
  08:22:38 STEP 2723 H=2723:00 +CPU= 19.948
  08:23:13 STEP 2724 H=2724:00 +CPU= 33.733
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002724
  08:23:34 STEP 2725 H=2725:00 +CPU= 20.175
  08:23:55 STEP 2726 H=2726:00 +CPU= 20.952
  08:24:27 STEP 2727 H=2727:00 +CPU= 30.295
  08:24:48 STEP 2728 H=2728:00 +CPU= 21.291
  08:25:08 STEP 2729 H=2729:00 +CPU= 19.609
  08:25:41 STEP 2730 H=2730:00 +CPU= 31.616
  08:26:02 STEP 2731 H=2731:00 +CPU= 20.004
  08:26:23 STEP 2732 H=2732:00 +CPU= 20.705
  08:26:55 STEP 2733 H=2733:00 +CPU= 30.869
  08:27:16 STEP 2734 H=2734:00 +CPU= 20.749
  08:27:38 STEP 2735 H=2735:00 +CPU= 20.554
  08:28:16 STEP 2736 H=2736:00 +CPU= 36.910
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_113.zip
Uploading the intermediate file: upload_file_113.zip
  08:28:37 STEP 2737 H=2737:00 +CPU= 20.229
  08:28:59 STEP 2738 H=2738:00 +CPU= 21.134
  08:29:31 STEP 2739 H=2739:00 +CPU= 30.624
  08:29:52 STEP 2740 H=2740:00 +CPU= 20.587
  08:30:13 STEP 2741 H=2741:00 +CPU= 20.223
  08:30:46 STEP 2742 H=2742:00 +CPU= 31.471
  08:31:06 STEP 2743 H=2743:00 +CPU= 19.556
  08:31:28 STEP 2744 H=2744:00 +CPU= 21.096
  08:32:00 STEP 2745 H=2745:00 +CPU= 30.936
  08:32:22 STEP 2746 H=2746:00 +CPU= 20.819
  08:32:45 STEP 2747 H=2747:00 +CPU= 20.474
  08:33:19 STEP 2748 H=2748:00 +CPU= 33.747
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002748
  08:33:40 STEP 2749 H=2749:00 +CPU= 19.957
  08:34:01 STEP 2750 H=2750:00 +CPU= 20.584
  08:34:32 STEP 2751 H=2751:00 +CPU= 30.422
  08:34:53 STEP 2752 H=2752:00 +CPU= 20.456
  08:35:14 STEP 2753 H=2753:00 +CPU= 20.353
  08:35:47 STEP 2754 H=2754:00 +CPU= 31.718
  08:36:07 STEP 2755 H=2755:00 +CPU= 19.954
  08:36:28 STEP 2756 H=2756:00 +CPU= 20.267
  08:37:00 STEP 2757 H=2757:00 +CPU= 30.475
  08:37:21 STEP 2758 H=2758:00 +CPU= 20.438
  08:37:43 STEP 2759 H=2759:00 +CPU= 21.362
  08:38:20 STEP 2760 H=2760:00 +CPU= 36.067
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_114.zip
Uploading the intermediate file: upload_file_114.zip
  08:38:41 STEP 2761 H=2761:00 +CPU= 19.618
  08:39:03 STEP 2762 H=2762:00 +CPU= 20.808
  08:39:35 STEP 2763 H=2763:00 +CPU= 30.898
  08:39:57 STEP 2764 H=2764:00 +CPU= 20.526
  08:40:17 STEP 2765 H=2765:00 +CPU= 19.876
  08:40:50 STEP 2766 H=2766:00 +CPU= 31.733
  08:41:11 STEP 2767 H=2767:00 +CPU= 20.425
  08:41:33 STEP 2768 H=2768:00 +CPU= 20.487
  08:42:04 STEP 2769 H=2769:00 +CPU= 31.016
  08:42:26 STEP 2770 H=2770:00 +CPU= 20.894
  08:42:46 STEP 2771 H=2771:00 +CPU= 19.578
  08:43:21 STEP 2772 H=2772:00 +CPU= 33.297
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002772
  08:43:41 STEP 2773 H=2773:00 +CPU= 19.870
  08:44:03 STEP 2774 H=2774:00 +CPU= 20.628
  08:44:34 STEP 2775 H=2775:00 +CPU= 30.750
  08:44:55 STEP 2776 H=2776:00 +CPU= 20.417
  08:45:16 STEP 2777 H=2777:00 +CPU= 20.074
  08:45:48 STEP 2778 H=2778:00 +CPU= 31.379
  08:46:09 STEP 2779 H=2779:00 +CPU= 20.179
  08:46:31 STEP 2780 H=2780:00 +CPU= 20.608
  08:47:02 STEP 2781 H=2781:00 +CPU= 30.726
  08:47:23 STEP 2782 H=2782:00 +CPU= 20.279
  08:47:45 STEP 2783 H=2783:00 +CPU= 20.582
  08:48:22 STEP 2784 H=2784:00 +CPU= 36.179
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_115.zip
Uploading the intermediate file: upload_file_115.zip
  08:48:43 STEP 2785 H=2785:00 +CPU= 19.895
  08:49:04 STEP 2786 H=2786:00 +CPU= 20.465
  08:49:36 STEP 2787 H=2787:00 +CPU= 30.429
  08:49:57 STEP 2788 H=2788:00 +CPU= 20.542
  08:50:18 STEP 2789 H=2789:00 +CPU= 19.892
  08:50:50 STEP 2790 H=2790:00 +CPU= 31.096
  08:51:10 STEP 2791 H=2791:00 +CPU= 19.547
  08:51:31 STEP 2792 H=2792:00 +CPU= 20.505
  08:52:03 STEP 2793 H=2793:00 +CPU= 30.727
  08:52:24 STEP 2794 H=2794:00 +CPU= 20.447
  08:52:45 STEP 2795 H=2795:00 +CPU= 20.355
  08:53:20 STEP 2796 H=2796:00 +CPU= 33.755
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002796
  08:53:41 STEP 2797 H=2797:00 +CPU= 20.276
  08:54:02 STEP 2798 H=2798:00 +CPU= 20.556
  08:54:37 STEP 2799 H=2799:00 +CPU= 32.388
  08:55:00 STEP 2800 H=2800:00 +CPU= 22.164
  08:55:24 STEP 2801 H=2801:00 +CPU= 22.613
  08:56:01 STEP 2802 H=2802:00 +CPU= 35.411
  08:56:24 STEP 2803 H=2803:00 +CPU= 21.619
  08:56:49 STEP 2804 H=2804:00 +CPU= 23.028
  08:57:26 STEP 2805 H=2805:00 +CPU= 33.609
  08:57:48 STEP 2806 H=2806:00 +CPU= 22.403
  08:58:11 STEP 2807 H=2807:00 +CPU= 22.551
  08:58:55 STEP 2808 H=2808:00 +CPU= 40.219
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_116.zip
Uploading the intermediate file: upload_file_116.zip
  08:59:19 STEP 2809 H=2809:00 +CPU= 21.567
  08:59:42 STEP 2810 H=2810:00 +CPU= 22.517
  09:00:18 STEP 2811 H=2811:00 +CPU= 33.800
  09:00:41 STEP 2812 H=2812:00 +CPU= 22.850
  09:01:06 STEP 2813 H=2813:00 +CPU= 22.538
  09:01:44 STEP 2814 H=2814:00 +CPU= 35.823
  09:02:06 STEP 2815 H=2815:00 +CPU= 22.004
  09:02:31 STEP 2816 H=2816:00 +CPU= 23.077
  09:03:06 STEP 2817 H=2817:00 +CPU= 33.803
  09:03:31 STEP 2818 H=2818:00 +CPU= 23.153
  09:03:54 STEP 2819 H=2819:00 +CPU= 22.449
  09:04:35 STEP 2820 H=2820:00 +CPU= 38.030
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002820
  09:05:00 STEP 2821 H=2821:00 +CPU= 22.877
  09:05:24 STEP 2822 H=2822:00 +CPU= 23.112
  09:06:01 STEP 2823 H=2823:00 +CPU= 34.005
  09:06:26 STEP 2824 H=2824:00 +CPU= 22.880
  09:06:50 STEP 2825 H=2825:00 +CPU= 21.863
  09:07:27 STEP 2826 H=2826:00 +CPU= 35.144
  09:07:49 STEP 2827 H=2827:00 +CPU= 21.301
  09:08:15 STEP 2828 H=2828:00 +CPU= 23.234
  09:08:52 STEP 2829 H=2829:00 +CPU= 33.863
  09:09:17 STEP 2830 H=2830:00 +CPU= 22.947
  09:09:43 STEP 2831 H=2831:00 +CPU= 23.467
  09:10:25 STEP 2832 H=2832:00 +CPU= 39.486
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_117.zip
Uploading the intermediate file: upload_file_117.zip
  09:10:49 STEP 2833 H=2833:00 +CPU= 21.431
  09:11:12 STEP 2834 H=2834:00 +CPU= 21.678
  09:11:48 STEP 2835 H=2835:00 +CPU= 34.323
  09:12:11 STEP 2836 H=2836:00 +CPU= 22.589
  09:12:35 STEP 2837 H=2837:00 +CPU= 22.796
  09:13:14 STEP 2838 H=2838:00 +CPU= 35.597
  09:13:38 STEP 2839 H=2839:00 +CPU= 22.561
  09:14:02 STEP 2840 H=2840:00 +CPU= 23.478
  09:14:39 STEP 2841 H=2841:00 +CPU= 34.831
  09:15:03 STEP 2842 H=2842:00 +CPU= 22.624
  09:15:29 STEP 2843 H=2843:00 +CPU= 23.289
  09:16:07 STEP 2844 H=2844:00 +CPU= 36.154
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002844
  09:16:32 STEP 2845 H=2845:00 +CPU= 22.211
  09:16:56 STEP 2846 H=2846:00 +CPU= 22.757
  09:17:32 STEP 2847 H=2847:00 +CPU= 32.978
  09:17:56 STEP 2848 H=2848:00 +CPU= 22.908
  09:18:19 STEP 2849 H=2849:00 +CPU= 21.596
  09:18:56 STEP 2850 H=2850:00 +CPU= 34.931
  09:19:20 STEP 2851 H=2851:00 +CPU= 21.872
  09:19:46 STEP 2852 H=2852:00 +CPU= 23.426
  09:20:21 STEP 2853 H=2853:00 +CPU= 32.389
  09:20:45 STEP 2854 H=2854:00 +CPU= 22.760
  09:21:10 STEP 2855 H=2855:00 +CPU= 22.954
  09:21:52 STEP 2856 H=2856:00 +CPU= 39.378
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_118.zip
Uploading the intermediate file: upload_file_118.zip
  09:22:15 STEP 2857 H=2857:00 +CPU= 22.263
Uploading trickle at timestep: 10281600
  09:22:40 STEP 2858 H=2858:00 +CPU= 22.798
  09:23:14 STEP 2859 H=2859:00 +CPU= 33.203
  09:23:38 STEP 2860 H=2860:00 +CPU= 22.741
  09:24:01 STEP 2861 H=2861:00 +CPU= 22.395
  09:24:37 STEP 2862 H=2862:00 +CPU= 33.472
  09:25:00 STEP 2863 H=2863:00 +CPU= 21.543
  09:25:26 STEP 2864 H=2864:00 +CPU= 23.800
  09:26:01 STEP 2865 H=2865:00 +CPU= 34.013
  09:26:26 STEP 2866 H=2866:00 +CPU= 22.372
  09:26:49 STEP 2867 H=2867:00 +CPU= 22.783
  09:27:29 STEP 2868 H=2868:00 +CPU= 37.623
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002868
  09:27:53 STEP 2869 H=2869:00 +CPU= 21.974
  09:28:18 STEP 2870 H=2870:00 +CPU= 23.901
  09:28:54 STEP 2871 H=2871:00 +CPU= 33.469
  09:29:18 STEP 2872 H=2872:00 +CPU= 22.515
  09:29:43 STEP 2873 H=2873:00 +CPU= 23.115
  09:30:21 STEP 2874 H=2874:00 +CPU= 34.291
  09:30:45 STEP 2875 H=2875:00 +CPU= 22.162
  09:31:10 STEP 2876 H=2876:00 +CPU= 23.364
  09:31:46 STEP 2877 H=2877:00 +CPU= 33.591
  09:32:10 STEP 2878 H=2878:00 +CPU= 22.855
  09:32:36 STEP 2879 H=2879:00 +CPU= 23.176
  09:33:19 STEP 2880 H=2880:00 +CPU= 38.853
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_119.zip
Uploading the intermediate file: upload_file_119.zip
  09:33:42 STEP 2881 H=2881:00 +CPU= 21.823
  09:34:06 STEP 2882 H=2882:00 +CPU= 22.207
  09:34:41 STEP 2883 H=2883:00 +CPU= 32.696
  09:35:05 STEP 2884 H=2884:00 +CPU= 22.078
  09:35:30 STEP 2885 H=2885:00 +CPU= 22.771
  09:36:06 STEP 2886 H=2886:00 +CPU= 34.114
  09:36:29 STEP 2887 H=2887:00 +CPU= 22.091
  09:36:54 STEP 2888 H=2888:00 +CPU= 23.473
  09:37:31 STEP 2889 H=2889:00 +CPU= 33.767
  09:37:56 STEP 2890 H=2890:00 +CPU= 22.766
  09:38:20 STEP 2891 H=2891:00 +CPU= 21.919
  09:38:59 STEP 2892 H=2892:00 +CPU= 36.605
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002892
  09:39:23 STEP 2893 H=2893:00 +CPU= 21.482
  09:39:47 STEP 2894 H=2894:00 +CPU= 23.023
  09:40:23 STEP 2895 H=2895:00 +CPU= 33.739
  09:40:47 STEP 2896 H=2896:00 +CPU= 22.229
  09:41:08 STEP 2897 H=2897:00 +CPU= 21.066
  09:41:44 STEP 2898 H=2898:00 +CPU= 34.659
  09:42:06 STEP 2899 H=2899:00 +CPU= 21.027
  09:42:32 STEP 2900 H=2900:00 +CPU= 23.873
  09:43:08 STEP 2901 H=2901:00 +CPU= 34.244
  09:43:33 STEP 2902 H=2902:00 +CPU= 22.796
  09:43:57 STEP 2903 H=2903:00 +CPU= 23.194
  09:44:40 STEP 2904 H=2904:00 +CPU= 39.749
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_120.zip
Uploading the intermediate file: upload_file_120.zip
  09:45:03 STEP 2905 H=2905:00 +CPU= 21.588
  09:45:27 STEP 2906 H=2906:00 +CPU= 22.511
  09:46:03 STEP 2907 H=2907:00 +CPU= 33.855
  09:46:26 STEP 2908 H=2908:00 +CPU= 22.549
  09:46:51 STEP 2909 H=2909:00 +CPU= 23.312
  09:47:28 STEP 2910 H=2910:00 +CPU= 35.106
  09:47:50 STEP 2911 H=2911:00 +CPU= 20.275
  09:48:14 STEP 2912 H=2912:00 +CPU= 23.370
  09:48:51 STEP 2913 H=2913:00 +CPU= 34.130
  09:49:15 STEP 2914 H=2914:00 +CPU= 22.196
  09:49:39 STEP 2915 H=2915:00 +CPU= 22.040
  09:50:18 STEP 2916 H=2916:00 +CPU= 37.724
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002916
  09:50:41 STEP 2917 H=2917:00 +CPU= 20.933
  09:51:07 STEP 2918 H=2918:00 +CPU= 24.391
  09:51:44 STEP 2919 H=2919:00 +CPU= 35.146
  09:52:07 STEP 2920 H=2920:00 +CPU= 21.925
  09:52:31 STEP 2921 H=2921:00 +CPU= 22.560
  09:53:07 STEP 2922 H=2922:00 +CPU= 33.855
  09:53:29 STEP 2923 H=2923:00 +CPU= 20.574
  09:53:55 STEP 2924 H=2924:00 +CPU= 24.519
  09:54:34 STEP 2925 H=2925:00 +CPU= 35.143
  09:54:57 STEP 2926 H=2926:00 +CPU= 22.007
  09:55:21 STEP 2927 H=2927:00 +CPU= 23.500
  09:56:04 STEP 2928 H=2928:00 +CPU= 41.330
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_121.zip
Uploading the intermediate file: upload_file_121.zip
  09:56:27 STEP 2929 H=2929:00 +CPU= 21.532
  09:56:51 STEP 2930 H=2930:00 +CPU= 23.351
  09:57:28 STEP 2931 H=2931:00 +CPU= 34.280
  09:57:51 STEP 2932 H=2932:00 +CPU= 21.516
  09:58:16 STEP 2933 H=2933:00 +CPU= 22.516
  09:58:53 STEP 2934 H=2934:00 +CPU= 34.958
  09:59:15 STEP 2935 H=2935:00 +CPU= 20.987
  09:59:40 STEP 2936 H=2936:00 +CPU= 23.899
  10:00:17 STEP 2937 H=2937:00 +CPU= 35.048
  10:00:39 STEP 2938 H=2938:00 +CPU= 20.727
  10:01:03 STEP 2939 H=2939:00 +CPU= 22.147
  10:01:44 STEP 2940 H=2940:00 +CPU= 38.085
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002940
  10:02:05 STEP 2941 H=2941:00 +CPU= 20.020
  10:02:30 STEP 2942 H=2942:00 +CPU= 23.494
  10:03:08 STEP 2943 H=2943:00 +CPU= 35.953
  10:03:30 STEP 2944 H=2944:00 +CPU= 20.949
  10:03:54 STEP 2945 H=2945:00 +CPU= 23.024
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0002
(argv4) batchid: 966
(argv5) wuid: 12182646
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f1d6b26638c2dd2c960166d31a1fa56a to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0002_1997050100_123_966_12182646.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0002_1997050100_123_966_12182646.zip
ic_ancil_file: ic_ancil_12182646
ifsdata_file: ifsdata_12182646
climate_data_file: clim_data_12182646
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ce9aeb42b11f279c0b7a5b6599ef4ecb to: /var/lib/boinc-client/slots/2/ic_ancil_12182646.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12182646.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_da905816475439c4ad7149c1ff830c10 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12182646.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12182646.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12182646.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12182646.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12182646.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12182646.xml
-- Model is restarting --
Adjusting last_iter, 2946, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12182646.xml
last_cpu_time: 62827
upload_file_number: 122
last_iter: 2927
last_upload: 10540800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357
The child process has been launched with process id: 39729
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-16:1:1:39729:39729] [20230121:145843:1674334723:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:59:59 STEP 2928 H=2928:00 +CPU= 29.793
  15:00:21 STEP 2929 H=2929:00 +CPU= 20.779
  15:00:45 STEP 2930 H=2930:00 +CPU= 23.539
  15:01:20 STEP 2931 H=2931:00 +CPU= 33.490
  15:01:45 STEP 2932 H=2932:00 +CPU= 23.587
  15:02:09 STEP 2933 H=2933:00 +CPU= 23.239
  15:02:43 STEP 2934 H=2934:00 +CPU= 33.401
  15:03:07 STEP 2935 H=2935:00 +CPU= 23.503
  15:03:32 STEP 2936 H=2936:00 +CPU= 23.673
  15:04:06 STEP 2937 H=2937:00 +CPU= 33.282
  15:04:30 STEP 2938 H=2938:00 +CPU= 23.364
  15:04:54 STEP 2939 H=2939:00 +CPU= 22.725
  15:05:33 STEP 2940 H=2940:00 +CPU= 36.492
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002940
  15:05:56 STEP 2941 H=2941:00 +CPU= 22.314
  15:06:21 STEP 2942 H=2942:00 +CPU= 23.250
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0002
(argv4) batchid: 966
(argv5) wuid: 12182646
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f1d6b26638c2dd2c960166d31a1fa56a to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0002_1997050100_123_966_12182646.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0002_1997050100_123_966_12182646.zip
ic_ancil_file: ic_ancil_12182646
ifsdata_file: ifsdata_12182646
climate_data_file: clim_data_12182646
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ce9aeb42b11f279c0b7a5b6599ef4ecb to: /var/lib/boinc-client/slots/2/ic_ancil_12182646.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12182646.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_da905816475439c4ad7149c1ff830c10 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12182646.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12182646.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12182646.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12182646.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12182646.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12182646.xml
-- Model is restarting --
Adjusting last_iter, 2943, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12182646.xml
last_cpu_time: 63227
upload_file_number: 122
last_iter: 2927
last_upload: 10540800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357
The child process has been launched with process id: 315979
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-16:1:1:315979:315979] [20230123:172224:1674516144:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:23:05 STEP 2928 H=2928:00 +CPU= 37.176
  17:23:27 STEP 2929 H=2929:00 +CPU= 20.082
  17:23:49 STEP 2930 H=2930:00 +CPU= 20.938
  17:24:21 STEP 2931 H=2931:00 +CPU= 29.909
  17:24:43 STEP 2932 H=2932:00 +CPU= 20.907
  17:25:05 STEP 2933 H=2933:00 +CPU= 21.313
  17:25:40 STEP 2934 H=2934:00 +CPU= 33.483
  17:26:01 STEP 2935 H=2935:00 +CPU= 19.442
  17:26:19 STEP 2936 H=2936:00 +CPU= 17.911
  17:26:47 STEP 2937 H=2937:00 +CPU= 26.417
  17:27:05 STEP 2938 H=2938:00 +CPU= 17.125
  17:27:22 STEP 2939 H=2939:00 +CPU= 17.059
  17:27:59 STEP 2940 H=2940:00 +CPU= 34.426
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002940
  17:28:22 STEP 2941 H=2941:00 +CPU= 21.227
  17:28:45 STEP 2942 H=2942:00 +CPU= 21.482
  17:29:18 STEP 2943 H=2943:00 +CPU= 31.516
  17:29:39 STEP 2944 H=2944:00 +CPU= 20.489
  17:30:02 STEP 2945 H=2945:00 +CPU= 21.574
  17:30:38 STEP 2946 H=2946:00 +CPU= 33.965
  17:30:59 STEP 2947 H=2947:00 +CPU= 21.076
  17:31:21 STEP 2948 H=2948:00 +CPU= 21.348
  17:31:52 STEP 2949 H=2949:00 +CPU= 28.932
  17:32:09 STEP 2950 H=2950:00 +CPU= 16.752
  17:32:27 STEP 2951 H=2951:00 +CPU= 16.834
  17:32:57 STEP 2952 H=2952:00 +CPU= 28.816
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182646/ICMUAhq0f+002952
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0002_1997050100_123_966_12182646_1_r1259442357_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
17:36:21 (314898): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:38:31 1535273 22296071 oifs_43r3_ps_0002_1997050100_123_966_12182646_1 10,623,600 63,829 0.0060
09 May 2023 16:33:28 1535273 22296071 oifs_43r3_ps_0002_1997050100_123_966_12182646_1 10,281,600 60,459 0.0059
09 May 2023 16:33:22 1535273 22296071 oifs_43r3_ps_0002_1997050100_123_966_12182646_1 9,417,600 54,487 0.0058
09 May 2023 16:33:10 1535273 22296071 oifs_43r3_ps_0002_1997050100_123_966_12182646_1 8,553,600 48,645 0.0057
09 May 2023 16:32:44 1535273 22296071 oifs_43r3_ps_0002_1997050100_123_966_12182646_1 7,689,600 42,285 0.0055
09 May 2023 16:32:08 1535273 22296071 oifs_43r3_ps_0002_1997050100_123_966_12182646_1 6,048,000 31,420 0.0052
09 May 2023 16:31:13 1535273 22296071 oifs_43r3_ps_0002_1997050100_123_966_12182646_1 4,838,400 24,252 0.0050
09 May 2023 16:31:02 1535273 22296071 oifs_43r3_ps_0002_1997050100_123_966_12182646_1 3,974,400 19,831 0.0050
09 May 2023 16:30:56 1535273 22296071 oifs_43r3_ps_0002_1997050100_123_966_12182646_1 3,110,400 15,814 0.0051
09 May 2023 16:30:50 1535273 22296071 oifs_43r3_ps_0002_1997050100_123_966_12182646_1 2,246,400 11,791 0.0052
09 May 2023 16:30:39 1535273 22296071 oifs_43r3_ps_0002_1997050100_123_966_12182646_1 1,296,000 7,229 0.0056


©2024 climateprediction.net