climateprediction.net home page
Task 22316006

Task 22316006

Name oifs_43r3_00x6_2019110100_123_993_12214632_0
Workunit 12214632
Created 24 Feb 2023, 15:42:05 UTC
Sent 24 Feb 2023, 15:46:33 UTC
Report deadline 25 Apr 2023, 15:46:33 UTC
Received 7 Mar 2023, 18:35:09 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537224
Run time 18 hours 19 min
CPU time 18 hours 19 min
Validate state Valid
Credit 14,873.04
Device peak FLOPS 5.24 GFLOPS
Application version OpenIFS 43r3 v1.21
x86_64-pc-linux-gnu
Peak working set size 4,420.93 MB
Peak swap size 4,983.91 MB
Peak disk usage 1,984.54 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<stderr_txt>
5:56 STEP 2522 H=2522:00 +CPU= 15.650
  19:46:19 STEP 2523 H=2523:00 +CPU= 23.896
  19:46:35 STEP 2524 H=2524:00 +CPU= 15.647
  19:46:50 STEP 2525 H=2525:00 +CPU= 15.193
  19:47:15 STEP 2526 H=2526:00 +CPU= 24.326
  19:47:30 STEP 2527 H=2527:00 +CPU= 15.236
  19:47:46 STEP 2528 H=2528:00 +CPU= 15.625
  19:48:10 STEP 2529 H=2529:00 +CPU= 23.946
  19:48:25 STEP 2530 H=2530:00 +CPU= 15.673
  19:48:41 STEP 2531 H=2531:00 +CPU= 15.244
  19:49:07 STEP 2532 H=2532:00 +CPU= 25.956
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002532
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002532
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002532
  19:49:22 STEP 2533 H=2533:00 +CPU= 15.230
  19:49:37 STEP 2534 H=2534:00 +CPU= 15.648
  19:50:01 STEP 2535 H=2535:00 +CPU= 23.888
  19:50:17 STEP 2536 H=2536:00 +CPU= 15.575
  19:50:32 STEP 2537 H=2537:00 +CPU= 15.138
  19:50:57 STEP 2538 H=2538:00 +CPU= 24.290
  19:51:12 STEP 2539 H=2539:00 +CPU= 15.207
  19:51:27 STEP 2540 H=2540:00 +CPU= 15.609
  19:51:51 STEP 2541 H=2541:00 +CPU= 23.764
  19:52:07 STEP 2542 H=2542:00 +CPU= 15.564
  19:52:22 STEP 2543 H=2543:00 +CPU= 15.562
  19:52:50 STEP 2544 H=2544:00 +CPU= 27.707
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002544
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002544
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002544
  19:53:05 STEP 2545 H=2545:00 +CPU= 15.182
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002544
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_105.zip
Uploading the intermediate file: upload_file_105.zip
  19:53:21 STEP 2546 H=2546:00 +CPU= 15.580
  19:53:45 STEP 2547 H=2547:00 +CPU= 23.952
  19:54:01 STEP 2548 H=2548:00 +CPU= 15.616
  19:54:16 STEP 2549 H=2549:00 +CPU= 15.193
  19:54:40 STEP 2550 H=2550:00 +CPU= 24.281
  19:54:55 STEP 2551 H=2551:00 +CPU= 15.250
  19:55:11 STEP 2552 H=2552:00 +CPU= 15.558
  19:55:35 STEP 2553 H=2553:00 +CPU= 23.842
  19:55:50 STEP 2554 H=2554:00 +CPU= 15.637
  19:56:06 STEP 2555 H=2555:00 +CPU= 15.146
  19:56:32 STEP 2556 H=2556:00 +CPU= 25.930
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002556
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002556
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002556
  19:56:47 STEP 2557 H=2557:00 +CPU= 15.273
  19:57:03 STEP 2558 H=2558:00 +CPU= 15.677
  19:57:27 STEP 2559 H=2559:00 +CPU= 23.858
  19:57:42 STEP 2560 H=2560:00 +CPU= 15.536
  19:57:57 STEP 2561 H=2561:00 +CPU= 15.195
  19:58:21 STEP 2562 H=2562:00 +CPU= 24.171
  19:58:37 STEP 2563 H=2563:00 +CPU= 15.179
  19:58:52 STEP 2564 H=2564:00 +CPU= 15.563
  19:59:16 STEP 2565 H=2565:00 +CPU= 23.762
  19:59:32 STEP 2566 H=2566:00 +CPU= 15.552
  19:59:47 STEP 2567 H=2567:00 +CPU= 15.620
  20:00:15 STEP 2568 H=2568:00 +CPU= 27.658
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002568
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002568
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002568
  20:00:30 STEP 2569 H=2569:00 +CPU= 15.145
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002568
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_106.zip
Uploading the intermediate file: upload_file_106.zip
  20:00:46 STEP 2570 H=2570:00 +CPU= 15.490
  20:01:10 STEP 2571 H=2571:00 +CPU= 23.807
  20:01:25 STEP 2572 H=2572:00 +CPU= 15.636
  20:01:40 STEP 2573 H=2573:00 +CPU= 15.119
  20:02:05 STEP 2574 H=2574:00 +CPU= 24.252
  20:02:20 STEP 2575 H=2575:00 +CPU= 15.188
  20:02:35 STEP 2576 H=2576:00 +CPU= 15.506
  20:02:59 STEP 2577 H=2577:00 +CPU= 23.788
  20:03:15 STEP 2578 H=2578:00 +CPU= 15.529
  20:03:30 STEP 2579 H=2579:00 +CPU= 15.163
  20:03:56 STEP 2580 H=2580:00 +CPU= 25.963
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002580
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002580
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002580
  20:04:11 STEP 2581 H=2581:00 +CPU= 15.162
  20:04:27 STEP 2582 H=2582:00 +CPU= 15.612
  20:04:51 STEP 2583 H=2583:00 +CPU= 23.785
  20:05:06 STEP 2584 H=2584:00 +CPU= 15.558
  20:05:21 STEP 2585 H=2585:00 +CPU= 15.162
  20:05:46 STEP 2586 H=2586:00 +CPU= 24.332
  20:06:01 STEP 2587 H=2587:00 +CPU= 15.160
  20:06:16 STEP 2588 H=2588:00 +CPU= 15.490
  20:06:40 STEP 2589 H=2589:00 +CPU= 23.768
  20:06:56 STEP 2590 H=2590:00 +CPU= 15.562
  20:07:11 STEP 2591 H=2591:00 +CPU= 15.550
  20:07:39 STEP 2592 H=2592:00 +CPU= 27.560
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002592
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002592
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002592
  20:07:54 STEP 2593 H=2593:00 +CPU= 15.164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002592
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_107.zip
Uploading the intermediate file: upload_file_107.zip
  20:08:10 STEP 2594 H=2594:00 +CPU= 15.482
  20:08:33 STEP 2595 H=2595:00 +CPU= 23.732
  20:08:49 STEP 2596 H=2596:00 +CPU= 15.497
  20:09:04 STEP 2597 H=2597:00 +CPU= 15.155
  20:09:28 STEP 2598 H=2598:00 +CPU= 24.150
  20:09:44 STEP 2599 H=2599:00 +CPU= 15.171
  20:09:59 STEP 2600 H=2600:00 +CPU= 15.586
  20:10:23 STEP 2601 H=2601:00 +CPU= 23.818
  20:10:39 STEP 2602 H=2602:00 +CPU= 15.540
  20:10:54 STEP 2603 H=2603:00 +CPU= 15.173
  20:11:20 STEP 2604 H=2604:00 +CPU= 26.085
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002604
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002604
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002604
  20:11:35 STEP 2605 H=2605:00 +CPU= 15.162
  20:11:51 STEP 2606 H=2606:00 +CPU= 15.660
  20:12:15 STEP 2607 H=2607:00 +CPU= 23.696
  20:12:30 STEP 2608 H=2608:00 +CPU= 15.509
  20:12:45 STEP 2609 H=2609:00 +CPU= 15.118
  20:13:09 STEP 2610 H=2610:00 +CPU= 24.184
  20:13:24 STEP 2611 H=2611:00 +CPU= 15.042
  20:13:40 STEP 2612 H=2612:00 +CPU= 15.483
  20:14:04 STEP 2613 H=2613:00 +CPU= 23.724
  20:14:19 STEP 2614 H=2614:00 +CPU= 15.531
  20:14:35 STEP 2615 H=2615:00 +CPU= 15.496
  20:15:02 STEP 2616 H=2616:00 +CPU= 27.593
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002616
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002616
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002616
  20:15:18 STEP 2617 H=2617:00 +CPU= 15.068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002616
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_108.zip
Uploading the intermediate file: upload_file_108.zip
  20:15:33 STEP 2618 H=2618:00 +CPU= 15.536
  20:15:57 STEP 2619 H=2619:00 +CPU= 23.720
  20:16:12 STEP 2620 H=2620:00 +CPU= 15.427
  20:16:27 STEP 2621 H=2621:00 +CPU= 15.009
  20:16:52 STEP 2622 H=2622:00 +CPU= 24.168
  20:17:07 STEP 2623 H=2623:00 +CPU= 15.110
  20:17:22 STEP 2624 H=2624:00 +CPU= 15.508
  20:17:46 STEP 2625 H=2625:00 +CPU= 23.796
  20:18:02 STEP 2626 H=2626:00 +CPU= 15.601
  20:18:17 STEP 2627 H=2627:00 +CPU= 15.671
  20:18:43 STEP 2628 H=2628:00 +CPU= 25.915
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002628
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002628
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002628
  20:18:58 STEP 2629 H=2629:00 +CPU= 15.116
  20:19:14 STEP 2630 H=2630:00 +CPU= 15.560
  20:19:38 STEP 2631 H=2631:00 +CPU= 23.755
  20:19:53 STEP 2632 H=2632:00 +CPU= 15.505
  20:20:09 STEP 2633 H=2633:00 +CPU= 15.143
  20:20:33 STEP 2634 H=2634:00 +CPU= 24.237
  20:20:48 STEP 2635 H=2635:00 +CPU= 15.157
  20:21:04 STEP 2636 H=2636:00 +CPU= 15.503
  20:21:27 STEP 2637 H=2637:00 +CPU= 23.836
  20:21:43 STEP 2638 H=2638:00 +CPU= 15.440
  20:21:58 STEP 2639 H=2639:00 +CPU= 15.523
  20:22:26 STEP 2640 H=2640:00 +CPU= 27.632
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002640
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002640
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002640
  20:22:41 STEP 2641 H=2641:00 +CPU= 15.186
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002640
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_109.zip
Uploading the intermediate file: upload_file_109.zip
  20:22:57 STEP 2642 H=2642:00 +CPU= 15.549
Uploading trickle at timestep: 9507600
  20:23:21 STEP 2643 H=2643:00 +CPU= 23.843
  20:23:36 STEP 2644 H=2644:00 +CPU= 15.539
  20:23:52 STEP 2645 H=2645:00 +CPU= 15.376
  20:24:16 STEP 2646 H=2646:00 +CPU= 24.288
  20:24:31 STEP 2647 H=2647:00 +CPU= 15.164
  20:24:47 STEP 2648 H=2648:00 +CPU= 15.536
  20:25:11 STEP 2649 H=2649:00 +CPU= 23.873
  20:25:26 STEP 2650 H=2650:00 +CPU= 15.585
  20:25:41 STEP 2651 H=2651:00 +CPU= 15.158
  20:26:07 STEP 2652 H=2652:00 +CPU= 25.933
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002652
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002652
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002652
  20:26:23 STEP 2653 H=2653:00 +CPU= 15.116
  20:26:38 STEP 2654 H=2654:00 +CPU= 15.570
  20:27:02 STEP 2655 H=2655:00 +CPU= 23.902
  20:27:18 STEP 2656 H=2656:00 +CPU= 15.657
  20:27:33 STEP 2657 H=2657:00 +CPU= 15.204
  20:27:57 STEP 2658 H=2658:00 +CPU= 24.256
  20:28:13 STEP 2659 H=2659:00 +CPU= 15.196
  20:28:28 STEP 2660 H=2660:00 +CPU= 15.558
  20:28:52 STEP 2661 H=2661:00 +CPU= 23.840
  20:29:08 STEP 2662 H=2662:00 +CPU= 15.513
  20:29:23 STEP 2663 H=2663:00 +CPU= 15.557
  20:29:51 STEP 2664 H=2664:00 +CPU= 27.664
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002664
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002664
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002664
  20:30:06 STEP 2665 H=2665:00 +CPU= 15.096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002664
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_110.zip
Uploading the intermediate file: upload_file_110.zip
  20:30:22 STEP 2666 H=2666:00 +CPU= 15.603
  20:30:46 STEP 2667 H=2667:00 +CPU= 23.984
  20:31:01 STEP 2668 H=2668:00 +CPU= 15.601
  20:31:16 STEP 2669 H=2669:00 +CPU= 15.190
  20:31:41 STEP 2670 H=2670:00 +CPU= 24.362
  20:31:56 STEP 2671 H=2671:00 +CPU= 15.198
  20:32:12 STEP 2672 H=2672:00 +CPU= 15.776
  20:32:36 STEP 2673 H=2673:00 +CPU= 23.896
  20:32:51 STEP 2674 H=2674:00 +CPU= 15.568
  20:33:07 STEP 2675 H=2675:00 +CPU= 15.257
  20:33:33 STEP 2676 H=2676:00 +CPU= 25.989
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002676
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002676
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002676
  20:33:48 STEP 2677 H=2677:00 +CPU= 15.149
  20:34:04 STEP 2678 H=2678:00 +CPU= 15.728
  20:34:28 STEP 2679 H=2679:00 +CPU= 23.935
  20:34:43 STEP 2680 H=2680:00 +CPU= 15.586
  20:34:59 STEP 2681 H=2681:00 +CPU= 15.253
  20:35:23 STEP 2682 H=2682:00 +CPU= 24.297
  20:35:38 STEP 2683 H=2683:00 +CPU= 15.091
  20:35:54 STEP 2684 H=2684:00 +CPU= 15.670
  20:36:18 STEP 2685 H=2685:00 +CPU= 23.826
  20:36:33 STEP 2686 H=2686:00 +CPU= 15.455
  20:36:49 STEP 2687 H=2687:00 +CPU= 15.527
  20:37:16 STEP 2688 H=2688:00 +CPU= 27.699
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002688
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002688
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002688
  20:37:31 STEP 2689 H=2689:00 +CPU= 15.080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002688
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_111.zip
Uploading the intermediate file: upload_file_111.zip
  20:37:47 STEP 2690 H=2690:00 +CPU= 15.567
  20:38:11 STEP 2691 H=2691:00 +CPU= 23.962
  20:38:27 STEP 2692 H=2692:00 +CPU= 15.565
  20:38:42 STEP 2693 H=2693:00 +CPU= 15.179
  20:39:06 STEP 2694 H=2694:00 +CPU= 24.407
  20:39:21 STEP 2695 H=2695:00 +CPU= 15.119
  20:39:37 STEP 2696 H=2696:00 +CPU= 15.632
  20:40:01 STEP 2697 H=2697:00 +CPU= 23.831
  20:40:17 STEP 2698 H=2698:00 +CPU= 15.691
  20:40:32 STEP 2699 H=2699:00 +CPU= 15.213
  20:40:58 STEP 2700 H=2700:00 +CPU= 26.068
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002700
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002700
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002700
  20:41:13 STEP 2701 H=2701:00 +CPU= 15.270
  20:41:29 STEP 2702 H=2702:00 +CPU= 15.642
  20:41:53 STEP 2703 H=2703:00 +CPU= 23.905
  20:42:09 STEP 2704 H=2704:00 +CPU= 15.552
  20:42:24 STEP 2705 H=2705:00 +CPU= 15.232
  20:42:48 STEP 2706 H=2706:00 +CPU= 24.381
  20:43:03 STEP 2707 H=2707:00 +CPU= 15.086
  20:43:19 STEP 2708 H=2708:00 +CPU= 15.607
  20:43:43 STEP 2709 H=2709:00 +CPU= 23.864
  20:43:58 STEP 2710 H=2710:00 +CPU= 15.535
  20:44:14 STEP 2711 H=2711:00 +CPU= 15.593
  20:44:42 STEP 2712 H=2712:00 +CPU= 27.722
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002712
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002712
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002712
  20:44:57 STEP 2713 H=2713:00 +CPU= 15.024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002712
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_112.zip
Uploading the intermediate file: upload_file_112.zip
  20:45:12 STEP 2714 H=2714:00 +CPU= 15.596
  20:45:36 STEP 2715 H=2715:00 +CPU= 23.851
  20:45:52 STEP 2716 H=2716:00 +CPU= 15.603
  20:46:07 STEP 2717 H=2717:00 +CPU= 15.173
  20:46:32 STEP 2718 H=2718:00 +CPU= 24.388
  20:46:47 STEP 2719 H=2719:00 +CPU= 15.235
  20:47:03 STEP 2720 H=2720:00 +CPU= 15.656
  20:47:26 STEP 2721 H=2721:00 +CPU= 23.865
  20:47:42 STEP 2722 H=2722:00 +CPU= 15.557
  20:47:57 STEP 2723 H=2723:00 +CPU= 15.150
  20:48:23 STEP 2724 H=2724:00 +CPU= 26.054
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002724
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002724
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002724
  20:48:38 STEP 2725 H=2725:00 +CPU= 15.106
  20:48:54 STEP 2726 H=2726:00 +CPU= 15.617
  20:49:18 STEP 2727 H=2727:00 +CPU= 23.894
  20:49:33 STEP 2728 H=2728:00 +CPU= 15.444
  20:49:49 STEP 2729 H=2729:00 +CPU= 15.267
  20:50:13 STEP 2730 H=2730:00 +CPU= 24.441
  20:50:28 STEP 2731 H=2731:00 +CPU= 15.097
  20:50:44 STEP 2732 H=2732:00 +CPU= 15.682
  20:51:08 STEP 2733 H=2733:00 +CPU= 23.843
  20:51:23 STEP 2734 H=2734:00 +CPU= 15.491
  20:51:39 STEP 2735 H=2735:00 +CPU= 15.647
  20:52:07 STEP 2736 H=2736:00 +CPU= 27.782
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002736
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002736
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002736
  20:52:22 STEP 2737 H=2737:00 +CPU= 15.097
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002736
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_113.zip
Uploading the intermediate file: upload_file_113.zip
  20:52:38 STEP 2738 H=2738:00 +CPU= 15.723
  20:53:02 STEP 2739 H=2739:00 +CPU= 23.933
  20:53:17 STEP 2740 H=2740:00 +CPU= 15.499
  20:53:33 STEP 2741 H=2741:00 +CPU= 15.135
  20:53:57 STEP 2742 H=2742:00 +CPU= 24.320
  20:54:12 STEP 2743 H=2743:00 +CPU= 15.089
  20:54:28 STEP 2744 H=2744:00 +CPU= 15.736
  20:54:52 STEP 2745 H=2745:00 +CPU= 23.966
  20:55:07 STEP 2746 H=2746:00 +CPU= 15.580
  20:55:23 STEP 2747 H=2747:00 +CPU= 15.331
  20:55:49 STEP 2748 H=2748:00 +CPU= 26.152
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002748
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002748
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002748
  20:56:04 STEP 2749 H=2749:00 +CPU= 15.347
  20:56:20 STEP 2750 H=2750:00 +CPU= 15.801
  20:56:44 STEP 2751 H=2751:00 +CPU= 24.000
  20:57:00 STEP 2752 H=2752:00 +CPU= 15.585
  20:57:15 STEP 2753 H=2753:00 +CPU= 15.233
  20:57:40 STEP 2754 H=2754:00 +CPU= 24.519
  20:57:55 STEP 2755 H=2755:00 +CPU= 15.183
  20:58:10 STEP 2756 H=2756:00 +CPU= 15.659
  20:58:34 STEP 2757 H=2757:00 +CPU= 23.941
  20:58:50 STEP 2758 H=2758:00 +CPU= 15.532
  20:59:06 STEP 2759 H=2759:00 +CPU= 15.594
  20:59:34 STEP 2760 H=2760:00 +CPU= 27.876
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002760
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002760
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002760
  20:59:49 STEP 2761 H=2761:00 +CPU= 15.194
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002760
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_114.zip
Uploading the intermediate file: upload_file_114.zip
  21:00:05 STEP 2762 H=2762:00 +CPU= 15.697
  21:00:28 STEP 2763 H=2763:00 +CPU= 23.922
  21:00:44 STEP 2764 H=2764:00 +CPU= 15.541
  21:00:59 STEP 2765 H=2765:00 +CPU= 15.220
  21:01:24 STEP 2766 H=2766:00 +CPU= 24.563
  21:01:39 STEP 2767 H=2767:00 +CPU= 15.087
  21:01:55 STEP 2768 H=2768:00 +CPU= 15.700
  21:02:19 STEP 2769 H=2769:00 +CPU= 23.943
  21:02:34 STEP 2770 H=2770:00 +CPU= 15.518
  21:02:49 STEP 2771 H=2771:00 +CPU= 15.206
  21:03:16 STEP 2772 H=2772:00 +CPU= 26.056
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002772
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002772
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002772
  21:03:31 STEP 2773 H=2773:00 +CPU= 15.092
  21:03:46 STEP 2774 H=2774:00 +CPU= 15.673
  21:04:10 STEP 2775 H=2775:00 +CPU= 23.920
  21:04:26 STEP 2776 H=2776:00 +CPU= 15.529
  21:04:41 STEP 2777 H=2777:00 +CPU= 15.323
  21:05:06 STEP 2778 H=2778:00 +CPU= 24.367
  21:05:21 STEP 2779 H=2779:00 +CPU= 15.192
  21:05:37 STEP 2780 H=2780:00 +CPU= 15.678
  21:06:00 STEP 2781 H=2781:00 +CPU= 23.813
  21:06:16 STEP 2782 H=2782:00 +CPU= 15.518
  21:06:32 STEP 2783 H=2783:00 +CPU= 15.670
  21:06:59 STEP 2784 H=2784:00 +CPU= 27.767
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002784
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002784
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002784
  21:07:15 STEP 2785 H=2785:00 +CPU= 15.126
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002784
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_115.zip
Uploading the intermediate file: upload_file_115.zip
  21:07:30 STEP 2786 H=2786:00 +CPU= 15.523
  21:07:54 STEP 2787 H=2787:00 +CPU= 23.955
  21:08:10 STEP 2788 H=2788:00 +CPU= 15.537
  21:08:25 STEP 2789 H=2789:00 +CPU= 15.182
  21:08:49 STEP 2790 H=2790:00 +CPU= 24.425
  21:09:05 STEP 2791 H=2791:00 +CPU= 15.112
  21:09:20 STEP 2792 H=2792:00 +CPU= 15.676
  21:09:44 STEP 2793 H=2793:00 +CPU= 23.885
  21:10:00 STEP 2794 H=2794:00 +CPU= 15.541
  21:10:15 STEP 2795 H=2795:00 +CPU= 15.271
  21:10:41 STEP 2796 H=2796:00 +CPU= 26.218
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002796
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002796
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002796
  21:10:56 STEP 2797 H=2797:00 +CPU= 15.174
  21:11:12 STEP 2798 H=2798:00 +CPU= 15.543
  21:11:36 STEP 2799 H=2799:00 +CPU= 24.001
  21:11:52 STEP 2800 H=2800:00 +CPU= 15.465
  21:12:07 STEP 2801 H=2801:00 +CPU= 15.282
  21:12:31 STEP 2802 H=2802:00 +CPU= 24.399
  21:12:46 STEP 2803 H=2803:00 +CPU= 15.083
  21:13:02 STEP 2804 H=2804:00 +CPU= 15.619
  21:13:26 STEP 2805 H=2805:00 +CPU= 23.974
  21:13:42 STEP 2806 H=2806:00 +CPU= 15.413
  21:13:57 STEP 2807 H=2807:00 +CPU= 15.606
  21:14:25 STEP 2808 H=2808:00 +CPU= 27.837
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002808
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002808
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002808
  21:14:40 STEP 2809 H=2809:00 +CPU= 15.108
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002808
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_116.zip
Uploading the intermediate file: upload_file_116.zip
  21:14:56 STEP 2810 H=2810:00 +CPU= 15.607
  21:15:20 STEP 2811 H=2811:00 +CPU= 23.992
  21:15:35 STEP 2812 H=2812:00 +CPU= 15.572
  21:15:51 STEP 2813 H=2813:00 +CPU= 15.198
  21:16:15 STEP 2814 H=2814:00 +CPU= 24.487
  21:16:30 STEP 2815 H=2815:00 +CPU= 15.086
  21:16:46 STEP 2816 H=2816:00 +CPU= 15.601
  21:17:10 STEP 2817 H=2817:00 +CPU= 24.024
  21:17:26 STEP 2818 H=2818:00 +CPU= 15.569
  21:17:41 STEP 2819 H=2819:00 +CPU= 15.237
  21:18:07 STEP 2820 H=2820:00 +CPU= 26.082
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002820
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002820
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002820
  21:18:22 STEP 2821 H=2821:00 +CPU= 15.118
  21:18:38 STEP 2822 H=2822:00 +CPU= 15.544
  21:19:02 STEP 2823 H=2823:00 +CPU= 24.064
  21:19:17 STEP 2824 H=2824:00 +CPU= 15.432
  21:19:32 STEP 2825 H=2825:00 +CPU= 15.205
  21:19:57 STEP 2826 H=2826:00 +CPU= 24.436
  21:20:12 STEP 2827 H=2827:00 +CPU= 15.106
  21:20:28 STEP 2828 H=2828:00 +CPU= 15.589
  21:20:52 STEP 2829 H=2829:00 +CPU= 23.948
  21:21:07 STEP 2830 H=2830:00 +CPU= 15.378
  21:21:23 STEP 2831 H=2831:00 +CPU= 15.597
  21:21:51 STEP 2832 H=2832:00 +CPU= 27.878
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002832
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002832
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002832
  21:22:06 STEP 2833 H=2833:00 +CPU= 15.053
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002832
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_117.zip
Uploading the intermediate file: upload_file_117.zip
  21:22:21 STEP 2834 H=2834:00 +CPU= 15.517
  21:22:45 STEP 2835 H=2835:00 +CPU= 24.065
  21:23:01 STEP 2836 H=2836:00 +CPU= 15.657
  21:23:16 STEP 2837 H=2837:00 +CPU= 15.258
  21:23:41 STEP 2838 H=2838:00 +CPU= 24.474
  21:23:56 STEP 2839 H=2839:00 +CPU= 15.118
  21:24:12 STEP 2840 H=2840:00 +CPU= 15.610
  21:24:36 STEP 2841 H=2841:00 +CPU= 24.039
  21:24:51 STEP 2842 H=2842:00 +CPU= 15.489
  21:25:07 STEP 2843 H=2843:00 +CPU= 15.347
  21:25:33 STEP 2844 H=2844:00 +CPU= 26.224
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002844
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002844
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002844
  21:25:48 STEP 2845 H=2845:00 +CPU= 15.192
  21:26:04 STEP 2846 H=2846:00 +CPU= 15.589
  21:26:28 STEP 2847 H=2847:00 +CPU= 24.040
  21:26:43 STEP 2848 H=2848:00 +CPU= 15.496
  21:26:59 STEP 2849 H=2849:00 +CPU= 15.278
  21:27:23 STEP 2850 H=2850:00 +CPU= 24.405
  21:27:38 STEP 2851 H=2851:00 +CPU= 15.189
  21:27:54 STEP 2852 H=2852:00 +CPU= 15.599
  21:28:18 STEP 2853 H=2853:00 +CPU= 24.096
  21:28:33 STEP 2854 H=2854:00 +CPU= 15.494
  21:28:49 STEP 2855 H=2855:00 +CPU= 15.592
  21:29:17 STEP 2856 H=2856:00 +CPU= 27.787
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002856
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002856
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002856
  21:29:32 STEP 2857 H=2857:00 +CPU= 15.151
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002856
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_118.zip
Uploading the intermediate file: upload_file_118.zip
  21:29:48 STEP 2858 H=2858:00 +CPU= 15.634
  21:30:12 STEP 2859 H=2859:00 +CPU= 24.068
  21:30:27 STEP 2860 H=2860:00 +CPU= 15.522
  21:30:43 STEP 2861 H=2861:00 +CPU= 15.251
  21:31:07 STEP 2862 H=2862:00 +CPU= 24.441
  21:31:22 STEP 2863 H=2863:00 +CPU= 15.134
  21:31:38 STEP 2864 H=2864:00 +CPU= 15.662
  21:32:02 STEP 2865 H=2865:00 +CPU= 24.099
  21:32:18 STEP 2866 H=2866:00 +CPU= 15.477
  21:32:33 STEP 2867 H=2867:00 +CPU= 15.268
  21:32:59 STEP 2868 H=2868:00 +CPU= 26.166
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002868
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002868
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002868
  21:33:14 STEP 2869 H=2869:00 +CPU= 15.212
  21:33:30 STEP 2870 H=2870:00 +CPU= 15.662
  21:33:54 STEP 2871 H=2871:00 +CPU= 24.049
  21:34:10 STEP 2872 H=2872:00 +CPU= 15.522
  21:34:25 STEP 2873 H=2873:00 +CPU= 15.237
  21:34:49 STEP 2874 H=2874:00 +CPU= 24.454
  21:35:05 STEP 2875 H=2875:00 +CPU= 15.103
  21:35:20 STEP 2876 H=2876:00 +CPU= 15.650
  21:35:44 STEP 2877 H=2877:00 +CPU= 24.060
  21:36:00 STEP 2878 H=2878:00 +CPU= 15.458
  21:36:15 STEP 2879 H=2879:00 +CPU= 15.543
  21:36:43 STEP 2880 H=2880:00 +CPU= 27.888
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002880
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002880
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002880
  21:36:58 STEP 2881 H=2881:00 +CPU= 15.072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002880
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_119.zip
Uploading the intermediate file: upload_file_119.zip
  21:37:14 STEP 2882 H=2882:00 +CPU= 15.639
Uploading trickle at timestep: 10371600
  21:37:38 STEP 2883 H=2883:00 +CPU= 24.072
  21:37:54 STEP 2884 H=2884:00 +CPU= 15.547
  21:38:09 STEP 2885 H=2885:00 +CPU= 15.225
  21:38:34 STEP 2886 H=2886:00 +CPU= 24.533
  21:38:49 STEP 2887 H=2887:00 +CPU= 15.099
  21:39:04 STEP 2888 H=2888:00 +CPU= 15.622
  21:39:29 STEP 2889 H=2889:00 +CPU= 24.153
  21:39:44 STEP 2890 H=2890:00 +CPU= 15.587
  21:40:00 STEP 2891 H=2891:00 +CPU= 15.325
  21:40:26 STEP 2892 H=2892:00 +CPU= 26.129
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002892
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002892
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002892
  21:40:41 STEP 2893 H=2893:00 +CPU= 15.098
  21:40:57 STEP 2894 H=2894:00 +CPU= 15.611
  21:41:21 STEP 2895 H=2895:00 +CPU= 23.954
  21:41:36 STEP 2896 H=2896:00 +CPU= 15.439
  21:41:51 STEP 2897 H=2897:00 +CPU= 15.103
  21:42:15 STEP 2898 H=2898:00 +CPU= 24.363
  21:42:31 STEP 2899 H=2899:00 +CPU= 15.087
  21:42:46 STEP 2900 H=2900:00 +CPU= 15.567
  21:43:10 STEP 2901 H=2901:00 +CPU= 23.964
  21:43:26 STEP 2902 H=2902:00 +CPU= 15.459
  21:43:41 STEP 2903 H=2903:00 +CPU= 15.649
  21:44:09 STEP 2904 H=2904:00 +CPU= 27.811
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002904
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002904
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002904
  21:44:24 STEP 2905 H=2905:00 +CPU= 15.004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002904
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_120.zip
Uploading the intermediate file: upload_file_120.zip
  21:44:40 STEP 2906 H=2906:00 +CPU= 15.678
  21:45:04 STEP 2907 H=2907:00 +CPU= 24.068
  21:45:20 STEP 2908 H=2908:00 +CPU= 15.521
  21:45:35 STEP 2909 H=2909:00 +CPU= 15.244
  21:45:59 STEP 2910 H=2910:00 +CPU= 24.454
  21:46:15 STEP 2911 H=2911:00 +CPU= 15.247
  21:46:30 STEP 2912 H=2912:00 +CPU= 15.743
  21:46:55 STEP 2913 H=2913:00 +CPU= 24.092
  21:47:10 STEP 2914 H=2914:00 +CPU= 15.537
  21:47:25 STEP 2915 H=2915:00 +CPU= 15.187
  21:47:52 STEP 2916 H=2916:00 +CPU= 26.209
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002916
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002916
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002916
  21:48:07 STEP 2917 H=2917:00 +CPU= 15.109
  21:48:22 STEP 2918 H=2918:00 +CPU= 15.572
  21:48:46 STEP 2919 H=2919:00 +CPU= 23.954
  21:49:02 STEP 2920 H=2920:00 +CPU= 15.507
  21:49:17 STEP 2921 H=2921:00 +CPU= 15.185
  21:49:41 STEP 2922 H=2922:00 +CPU= 24.361
  21:49:57 STEP 2923 H=2923:00 +CPU= 15.094
  21:50:12 STEP 2924 H=2924:00 +CPU= 15.617
  21:50:36 STEP 2925 H=2925:00 +CPU= 23.927
  21:50:52 STEP 2926 H=2926:00 +CPU= 15.480
  21:51:07 STEP 2927 H=2927:00 +CPU= 15.627
  21:51:35 STEP 2928 H=2928:00 +CPU= 27.787
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002928
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002928
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002928
  21:51:50 STEP 2929 H=2929:00 +CPU= 15.028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002928
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_121.zip
Uploading the intermediate file: upload_file_121.zip
  21:52:06 STEP 2930 H=2930:00 +CPU= 15.627
  21:52:30 STEP 2931 H=2931:00 +CPU= 23.954
  21:52:45 STEP 2932 H=2932:00 +CPU= 15.473
  21:53:01 STEP 2933 H=2933:00 +CPU= 15.219
  21:53:25 STEP 2934 H=2934:00 +CPU= 24.405
  21:53:40 STEP 2935 H=2935:00 +CPU= 15.175
  21:53:56 STEP 2936 H=2936:00 +CPU= 15.638
  21:54:20 STEP 2937 H=2937:00 +CPU= 24.031
  21:54:35 STEP 2938 H=2938:00 +CPU= 15.481
  21:54:51 STEP 2939 H=2939:00 +CPU= 15.200
  21:55:17 STEP 2940 H=2940:00 +CPU= 26.171
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002940
  21:55:32 STEP 2941 H=2941:00 +CPU= 15.103
  21:55:48 STEP 2942 H=2942:00 +CPU= 15.552
  21:56:12 STEP 2943 H=2943:00 +CPU= 23.946
  21:56:27 STEP 2944 H=2944:00 +CPU= 15.521
  21:56:42 STEP 2945 H=2945:00 +CPU= 15.271
  21:57:07 STEP 2946 H=2946:00 +CPU= 24.427
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2521MB (heap), 4424MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x8102d0 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3843] 2604 MB (maxheap), 4424 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :         TRANSDIRH 
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :          TRANSDIR_MDL 
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :           DIR_TRANS 
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :            >TRS-FTDIR_CTL   - DIRECT ( 106) 
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215721:1678049841:9760.484] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :             >OMP-FTDIR_CTL            (1640) 
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215722:1678049842:9761.484] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:egonmint:1:1:2438:2438] [20230305:215722:1678049842:9761.484] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8440 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE89FB  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000810F01  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC8950  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000AD948E  fourier_out_mod_m          55  fourier_out_mod.F90
oifs_43r3_model.e  0000000000AD8890  ftdir_ctl_mod_mp_         174  ftdir_ctl_mod.F90
oifs_43r3_model.e  0000000000ACDB28  dir_trans_ctl_mod         171  dir_trans_ctl_mod.F90
oifs_43r3_model.e  0000000000859311  dir_trans_                488  dir_trans.F90
oifs_43r3_model.e  00000000013B3A81  transdir_mdl_             148  transdir_mdl.F90
oifs_43r3_model.e  00000000013B1EE0  transdirh_                 56  transdirh.F90
oifs_43r3_model.e  00000000004D6323  stepo_                    375  stepo.F90
oifs_43r3_model.e  00000000004339DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  000000000041D979  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  000000000041C8F2  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000407BD8  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040734D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9C50  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 00x6
(argv4) batchid: 993
(argv5) wuid: 12214632
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_57896969ac832d2b8f5783d1492679f3 to: /var/lib/boinc-client/slots/2/oifs_43r3_00x6_2019110100_123_993_12214632.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_00x6_2019110100_123_993_12214632.zip
ic_ancil_file: ic_ancil_12214632
ifsdata_file: ifsdata_12214632
climate_data_file: clim_data_12214632
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_59db7050553ba619e2cc1d7d3f9bf788 to: /var/lib/boinc-client/slots/2/ic_ancil_12214632.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12214632.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f4135d94bd4ac95cff0963561f2c50ae to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12214632.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12214632.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12214632.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12214632.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12214632.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12214632.xml
-- Model is restarting --
Adjusting last_iter, 2947, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12214632.xml
last_cpu_time: 65463
upload_file_number: 122
last_iter: 2929
last_upload: 10544400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705
The child process has been launched with process id: 2421
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2421:2421] [20230306:185715:1678125435:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:57:48 STEP 2928 H=2928:00 +CPU= 27.517
  18:58:03 STEP 2929 H=2929:00 +CPU= 15.294
  18:58:19 STEP 2930 H=2930:00 +CPU= 15.556
  18:58:43 STEP 2931 H=2931:00 +CPU= 23.985
  18:58:58 STEP 2932 H=2932:00 +CPU= 15.700
  18:59:14 STEP 2933 H=2933:00 +CPU= 15.268
  18:59:38 STEP 2934 H=2934:00 +CPU= 24.485
  18:59:54 STEP 2935 H=2935:00 +CPU= 15.409
  19:00:09 STEP 2936 H=2936:00 +CPU= 15.715
  19:00:33 STEP 2937 H=2937:00 +CPU= 23.938
  19:00:49 STEP 2938 H=2938:00 +CPU= 15.639
  19:01:04 STEP 2939 H=2939:00 +CPU= 15.310
  19:01:30 STEP 2940 H=2940:00 +CPU= 25.945
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002940
  19:01:46 STEP 2941 H=2941:00 +CPU= 15.218
  19:02:01 STEP 2942 H=2942:00 +CPU= 15.750
  19:02:26 STEP 2943 H=2943:00 +CPU= 24.173
  19:02:42 STEP 2944 H=2944:00 +CPU= 15.899
  19:02:57 STEP 2945 H=2945:00 +CPU= 15.454
  19:03:22 STEP 2946 H=2946:00 +CPU= 24.733
  19:03:37 STEP 2947 H=2947:00 +CPU= 15.466
  19:03:53 STEP 2948 H=2948:00 +CPU= 15.839
  19:04:18 STEP 2949 H=2949:00 +CPU= 24.265
  19:04:33 STEP 2950 H=2950:00 +CPU= 15.840
  19:04:49 STEP 2951 H=2951:00 +CPU= 15.892
  19:05:17 STEP 2952 H=2952:00 +CPU= 27.286
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 4.865283731608E-01 1.865286143845E-01 6.362598939121E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.260318722649E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.48989191816039E-02 -1.72994898449055E+00  1.89761310057317E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.19925486346338E-02 -1.00000000000000E+00  1.00000000000000E+00
  MAXGPFV : MAX. VALUE =    1.08700852859020     
  MAXGPFV : MAX. VALUE =   0.749709513352302     
 NSTEP =  2952 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGhnw4+002952 MODE=a
 IO-STREAM CLOSED - ICMGGhnw4+002952
 MPI-TASK:   1 -    346182074 BYTES IN      8 RECORDS TRANSFERRED IN    0.0004 SECONDS865455.2 Mbytes/s, TOTAL TIME=    0.0071( 5.6%)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302231500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302233000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302234500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200303000000  !!!!!!!!!!!! 
 19:05:17 STEP 2952 H=2952:00 +CPU= 27.286
 END CNT3
 NSTEP =  2952 CNT0   000000000  23.963  23.963  24.008   0.998               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           35/   804992476 WRITTEN          373/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     12    0       35       786125      373 ************     4.2     3.0     1.2     3.4     1.8
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1  481258.7  481258.7    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1  477169.6  477169.6     99.15      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   27    1688.9    1688.9      9.48      0.00
   9 SPCM     - SPECTRAL COMP.                      24     193.9     193.9      0.97      0.00
  10 SCAN2M - PHYSICS                               25    8532.2    8532.2     44.32      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  2     828.0     828.0      0.34      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  2      11.5      11.5      0.00      0.00
  13 SCAN2M - RADIATION CALC.                        9    8655.6    8655.6     16.19      0.00
  17 GRIDFPOS IN CNT4                                2      31.5      31.5      0.01      0.00
  27 RESTART FILES - WRITING                         1    1656.0    1656.0      0.34      0.00
  28 RESTART FILES - READING                         1    3757.1    3757.1      0.78      0.00
  29 SU4FPOS IN CNT4                                 2       0.0       0.0      0.00      0.00
  30 DYNFPOS IN CNT4                                 2    1513.5    1513.5      0.63      0.00
  37 CPGLAG   - SL COMPUTATIONS                     25    2317.8    2317.8     12.04      0.00
  38 WAM      - TOTAL COST OF WAVE MODEL            25    1910.4    1910.4      9.92      0.00
  39 SU0YOMB                                         1     217.6     217.6      0.05      0.00
  51 SCAN2M   - SL COMM. PART 1                     25       8.3       8.3      0.04      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               24     137.8     137.8      0.69      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               24      24.7      24.7      0.12      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  2       0.1       0.1      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM       88      85.2      85.2      1.56      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM        50      87.7      87.7      0.91      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM         50      37.9      37.9      0.39      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM        88      29.4      29.4      0.54      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            2      14.5      14.5      0.01      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION             88      11.8      11.8      0.22      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION             50      10.8      10.8      0.11      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION             88      14.0      14.0      0.26      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION             50      11.3      11.3      0.12      0.00
 400 GSTATS                                       6030       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                  5706       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME      481.3 CPU TIME     477.5 VECTOR TIME      477.5


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY    22271.3
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhnw4+002952
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhnw4+002952
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhnw4+002952
Adding to the zip: /var/lib/boinc-client/slots/2/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/2/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMUAhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMSHhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214632/ICMGGhnw4+002952
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00x6_2019110100_123_993_12214632_0_r1232620705_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
19:08:39 (2369): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:55:25 1537224 22316006 oifs_43r3_00x6_2019110100_123_993_12214632_0 10,623,600 65,940 0.0062
09 May 2023 16:55:25 1537224 22316006 oifs_43r3_00x6_2019110100_123_993_12214632_0 10,371,600 64,251 0.0062
09 May 2023 16:55:23 1537224 22316006 oifs_43r3_00x6_2019110100_123_993_12214632_0 9,507,600 59,803 0.0063
09 May 2023 16:55:20 1537224 22316006 oifs_43r3_00x6_2019110100_123_993_12214632_0 8,121,600 52,456 0.0065
09 May 2023 16:55:15 1537224 22316006 oifs_43r3_00x6_2019110100_123_993_12214632_0 6,998,400 45,833 0.0065
09 May 2023 16:55:15 1537224 22316006 oifs_43r3_00x6_2019110100_123_993_12214632_0 6,134,400 41,020 0.0067
09 May 2023 16:54:52 1537224 22316006 oifs_43r3_00x6_2019110100_123_993_12214632_0 4,665,600 32,300 0.0069
09 May 2023 16:54:51 1537224 22316006 oifs_43r3_00x6_2019110100_123_993_12214632_0 3,801,600 27,543 0.0072
09 May 2023 16:54:35 1537224 22316006 oifs_43r3_00x6_2019110100_123_993_12214632_0 2,851,200 20,055 0.0070
09 May 2023 16:52:43 1537224 22316006 oifs_43r3_00x6_2019110100_123_993_12214632_0 864,000 4,742 0.0055


©2024 climateprediction.net