climateprediction.net home page
Task 22315904

Task 22315904

Name oifs_43r3_00ue_2019110100_123_993_12214532_0
Workunit 12214532
Created 24 Feb 2023, 15:20:43 UTC
Sent 24 Feb 2023, 15:36:36 UTC
Report deadline 25 Apr 2023, 15:36:36 UTC
Received 4 Mar 2023, 20:12:00 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537319
Run time 19 hours 27 min 13 sec
CPU time 19 hours 27 min 13 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 5.73 GFLOPS
Application version OpenIFS 43r3 v1.21
x86_64-pc-linux-gnu
Peak working set size 4,415.45 MB
Peak swap size 4,983.92 MB
Peak disk usage 1,606.01 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<stderr_txt>
6
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_104.zip
Uploading the intermediate file: upload_file_104.zip
  20:46:13 STEP 2521 H=2521:00 +CPU= 14.650
  20:46:28 STEP 2522 H=2522:00 +CPU= 15.028
  20:46:51 STEP 2523 H=2523:00 +CPU= 23.258
  20:47:06 STEP 2524 H=2524:00 +CPU= 15.120
  20:47:21 STEP 2525 H=2525:00 +CPU= 14.702
  20:47:44 STEP 2526 H=2526:00 +CPU= 23.312
  20:47:59 STEP 2527 H=2527:00 +CPU= 14.643
  20:48:14 STEP 2528 H=2528:00 +CPU= 15.141
  20:48:37 STEP 2529 H=2529:00 +CPU= 22.930
  20:48:52 STEP 2530 H=2530:00 +CPU= 15.068
  20:49:07 STEP 2531 H=2531:00 +CPU= 14.589
  20:49:32 STEP 2532 H=2532:00 +CPU= 24.977
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002532
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002532
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002532
  20:49:46 STEP 2533 H=2533:00 +CPU= 14.736
  20:50:02 STEP 2534 H=2534:00 +CPU= 15.211
  20:50:25 STEP 2535 H=2535:00 +CPU= 22.891
  20:50:40 STEP 2536 H=2536:00 +CPU= 15.075
  20:50:54 STEP 2537 H=2537:00 +CPU= 14.722
  20:51:18 STEP 2538 H=2538:00 +CPU= 23.406
  20:51:33 STEP 2539 H=2539:00 +CPU= 14.741
  20:51:48 STEP 2540 H=2540:00 +CPU= 15.191
  20:52:11 STEP 2541 H=2541:00 +CPU= 22.972
  20:52:26 STEP 2542 H=2542:00 +CPU= 15.090
  20:52:41 STEP 2543 H=2543:00 +CPU= 15.132
  20:53:08 STEP 2544 H=2544:00 +CPU= 26.716
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002544
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002544
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_105.zip
Uploading the intermediate file: upload_file_105.zip
  20:53:23 STEP 2545 H=2545:00 +CPU= 15.002
  20:53:38 STEP 2546 H=2546:00 +CPU= 15.077
  20:54:01 STEP 2547 H=2547:00 +CPU= 23.239
  20:54:16 STEP 2548 H=2548:00 +CPU= 15.055
  20:54:31 STEP 2549 H=2549:00 +CPU= 14.840
  20:54:54 STEP 2550 H=2550:00 +CPU= 23.354
  20:55:09 STEP 2551 H=2551:00 +CPU= 14.892
  20:55:24 STEP 2552 H=2552:00 +CPU= 15.094
  20:55:47 STEP 2553 H=2553:00 +CPU= 23.116
  20:56:03 STEP 2554 H=2554:00 +CPU= 15.123
  20:56:17 STEP 2555 H=2555:00 +CPU= 14.664
  20:56:42 STEP 2556 H=2556:00 +CPU= 25.024
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002556
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002556
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002556
  20:56:57 STEP 2557 H=2557:00 +CPU= 14.790
  20:57:12 STEP 2558 H=2558:00 +CPU= 15.152
  20:57:35 STEP 2559 H=2559:00 +CPU= 23.134
  20:57:51 STEP 2560 H=2560:00 +CPU= 15.070
  20:58:05 STEP 2561 H=2561:00 +CPU= 14.725
  20:58:29 STEP 2562 H=2562:00 +CPU= 23.359
  20:58:43 STEP 2563 H=2563:00 +CPU= 14.754
  20:58:58 STEP 2564 H=2564:00 +CPU= 15.082
  20:59:22 STEP 2565 H=2565:00 +CPU= 23.078
  20:59:37 STEP 2566 H=2566:00 +CPU= 15.043
  20:59:52 STEP 2567 H=2567:00 +CPU= 15.198
  21:00:19 STEP 2568 H=2568:00 +CPU= 26.814
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002568
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002568
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_106.zip
Uploading the intermediate file: upload_file_106.zip
  21:00:33 STEP 2569 H=2569:00 +CPU= 14.840
  21:00:49 STEP 2570 H=2570:00 +CPU= 15.172
  21:01:12 STEP 2571 H=2571:00 +CPU= 22.990
  21:01:27 STEP 2572 H=2572:00 +CPU= 15.089
  21:01:42 STEP 2573 H=2573:00 +CPU= 14.866
  21:02:05 STEP 2574 H=2574:00 +CPU= 23.452
  21:02:20 STEP 2575 H=2575:00 +CPU= 14.799
  21:02:35 STEP 2576 H=2576:00 +CPU= 15.178
  21:02:58 STEP 2577 H=2577:00 +CPU= 23.002
  21:03:13 STEP 2578 H=2578:00 +CPU= 15.105
  21:03:28 STEP 2579 H=2579:00 +CPU= 14.700
  21:03:53 STEP 2580 H=2580:00 +CPU= 25.090
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002580
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002580
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002580
  21:04:08 STEP 2581 H=2581:00 +CPU= 14.730
  21:04:23 STEP 2582 H=2582:00 +CPU= 15.143
  21:04:46 STEP 2583 H=2583:00 +CPU= 22.931
  21:05:01 STEP 2584 H=2584:00 +CPU= 15.060
  21:05:16 STEP 2585 H=2585:00 +CPU= 14.710
  21:05:39 STEP 2586 H=2586:00 +CPU= 23.383
  21:05:54 STEP 2587 H=2587:00 +CPU= 14.814
  21:06:09 STEP 2588 H=2588:00 +CPU= 15.023
  21:06:32 STEP 2589 H=2589:00 +CPU= 22.994
  21:06:47 STEP 2590 H=2590:00 +CPU= 15.060
  21:07:02 STEP 2591 H=2591:00 +CPU= 15.049
  21:07:29 STEP 2592 H=2592:00 +CPU= 26.733
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002592
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002592
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_107.zip
Uploading the intermediate file: upload_file_107.zip
  21:07:44 STEP 2593 H=2593:00 +CPU= 15.081
  21:07:59 STEP 2594 H=2594:00 +CPU= 15.123
  21:08:22 STEP 2595 H=2595:00 +CPU= 22.957
  21:08:37 STEP 2596 H=2596:00 +CPU= 15.131
  21:08:52 STEP 2597 H=2597:00 +CPU= 14.844
  21:09:16 STEP 2598 H=2598:00 +CPU= 23.478
  21:09:30 STEP 2599 H=2599:00 +CPU= 14.643
  21:09:45 STEP 2600 H=2600:00 +CPU= 15.092
  21:10:08 STEP 2601 H=2601:00 +CPU= 22.996
  21:10:24 STEP 2602 H=2602:00 +CPU= 15.204
  21:10:38 STEP 2603 H=2603:00 +CPU= 14.611
  21:11:03 STEP 2604 H=2604:00 +CPU= 25.086
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002604
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002604
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002604
  21:11:18 STEP 2605 H=2605:00 +CPU= 14.795
  21:11:33 STEP 2606 H=2606:00 +CPU= 15.168
  21:11:56 STEP 2607 H=2607:00 +CPU= 22.968
  21:12:11 STEP 2608 H=2608:00 +CPU= 15.015
  21:12:26 STEP 2609 H=2609:00 +CPU= 14.772
  21:12:49 STEP 2610 H=2610:00 +CPU= 23.377
  21:13:04 STEP 2611 H=2611:00 +CPU= 14.707
  21:13:19 STEP 2612 H=2612:00 +CPU= 15.066
  21:13:42 STEP 2613 H=2613:00 +CPU= 23.211
  21:13:58 STEP 2614 H=2614:00 +CPU= 15.036
  21:14:13 STEP 2615 H=2615:00 +CPU= 15.129
  21:14:39 STEP 2616 H=2616:00 +CPU= 26.779
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002616
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002616
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_108.zip
Uploading the intermediate file: upload_file_108.zip
  21:14:54 STEP 2617 H=2617:00 +CPU= 14.993
  21:15:10 STEP 2618 H=2618:00 +CPU= 15.193
  21:15:33 STEP 2619 H=2619:00 +CPU= 23.043
  21:15:48 STEP 2620 H=2620:00 +CPU= 15.275
  21:16:03 STEP 2621 H=2621:00 +CPU= 14.808
  21:16:26 STEP 2622 H=2622:00 +CPU= 23.533
  21:16:41 STEP 2623 H=2623:00 +CPU= 14.839
  21:16:56 STEP 2624 H=2624:00 +CPU= 15.057
  21:17:19 STEP 2625 H=2625:00 +CPU= 23.046
  21:17:35 STEP 2626 H=2626:00 +CPU= 15.109
  21:17:49 STEP 2627 H=2627:00 +CPU= 14.745
  21:18:14 STEP 2628 H=2628:00 +CPU= 25.087
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002628
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002628
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002628
  21:18:29 STEP 2629 H=2629:00 +CPU= 14.828
  21:18:44 STEP 2630 H=2630:00 +CPU= 15.189
  21:19:07 STEP 2631 H=2631:00 +CPU= 23.011
  21:19:23 STEP 2632 H=2632:00 +CPU= 15.148
  21:19:37 STEP 2633 H=2633:00 +CPU= 14.744
  21:20:01 STEP 2634 H=2634:00 +CPU= 23.352
  21:20:16 STEP 2635 H=2635:00 +CPU= 14.764
  21:20:31 STEP 2636 H=2636:00 +CPU= 15.119
  21:20:54 STEP 2637 H=2637:00 +CPU= 22.847
  21:21:09 STEP 2638 H=2638:00 +CPU= 15.012
  21:21:24 STEP 2639 H=2639:00 +CPU= 15.064
  21:21:50 STEP 2640 H=2640:00 +CPU= 26.645
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002640
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002640
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_109.zip
Uploading the intermediate file: upload_file_109.zip
  21:22:05 STEP 2641 H=2641:00 +CPU= 14.861
  21:22:20 STEP 2642 H=2642:00 +CPU= 15.157
  21:22:43 STEP 2643 H=2643:00 +CPU= 22.956
  21:22:58 STEP 2644 H=2644:00 +CPU= 15.155
  21:23:13 STEP 2645 H=2645:00 +CPU= 14.821
  21:23:37 STEP 2646 H=2646:00 +CPU= 23.431
  21:23:52 STEP 2647 H=2647:00 +CPU= 14.810
  21:24:07 STEP 2648 H=2648:00 +CPU= 15.068
  21:24:30 STEP 2649 H=2649:00 +CPU= 23.126
  21:24:45 STEP 2650 H=2650:00 +CPU= 15.099
  21:25:00 STEP 2651 H=2651:00 +CPU= 14.604
  21:25:25 STEP 2652 H=2652:00 +CPU= 24.984
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002652
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002652
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002652
  21:25:39 STEP 2653 H=2653:00 +CPU= 14.672
  21:25:54 STEP 2654 H=2654:00 +CPU= 15.057
  21:26:17 STEP 2655 H=2655:00 +CPU= 23.150
  21:26:33 STEP 2656 H=2656:00 +CPU= 15.089
  21:26:47 STEP 2657 H=2657:00 +CPU= 14.779
  21:27:11 STEP 2658 H=2658:00 +CPU= 23.362
  21:27:25 STEP 2659 H=2659:00 +CPU= 14.769
  21:27:41 STEP 2660 H=2660:00 +CPU= 15.061
  21:28:04 STEP 2661 H=2661:00 +CPU= 23.066
  21:28:19 STEP 2662 H=2662:00 +CPU= 15.074
  21:28:34 STEP 2663 H=2663:00 +CPU= 15.057
  21:29:00 STEP 2664 H=2664:00 +CPU= 26.709
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002664
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002664
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_110.zip
Uploading the intermediate file: upload_file_110.zip
  21:29:15 STEP 2665 H=2665:00 +CPU= 14.826
Uploading trickle at timestep: 9590400
  21:29:30 STEP 2666 H=2666:00 +CPU= 15.122
  21:29:53 STEP 2667 H=2667:00 +CPU= 22.967
  21:30:09 STEP 2668 H=2668:00 +CPU= 15.110
  21:30:23 STEP 2669 H=2669:00 +CPU= 14.750
  21:30:47 STEP 2670 H=2670:00 +CPU= 23.524
  21:31:02 STEP 2671 H=2671:00 +CPU= 14.798
  21:31:17 STEP 2672 H=2672:00 +CPU= 15.293
  21:31:40 STEP 2673 H=2673:00 +CPU= 23.070
  21:31:55 STEP 2674 H=2674:00 +CPU= 15.246
  21:32:10 STEP 2675 H=2675:00 +CPU= 14.720
  21:32:35 STEP 2676 H=2676:00 +CPU= 25.140
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002676
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002676
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002676
  21:32:50 STEP 2677 H=2677:00 +CPU= 14.663
  21:33:05 STEP 2678 H=2678:00 +CPU= 15.002
  21:33:28 STEP 2679 H=2679:00 +CPU= 22.968
  21:33:43 STEP 2680 H=2680:00 +CPU= 15.148
  21:33:58 STEP 2681 H=2681:00 +CPU= 14.769
  21:34:21 STEP 2682 H=2682:00 +CPU= 23.624
  21:34:36 STEP 2683 H=2683:00 +CPU= 14.782
  21:34:51 STEP 2684 H=2684:00 +CPU= 15.113
  21:35:14 STEP 2685 H=2685:00 +CPU= 22.942
  21:35:29 STEP 2686 H=2686:00 +CPU= 15.037
  21:35:45 STEP 2687 H=2687:00 +CPU= 15.156
  21:36:11 STEP 2688 H=2688:00 +CPU= 26.798
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002688
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002688
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_111.zip
Uploading the intermediate file: upload_file_111.zip
  21:36:26 STEP 2689 H=2689:00 +CPU= 14.748
  21:36:41 STEP 2690 H=2690:00 +CPU= 15.218
  21:37:04 STEP 2691 H=2691:00 +CPU= 22.918
  21:37:20 STEP 2692 H=2692:00 +CPU= 15.370
  21:37:35 STEP 2693 H=2693:00 +CPU= 15.009
  21:37:59 STEP 2694 H=2694:00 +CPU= 24.360
  21:38:14 STEP 2695 H=2695:00 +CPU= 15.357
  21:38:30 STEP 2696 H=2696:00 +CPU= 15.727
  21:38:54 STEP 2697 H=2697:00 +CPU= 23.720
  21:39:10 STEP 2698 H=2698:00 +CPU= 15.726
  21:39:25 STEP 2699 H=2699:00 +CPU= 15.112
  21:39:51 STEP 2700 H=2700:00 +CPU= 26.089
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002700
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002700
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002700
  21:40:06 STEP 2701 H=2701:00 +CPU= 15.320
  21:40:22 STEP 2702 H=2702:00 +CPU= 15.702
  21:40:46 STEP 2703 H=2703:00 +CPU= 23.846
  21:41:01 STEP 2704 H=2704:00 +CPU= 15.556
  21:41:17 STEP 2705 H=2705:00 +CPU= 15.294
  21:41:41 STEP 2706 H=2706:00 +CPU= 24.148
  21:41:56 STEP 2707 H=2707:00 +CPU= 15.439
  21:42:12 STEP 2708 H=2708:00 +CPU= 15.739
  21:42:36 STEP 2709 H=2709:00 +CPU= 23.692
  21:42:51 STEP 2710 H=2710:00 +CPU= 15.557
  21:43:07 STEP 2711 H=2711:00 +CPU= 15.637
  21:43:34 STEP 2712 H=2712:00 +CPU= 27.108
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002712
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002712
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_112.zip
Uploading the intermediate file: upload_file_112.zip
  21:43:49 STEP 2713 H=2713:00 +CPU= 14.948
  21:44:04 STEP 2714 H=2714:00 +CPU= 15.086
  21:44:27 STEP 2715 H=2715:00 +CPU= 23.186
  21:44:42 STEP 2716 H=2716:00 +CPU= 15.100
  21:44:57 STEP 2717 H=2717:00 +CPU= 14.862
  21:45:21 STEP 2718 H=2718:00 +CPU= 23.299
  21:45:36 STEP 2719 H=2719:00 +CPU= 14.953
  21:45:51 STEP 2720 H=2720:00 +CPU= 15.272
  21:46:14 STEP 2721 H=2721:00 +CPU= 23.169
  21:46:29 STEP 2722 H=2722:00 +CPU= 15.149
  21:46:44 STEP 2723 H=2723:00 +CPU= 14.778
  21:47:09 STEP 2724 H=2724:00 +CPU= 25.233
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002724
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002724
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002724
  21:47:24 STEP 2725 H=2725:00 +CPU= 14.689
  21:47:39 STEP 2726 H=2726:00 +CPU= 15.190
  21:48:02 STEP 2727 H=2727:00 +CPU= 23.029
  21:48:17 STEP 2728 H=2728:00 +CPU= 15.109
  21:48:32 STEP 2729 H=2729:00 +CPU= 14.896
  21:48:56 STEP 2730 H=2730:00 +CPU= 23.457
  21:49:11 STEP 2731 H=2731:00 +CPU= 14.970
  21:49:26 STEP 2732 H=2732:00 +CPU= 15.213
  21:49:49 STEP 2733 H=2733:00 +CPU= 23.062
  21:50:04 STEP 2734 H=2734:00 +CPU= 15.107
  21:50:19 STEP 2735 H=2735:00 +CPU= 15.169
  21:50:46 STEP 2736 H=2736:00 +CPU= 26.835
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002736
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002736
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_113.zip
Uploading the intermediate file: upload_file_113.zip
  21:51:01 STEP 2737 H=2737:00 +CPU= 14.695
  21:51:16 STEP 2738 H=2738:00 +CPU= 15.150
  21:51:39 STEP 2739 H=2739:00 +CPU= 23.008
  21:51:54 STEP 2740 H=2740:00 +CPU= 15.077
  21:52:09 STEP 2741 H=2741:00 +CPU= 14.824
  21:52:33 STEP 2742 H=2742:00 +CPU= 23.714
  21:52:48 STEP 2743 H=2743:00 +CPU= 14.866
  21:53:03 STEP 2744 H=2744:00 +CPU= 15.115
  21:53:26 STEP 2745 H=2745:00 +CPU= 23.211
  21:53:41 STEP 2746 H=2746:00 +CPU= 15.130
  21:53:56 STEP 2747 H=2747:00 +CPU= 14.849
  21:54:21 STEP 2748 H=2748:00 +CPU= 24.993
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002748
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002748
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002748
  21:54:36 STEP 2749 H=2749:00 +CPU= 14.768
  21:54:51 STEP 2750 H=2750:00 +CPU= 15.190
  21:55:14 STEP 2751 H=2751:00 +CPU= 22.997
  21:55:29 STEP 2752 H=2752:00 +CPU= 15.107
  21:55:44 STEP 2753 H=2753:00 +CPU= 14.775
  21:56:07 STEP 2754 H=2754:00 +CPU= 23.319
  21:56:22 STEP 2755 H=2755:00 +CPU= 14.716
  21:56:37 STEP 2756 H=2756:00 +CPU= 15.185
  21:57:00 STEP 2757 H=2757:00 +CPU= 23.151
  21:57:15 STEP 2758 H=2758:00 +CPU= 15.070
  21:57:30 STEP 2759 H=2759:00 +CPU= 15.222
  21:57:57 STEP 2760 H=2760:00 +CPU= 26.553
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002760
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002760
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_114.zip
Uploading the intermediate file: upload_file_114.zip
  21:58:12 STEP 2761 H=2761:00 +CPU= 14.658
  21:58:27 STEP 2762 H=2762:00 +CPU= 15.124
  21:58:50 STEP 2763 H=2763:00 +CPU= 23.021
  21:59:05 STEP 2764 H=2764:00 +CPU= 15.121
  21:59:20 STEP 2765 H=2765:00 +CPU= 14.863
  21:59:43 STEP 2766 H=2766:00 +CPU= 23.439
  21:59:58 STEP 2767 H=2767:00 +CPU= 14.810
  22:00:13 STEP 2768 H=2768:00 +CPU= 15.233
  22:00:37 STEP 2769 H=2769:00 +CPU= 23.044
  22:00:52 STEP 2770 H=2770:00 +CPU= 15.239
  22:01:07 STEP 2771 H=2771:00 +CPU= 14.823
  22:01:32 STEP 2772 H=2772:00 +CPU= 25.109
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002772
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002772
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002772
  22:01:47 STEP 2773 H=2773:00 +CPU= 14.834
  22:02:02 STEP 2774 H=2774:00 +CPU= 15.144
  22:02:25 STEP 2775 H=2775:00 +CPU= 22.997
  22:02:40 STEP 2776 H=2776:00 +CPU= 15.163
  22:02:55 STEP 2777 H=2777:00 +CPU= 14.787
  22:03:18 STEP 2778 H=2778:00 +CPU= 23.363
  22:03:33 STEP 2779 H=2779:00 +CPU= 14.722
  22:03:48 STEP 2780 H=2780:00 +CPU= 15.224
  22:04:11 STEP 2781 H=2781:00 +CPU= 22.933
  22:04:26 STEP 2782 H=2782:00 +CPU= 15.216
  22:04:42 STEP 2783 H=2783:00 +CPU= 15.377
  22:05:08 STEP 2784 H=2784:00 +CPU= 26.541
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002784
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002784
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_115.zip
Uploading the intermediate file: upload_file_115.zip
  22:05:23 STEP 2785 H=2785:00 +CPU= 14.735
  22:05:38 STEP 2786 H=2786:00 +CPU= 15.044
  22:06:01 STEP 2787 H=2787:00 +CPU= 22.895
  22:06:16 STEP 2788 H=2788:00 +CPU= 15.130
  22:06:31 STEP 2789 H=2789:00 +CPU= 14.758
  22:06:54 STEP 2790 H=2790:00 +CPU= 23.373
  22:07:09 STEP 2791 H=2791:00 +CPU= 14.860
  22:07:24 STEP 2792 H=2792:00 +CPU= 15.171
  22:07:47 STEP 2793 H=2793:00 +CPU= 22.902
  22:08:02 STEP 2794 H=2794:00 +CPU= 15.079
  22:08:17 STEP 2795 H=2795:00 +CPU= 14.718
  22:08:42 STEP 2796 H=2796:00 +CPU= 24.982
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002796
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002796
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002796
  22:08:57 STEP 2797 H=2797:00 +CPU= 14.751
  22:09:12 STEP 2798 H=2798:00 +CPU= 15.238
  22:09:35 STEP 2799 H=2799:00 +CPU= 22.964
  22:09:50 STEP 2800 H=2800:00 +CPU= 15.192
  22:10:05 STEP 2801 H=2801:00 +CPU= 14.829
  22:10:29 STEP 2802 H=2802:00 +CPU= 23.413
  22:10:43 STEP 2803 H=2803:00 +CPU= 14.773
  22:10:58 STEP 2804 H=2804:00 +CPU= 15.092
  22:11:22 STEP 2805 H=2805:00 +CPU= 23.066
  22:11:37 STEP 2806 H=2806:00 +CPU= 15.137
  22:11:52 STEP 2807 H=2807:00 +CPU= 15.166
  22:12:18 STEP 2808 H=2808:00 +CPU= 26.609
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002808
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002808
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_116.zip
Uploading the intermediate file: upload_file_116.zip
  22:12:33 STEP 2809 H=2809:00 +CPU= 14.730
  22:12:48 STEP 2810 H=2810:00 +CPU= 15.171
  22:13:11 STEP 2811 H=2811:00 +CPU= 22.956
  22:13:26 STEP 2812 H=2812:00 +CPU= 15.124
  22:13:41 STEP 2813 H=2813:00 +CPU= 14.855
  22:14:05 STEP 2814 H=2814:00 +CPU= 23.376
  22:14:20 STEP 2815 H=2815:00 +CPU= 14.752
  22:14:35 STEP 2816 H=2816:00 +CPU= 15.108
  22:14:58 STEP 2817 H=2817:00 +CPU= 22.923
  22:15:13 STEP 2818 H=2818:00 +CPU= 15.092
  22:15:27 STEP 2819 H=2819:00 +CPU= 14.721
  22:15:52 STEP 2820 H=2820:00 +CPU= 24.964
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002820
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002820
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002820
  22:16:07 STEP 2821 H=2821:00 +CPU= 14.789
  22:16:22 STEP 2822 H=2822:00 +CPU= 15.130
  22:16:45 STEP 2823 H=2823:00 +CPU= 22.873
  22:17:00 STEP 2824 H=2824:00 +CPU= 15.049
  22:17:15 STEP 2825 H=2825:00 +CPU= 14.777
  22:17:39 STEP 2826 H=2826:00 +CPU= 23.387
  22:17:53 STEP 2827 H=2827:00 +CPU= 14.879
  22:18:09 STEP 2828 H=2828:00 +CPU= 15.248
  22:18:32 STEP 2829 H=2829:00 +CPU= 23.026
  22:18:47 STEP 2830 H=2830:00 +CPU= 15.219
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           22  MAX ETADOT VERTICAL VEL.=   5.349421156844126E-007
  LON  =    162.500000000000       degrees
  LAT  =    35.3269057618723       degrees
  22:19:02 STEP 2831 H=2831:00 +CPU= 15.287
  22:19:29 STEP 2832 H=2832:00 +CPU= 26.757
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002832
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002832
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_117.zip
Uploading the intermediate file: upload_file_117.zip
  22:19:44 STEP 2833 H=2833:00 +CPU= 14.716
  22:19:59 STEP 2834 H=2834:00 +CPU= 15.122
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           25  MAX ETADOT VERTICAL VEL.=   2.011562134721182E-007
  LON  =    166.250000000000       degrees
  LAT  =    37.5698795904715       degrees
  22:20:22 STEP 2835 H=2835:00 +CPU= 23.040
  22:20:37 STEP 2836 H=2836:00 +CPU= 15.118
  22:20:52 STEP 2837 H=2837:00 +CPU= 14.803
  22:21:16 STEP 2838 H=2838:00 +CPU= 23.815
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.344] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4746MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.345] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.345] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x8102d0 (old at (nil))
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.345] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.345] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.345] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3843] 4746 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.345] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.346] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     2        (1005) 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :               LAPINEB 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :                LARCINB 
[EC_DRHOOK:egonmsi:1:1:1978:1978] [20230303:222120:1677878480:11823.347] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :                 LAITLI 
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 00ue
(argv4) batchid: 993
(argv5) wuid: 12214532
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/25
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/25/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/25/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_34efd4227a5100eec160cfdf0f30aa58 to: /var/lib/boinc-client/slots/25/oifs_43r3_00ue_2019110100_123_993_12214532.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/25/oifs_43r3_00ue_2019110100_123_993_12214532.zip
ic_ancil_file: ic_ancil_12214532
ifsdata_file: ifsdata_12214532
climate_data_file: clim_data_12214532
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_51746a20471766180a7c6aeb4481a0a8 to: /var/lib/boinc-client/slots/25/ic_ancil_12214532.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/25/ic_ancil_12214532.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ace498f51360ba3ea0e843b9cb440db0 to: /var/lib/boinc-client/slots/25/ifsdata/ifsdata_12214532.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/25/ifsdata/ifsdata_12214532.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/25/159l_2/clim_data_12214532.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/25/159l_2/clim_data_12214532.zip
Checking for progress XML file: /var/lib/boinc-client/slots/25/progress_file_12214532.xml
Opened progress file ok : /var/lib/boinc-client/slots/25/progress_file_12214532.xml
-- Model is restarting --
Adjusting last_iter, 2838, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/25/progress_file_12214532.xml
last_cpu_time: 67833
upload_file_number: 118
last_iter: 2833
last_upload: 10195200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958
The child process has been launched with process id: 1996
Executing the command: /var/lib/boinc-client/slots/25/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:egonmsi:1:1:1996:1996] [20230304:201101:1677957061:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:11:31 STEP 2832 H=2832:00 +CPU= 26.833
  20:11:46 STEP 2833 H=2833:00 +CPU= 15.121
  20:12:02 STEP 2834 H=2834:00 +CPU= 15.486
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           25  MAX ETADOT VERTICAL VEL.=   2.011562134721182E-007
  LON  =    166.250000000000       degrees
  LAT  =    37.5698795904715       degrees
  20:12:25 STEP 2835 H=2835:00 +CPU= 23.352
  20:12:41 STEP 2836 H=2836:00 +CPU= 15.381
  20:12:56 STEP 2837 H=2837:00 +CPU= 14.916
  20:13:19 STEP 2838 H=2838:00 +CPU= 23.732
  20:13:34 STEP 2839 H=2839:00 +CPU= 14.941
  20:13:50 STEP 2840 H=2840:00 +CPU= 15.355
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           29  MAX ETADOT VERTICAL VEL.=   2.159717017385827E-007
  LON  =    171.250000000000       degrees
  LAT  =    40.9343383242788       degrees
  20:14:13 STEP 2841 H=2841:00 +CPU= 23.381
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           14  MAX ETADOT VERTICAL VEL.=   1.854348436308624E-007
  LON  =    171.562500000000       degrees
  LAT  =    42.0558239509352       degrees
  20:14:28 STEP 2842 H=2842:00 +CPU= 15.295
  20:14:43 STEP 2843 H=2843:00 +CPU= 14.968
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           15  MAX ETADOT VERTICAL VEL.=   1.200183594847581E-007
  LON  =    172.968750000000       degrees
  LAT  =    43.1773092288349       degrees
  20:15:09 STEP 2844 H=2844:00 +CPU= 25.283
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002844
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002844
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002844
  20:15:24 STEP 2845 H=2845:00 +CPU= 14.956
  20:15:39 STEP 2846 H=2846:00 +CPU= 15.424
  20:16:02 STEP 2847 H=2847:00 +CPU= 23.239
  20:16:18 STEP 2848 H=2848:00 +CPU= 15.269
  20:16:32 STEP 2849 H=2849:00 +CPU= 14.847
  20:16:56 STEP 2850 H=2850:00 +CPU= 23.599
  20:17:11 STEP 2851 H=2851:00 +CPU= 14.877
  20:17:26 STEP 2852 H=2852:00 +CPU= 15.151
  20:17:49 STEP 2853 H=2853:00 +CPU= 23.261
  20:18:05 STEP 2854 H=2854:00 +CPU= 15.310
  20:18:20 STEP 2855 H=2855:00 +CPU= 15.624
  20:18:47 STEP 2856 H=2856:00 +CPU= 26.822
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002856
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002856
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_118.zip
Uploading the intermediate file: upload_file_118.zip
  20:19:02 STEP 2857 H=2857:00 +CPU= 15.221
  20:19:18 STEP 2858 H=2858:00 +CPU= 15.366
  20:19:41 STEP 2859 H=2859:00 +CPU= 23.204
  20:19:56 STEP 2860 H=2860:00 +CPU= 15.271
  20:20:11 STEP 2861 H=2861:00 +CPU= 15.028
  20:20:35 STEP 2862 H=2862:00 +CPU= 23.540
  20:20:50 STEP 2863 H=2863:00 +CPU= 14.965
  20:21:05 STEP 2864 H=2864:00 +CPU= 15.439
  20:21:29 STEP 2865 H=2865:00 +CPU= 23.374
  20:21:44 STEP 2866 H=2866:00 +CPU= 15.417
  20:21:59 STEP 2867 H=2867:00 +CPU= 14.932
  20:22:24 STEP 2868 H=2868:00 +CPU= 25.237
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002868
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002868
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002868
  20:22:39 STEP 2869 H=2869:00 +CPU= 14.948
  20:22:55 STEP 2870 H=2870:00 +CPU= 15.356
  20:23:18 STEP 2871 H=2871:00 +CPU= 23.122
  20:23:33 STEP 2872 H=2872:00 +CPU= 15.221
  20:23:48 STEP 2873 H=2873:00 +CPU= 14.899
  20:24:12 STEP 2874 H=2874:00 +CPU= 23.550
  20:24:27 STEP 2875 H=2875:00 +CPU= 14.903
  20:24:42 STEP 2876 H=2876:00 +CPU= 15.225
  20:25:05 STEP 2877 H=2877:00 +CPU= 23.026
  20:25:20 STEP 2878 H=2878:00 +CPU= 15.187
  20:25:35 STEP 2879 H=2879:00 +CPU= 15.264
  20:26:03 STEP 2880 H=2880:00 +CPU= 27.238
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002880
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002880
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_119.zip
Uploading the intermediate file: upload_file_119.zip
  20:26:18 STEP 2881 H=2881:00 +CPU= 14.878
  20:26:33 STEP 2882 H=2882:00 +CPU= 15.202
  20:26:56 STEP 2883 H=2883:00 +CPU= 23.065
  20:27:11 STEP 2884 H=2884:00 +CPU= 15.131
  20:27:26 STEP 2885 H=2885:00 +CPU= 14.770
  20:27:49 STEP 2886 H=2886:00 +CPU= 23.644
  20:28:04 STEP 2887 H=2887:00 +CPU= 14.825
  20:28:19 STEP 2888 H=2888:00 +CPU= 15.215
  20:28:43 STEP 2889 H=2889:00 +CPU= 23.223
  20:28:58 STEP 2890 H=2890:00 +CPU= 15.258
  20:29:13 STEP 2891 H=2891:00 +CPU= 14.870
  20:29:38 STEP 2892 H=2892:00 +CPU= 25.311
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002892
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002892
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002892
  20:29:53 STEP 2893 H=2893:00 +CPU= 15.053
  20:30:09 STEP 2894 H=2894:00 +CPU= 15.455
  20:30:32 STEP 2895 H=2895:00 +CPU= 23.279
  20:30:47 STEP 2896 H=2896:00 +CPU= 15.304
  20:31:02 STEP 2897 H=2897:00 +CPU= 14.983
  20:31:26 STEP 2898 H=2898:00 +CPU= 23.558
  20:31:41 STEP 2899 H=2899:00 +CPU= 14.927
  20:31:56 STEP 2900 H=2900:00 +CPU= 15.323
  20:32:19 STEP 2901 H=2901:00 +CPU= 23.260
  20:32:35 STEP 2902 H=2902:00 +CPU= 15.378
  20:32:50 STEP 2903 H=2903:00 +CPU= 15.354
  20:33:17 STEP 2904 H=2904:00 +CPU= 27.144
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002904
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002904
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_120.zip
Uploading the intermediate file: upload_file_120.zip
  20:33:33 STEP 2905 H=2905:00 +CPU= 15.129
  20:33:48 STEP 2906 H=2906:00 +CPU= 15.421
  20:34:12 STEP 2907 H=2907:00 +CPU= 23.472
  20:34:27 STEP 2908 H=2908:00 +CPU= 15.388
  20:34:42 STEP 2909 H=2909:00 +CPU= 15.128
  20:35:06 STEP 2910 H=2910:00 +CPU= 23.735
  20:35:21 STEP 2911 H=2911:00 +CPU= 15.066
  20:35:36 STEP 2912 H=2912:00 +CPU= 15.332
  20:36:00 STEP 2913 H=2913:00 +CPU= 23.366
  20:36:15 STEP 2914 H=2914:00 +CPU= 15.338
  20:36:30 STEP 2915 H=2915:00 +CPU= 14.949
  20:36:55 STEP 2916 H=2916:00 +CPU= 25.346
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002916
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002916
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002916
  20:37:10 STEP 2917 H=2917:00 +CPU= 15.104
  20:37:26 STEP 2918 H=2918:00 +CPU= 15.360
  20:37:49 STEP 2919 H=2919:00 +CPU= 23.393
  20:38:05 STEP 2920 H=2920:00 +CPU= 15.405
  20:38:20 STEP 2921 H=2921:00 +CPU= 15.028
  20:38:43 STEP 2922 H=2922:00 +CPU= 23.706
  20:38:58 STEP 2923 H=2923:00 +CPU= 14.981
  20:39:14 STEP 2924 H=2924:00 +CPU= 15.437
  20:39:37 STEP 2925 H=2925:00 +CPU= 23.272
  20:39:52 STEP 2926 H=2926:00 +CPU= 15.247
  20:40:08 STEP 2927 H=2927:00 +CPU= 15.387
  20:40:35 STEP 2928 H=2928:00 +CPU= 27.364
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002928
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002928
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_121.zip
Uploading the intermediate file: upload_file_121.zip
  20:40:50 STEP 2929 H=2929:00 +CPU= 15.024
  20:41:06 STEP 2930 H=2930:00 +CPU= 15.337
  20:41:29 STEP 2931 H=2931:00 +CPU= 23.106
  20:41:44 STEP 2932 H=2932:00 +CPU= 15.103
  20:41:59 STEP 2933 H=2933:00 +CPU= 14.793
  20:42:22 STEP 2934 H=2934:00 +CPU= 23.442
  20:42:37 STEP 2935 H=2935:00 +CPU= 14.800
  20:42:52 STEP 2936 H=2936:00 +CPU= 15.216
  20:43:15 STEP 2937 H=2937:00 +CPU= 22.852
  20:43:30 STEP 2938 H=2938:00 +CPU= 15.039
  20:43:45 STEP 2939 H=2939:00 +CPU= 14.630
  20:44:10 STEP 2940 H=2940:00 +CPU= 24.930
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002940
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002940
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002940
  20:44:24 STEP 2941 H=2941:00 +CPU= 14.627
  20:44:39 STEP 2942 H=2942:00 +CPU= 15.020
  20:45:02 STEP 2943 H=2943:00 +CPU= 22.949
  20:45:17 STEP 2944 H=2944:00 +CPU= 15.032
  20:45:32 STEP 2945 H=2945:00 +CPU= 14.704
  20:45:55 STEP 2946 H=2946:00 +CPU= 23.305
  20:46:10 STEP 2947 H=2947:00 +CPU= 14.751
  20:46:25 STEP 2948 H=2948:00 +CPU= 15.141
  20:46:48 STEP 2949 H=2949:00 +CPU= 23.012
  20:47:03 STEP 2950 H=2950:00 +CPU= 15.088
  20:47:18 STEP 2951 H=2951:00 +CPU= 15.149
  20:47:45 STEP 2952 H=2952:00 +CPU= 26.034
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 4.865283731608E-01 1.865286143845E-01 6.362598939121E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.260318722649E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.48989191816039E-02 -1.72994898449055E+00  1.89761310057317E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.19925486346338E-02 -1.00000000000000E+00  1.00000000000000E+00
  MAXGPFV : MAX. VALUE =    1.12970494489372     
  MAXGPFV : MAX. VALUE =   0.749663745091803     
 NSTEP =  2952 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGhnw4+002952 MODE=a
 IO-STREAM CLOSED - ICMGGhnw4+002952
 MPI-TASK:   1 -    305554410 BYTES IN      8 RECORDS TRANSFERRED IN    0.0004 SECONDS763886.0 Mbytes/s, TOTAL TIME=    0.0072( 5.6%)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302231500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302233000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302234500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200303000000  !!!!!!!!!!!! 
 20:47:45 STEP 2952 H=2952:00 +CPU= 26.034
 END CNT3
 NSTEP =  2952 CNT0   000000000  22.667  22.667  22.681   0.999               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           35/   804992476 WRITTEN         1865/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     48    0       35       786125     1865 ************     7.3     1.4     5.9     1.9     8.9
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 2203229.9 2203229.9    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 2200558.9 2200558.9     99.88      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  131    1598.4    1598.4      9.50      0.00
   9 SPCM     - SPECTRAL COMP.                     120     210.2     210.2      1.14      0.00
  10 SCAN2M - PHYSICS                              121    8285.0    8285.0     45.50      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 10     848.2     848.2      0.39      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  6      11.4      11.4      0.00      0.00
  13 SCAN2M - RADIATION CALC.                       41    8191.1    8191.1     15.24      0.00
  17 GRIDFPOS IN CNT4                               10      30.3      30.3      0.01      0.00
  27 RESTART FILES - WRITING                         5    1696.5    1696.5      0.38      0.00
  28 RESTART FILES - READING                         1    2333.1    2333.1      0.11      0.00
  29 SU4FPOS IN CNT4                                10       0.0       0.0      0.00      0.00
  30 DYNFPOS IN CNT4                                10    1431.6    1431.6      0.65      0.00
  31 POSDDH IN STEPO                                 1       9.3       9.3      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    121    2264.2    2264.2     12.43      0.00
  38 WAM      - TOTAL COST OF WAVE MODEL           121    1731.2    1731.2      9.51      0.00
  39 SU0YOMB                                         1     242.0     242.0      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                    121       7.4       7.4      0.04      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              120     158.6     158.6      0.86      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              120      22.1      22.1      0.12      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  6       0.1       0.1      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      416      85.1      85.1      1.61      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       250      89.0      89.0      1.01      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        250      36.7      36.7      0.42      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       416      27.9      27.9      0.53      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            2      13.4      13.4      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            416      13.1      13.1      0.25      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            250      12.3      12.3      0.14      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            416      13.8      13.8      0.26      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            250      12.2      12.2      0.14      0.00
 400 GSTATS                                      25115       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 23987       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     2203.2 CPU TIME    2200.5 VECTOR TIME     2200.5


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     4829.3
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMGGhnw4+002952
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMSHhnw4+002952
Moving to projects directory: /var/lib/boinc-client/slots/25/ICMUAhnw4+002952
Adding to the zip: /var/lib/boinc-client/slots/25/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/25/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMUAhnw4+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMSHhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214532/ICMGGhnw4+002928
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00ue_2019110100_123_993_12214532_0_r867478958_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
20:51:07 (1841): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:55:20 1537319 22315904 oifs_43r3_00ue_2019110100_123_993_12214532_0 10,623,600 70,033 0.0066
09 May 2023 16:55:15 1537319 22315904 oifs_43r3_00ue_2019110100_123_993_12214532_0 9,590,400 64,705 0.0067
09 May 2023 16:55:15 1537319 22315904 oifs_43r3_00ue_2019110100_123_993_12214532_0 8,726,400 60,404 0.0069
09 May 2023 16:55:01 1537319 22315904 oifs_43r3_00ue_2019110100_123_993_12214532_0 7,084,800 52,067 0.0073
09 May 2023 16:55:01 1537319 22315904 oifs_43r3_00ue_2019110100_123_993_12214532_0 6,220,800 47,059 0.0076
09 May 2023 16:55:00 1537319 22315904 oifs_43r3_00ue_2019110100_123_993_12214532_0 5,097,600 39,181 0.0077
09 May 2023 16:54:00 1537319 22315904 oifs_43r3_00ue_2019110100_123_993_12214532_0 4,060,800 31,705 0.0078
09 May 2023 16:53:58 1537319 22315904 oifs_43r3_00ue_2019110100_123_993_12214532_0 3,196,800 24,938 0.0078
09 May 2023 16:52:48 1537319 22315904 oifs_43r3_00ue_2019110100_123_993_12214532_0 1,900,800 14,542 0.0077
09 May 2023 16:52:44 1537319 22315904 oifs_43r3_00ue_2019110100_123_993_12214532_0 864,000 6,161 0.0071


©2024 climateprediction.net