<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
= 626:00 +CPU= 25.420
13:56:25 STEP 627 H= 627:00 +CPU= 37.053
13:56:54 STEP 628 H= 628:00 +CPU= 25.220
13:57:21 STEP 629 H= 629:00 +CPU= 24.337
13:58:04 STEP 630 H= 630:00 +CPU= 37.919
13:58:32 STEP 631 H= 631:00 +CPU= 24.462
13:59:02 STEP 632 H= 632:00 +CPU= 25.379
13:59:44 STEP 633 H= 633:00 +CPU= 37.330
14:00:13 STEP 634 H= 634:00 +CPU= 25.479
14:00:42 STEP 635 H= 635:00 +CPU= 25.080
14:01:32 STEP 636 H= 636:00 +CPU= 42.120
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000636
14:02:00 STEP 637 H= 637:00 +CPU= 24.762
14:02:31 STEP 638 H= 638:00 +CPU= 26.011
14:03:15 STEP 639 H= 639:00 +CPU= 38.281
14:03:44 STEP 640 H= 640:00 +CPU= 25.181
14:04:12 STEP 641 H= 641:00 +CPU= 24.498
14:04:52 STEP 642 H= 642:00 +CPU= 37.637
14:05:15 STEP 643 H= 643:00 +CPU= 22.320
14:05:38 STEP 644 H= 644:00 +CPU= 22.219
14:06:13 STEP 645 H= 645:00 +CPU= 33.971
14:06:37 STEP 646 H= 646:00 +CPU= 23.047
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:07:28 STEP 647 H= 647:00 +CPU= 26.336
14:08:17 STEP 648 H= 648:00 +CPU= 43.519
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0840_1982050100_123_951_12168484_0_r1408553325_26.zip
Uploading the intermediate file: upload_file_26.zip
14:08:45 STEP 649 H= 649:00 +CPU= 24.800
14:09:13 STEP 650 H= 650:00 +CPU= 24.751
14:09:55 STEP 651 H= 651:00 +CPU= 37.318
14:10:21 STEP 652 H= 652:00 +CPU= 24.504
14:10:45 STEP 653 H= 653:00 +CPU= 22.847
14:11:25 STEP 654 H= 654:00 +CPU= 36.749
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:12:00 STEP 655 H= 655:00 +CPU= 22.405
14:12:24 STEP 656 H= 656:00 +CPU= 22.872
14:12:59 STEP 657 H= 657:00 +CPU= 33.801
14:13:25 STEP 658 H= 658:00 +CPU= 23.747
14:13:50 STEP 659 H= 659:00 +CPU= 23.623
14:14:28 STEP 660 H= 660:00 +CPU= 37.048
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000660
14:14:53 STEP 661 H= 661:00 +CPU= 23.378
14:15:17 STEP 662 H= 662:00 +CPU= 23.024
14:15:55 STEP 663 H= 663:00 +CPU= 35.404
14:16:21 STEP 664 H= 664:00 +CPU= 24.490
14:16:45 STEP 665 H= 665:00 +CPU= 22.847
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:17:38 STEP 666 H= 666:00 +CPU= 37.662
14:18:04 STEP 667 H= 667:00 +CPU= 23.956
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:18:44 STEP 668 H= 668:00 +CPU= 25.460
14:19:20 STEP 669 H= 669:00 +CPU= 33.688
14:19:42 STEP 670 H= 670:00 +CPU= 22.021
14:20:05 STEP 671 H= 671:00 +CPU= 22.062
14:20:44 STEP 672 H= 672:00 +CPU= 38.347
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0840_1982050100_123_951_12168484_0_r1408553325_27.zip
Uploading the intermediate file: upload_file_27.zip
14:21:06 STEP 673 H= 673:00 +CPU= 21.638
14:21:27 STEP 674 H= 674:00 +CPU= 21.381
14:21:59 STEP 675 H= 675:00 +CPU= 31.765
14:22:21 STEP 676 H= 676:00 +CPU= 21.288
14:22:42 STEP 677 H= 677:00 +CPU= 20.557
14:23:15 STEP 678 H= 678:00 +CPU= 32.879
14:23:37 STEP 679 H= 679:00 +CPU= 21.844
14:24:00 STEP 680 H= 680:00 +CPU= 22.180
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:24:50 STEP 681 H= 681:00 +CPU= 35.845
14:25:15 STEP 682 H= 682:00 +CPU= 23.315
14:25:39 STEP 683 H= 683:00 +CPU= 22.612
14:26:21 STEP 684 H= 684:00 +CPU= 40.190
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000684
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:27:52 STEP 685 H= 685:00 +CPU= 25.103
14:28:19 STEP 686 H= 686:00 +CPU= 24.250
14:28:59 STEP 687 H= 687:00 +CPU= 35.894
14:29:27 STEP 688 H= 688:00 +CPU= 24.581
14:29:54 STEP 689 H= 689:00 +CPU= 23.830
14:30:36 STEP 690 H= 690:00 +CPU= 36.848
14:31:04 STEP 691 H= 691:00 +CPU= 24.104
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:31:54 STEP 692 H= 692:00 +CPU= 25.049
14:32:35 STEP 693 H= 693:00 +CPU= 36.192
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:33:33 STEP 694 H= 694:00 +CPU= 24.305
14:33:55 STEP 695 H= 695:00 +CPU= 21.680
14:34:36 STEP 696 H= 696:00 +CPU= 39.468
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0840_1982050100_123_951_12168484_0_r1408553325_28.zip
Uploading the intermediate file: upload_file_28.zip
14:35:02 STEP 697 H= 697:00 +CPU= 24.011
14:35:30 STEP 698 H= 698:00 +CPU= 25.668
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:36:22 STEP 699 H= 699:00 +CPU= 36.839
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:39:55 STEP 700 H= 700:00 +CPU= 26.513
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:40:55 STEP 701 H= 701:00 +CPU= 24.756
14:41:31 STEP 702 H= 702:00 +CPU= 34.535
14:41:55 STEP 703 H= 703:00 +CPU= 22.978
14:42:21 STEP 704 H= 704:00 +CPU= 24.253
14:42:57 STEP 705 H= 705:00 +CPU= 33.796
14:43:21 STEP 706 H= 706:00 +CPU= 23.263
14:43:45 STEP 707 H= 707:00 +CPU= 22.715
14:44:26 STEP 708 H= 708:00 +CPU= 38.859
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000708
14:44:50 STEP 709 H= 709:00 +CPU= 22.858
14:45:14 STEP 710 H= 710:00 +CPU= 23.319
14:45:50 STEP 711 H= 711:00 +CPU= 34.404
14:46:14 STEP 712 H= 712:00 +CPU= 22.766
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:46:50 STEP 713 H= 713:00 +CPU= 23.524
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:47:41 STEP 714 H= 714:00 +CPU= 36.581
14:48:03 STEP 715 H= 715:00 +CPU= 21.924
14:48:28 STEP 716 H= 716:00 +CPU= 23.185
14:49:04 STEP 717 H= 717:00 +CPU= 34.304
14:49:29 STEP 718 H= 718:00 +CPU= 23.108
14:49:54 STEP 719 H= 719:00 +CPU= 24.101
14:50:38 STEP 720 H= 720:00 +CPU= 41.525
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0840_1982050100_123_951_12168484_0_r1408553325_29.zip
Uploading the intermediate file: upload_file_29.zip
14:51:02 STEP 721 H= 721:00 +CPU= 22.443
14:51:28 STEP 722 H= 722:00 +CPU= 24.411
14:52:06 STEP 723 H= 723:00 +CPU= 35.595
14:52:31 STEP 724 H= 724:00 +CPU= 23.858
14:52:57 STEP 725 H= 725:00 +CPU= 23.623
14:53:36 STEP 726 H= 726:00 +CPU= 36.532
14:54:00 STEP 727 H= 727:00 +CPU= 23.201
14:54:25 STEP 728 H= 728:00 +CPU= 23.725
14:55:05 STEP 729 H= 729:00 +CPU= 36.639
14:55:31 STEP 730 H= 730:00 +CPU= 24.678
14:55:54 STEP 731 H= 731:00 +CPU= 21.854
14:56:35 STEP 732 H= 732:00 +CPU= 39.404
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000732
14:57:00 STEP 733 H= 733:00 +CPU= 23.194
14:57:25 STEP 734 H= 734:00 +CPU= 23.640
14:58:03 STEP 735 H= 735:00 +CPU= 35.500
14:58:29 STEP 736 H= 736:00 +CPU= 24.343
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:59:31 STEP 737 H= 737:00 +CPU= 26.158
15:00:12 STEP 738 H= 738:00 +CPU= 37.648
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:16:48 STEP 739 H= 739:00 +CPU= 24.751
15:17:17 STEP 740 H= 740:00 +CPU= 25.561
15:17:54 STEP 741 H= 741:00 +CPU= 34.385
15:18:23 STEP 742 H= 742:00 +CPU= 25.331
15:18:52 STEP 743 H= 743:00 +CPU= 25.694
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:20:12 STEP 744 H= 744:00 +CPU= 43.170
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000732
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0840_1982050100_123_951_12168484_0_r1408553325_30.zip
Uploading the intermediate file: upload_file_30.zip
15:20:36 STEP 745 H= 745:00 +CPU= 22.169
15:21:00 STEP 746 H= 746:00 +CPU= 22.572
15:21:37 STEP 747 H= 747:00 +CPU= 33.879
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:27:01 STEP 748 H= 748:00 +CPU= 25.762
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:27:50 STEP 749 H= 749:00 +CPU= 24.816
15:28:32 STEP 750 H= 750:00 +CPU= 37.466
15:28:58 STEP 751 H= 751:00 +CPU= 23.308
15:29:27 STEP 752 H= 752:00 +CPU= 24.972
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:30:58 STEP 753 H= 753:00 +CPU= 36.009
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:33:17 STEP 754 H= 754:00 +CPU= 24.757
15:33:44 STEP 755 H= 755:00 +CPU= 24.466
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:36:32 STEP 756 H= 756:00 +CPU= 40.709
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000756
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:41:42 STEP 757 H= 757:00 +CPU= 24.100
15:42:10 STEP 758 H= 758:00 +CPU= 25.086
15:42:51 STEP 759 H= 759:00 +CPU= 36.755
15:43:19 STEP 760 H= 760:00 +CPU= 24.658
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:44:07 STEP 761 H= 761:00 +CPU= 23.768
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:50:23 STEP 762 H= 762:00 +CPU= 37.744
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:51:24 STEP 763 H= 763:00 +CPU= 24.714
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:52:14 STEP 764 H= 764:00 +CPU= 24.933
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:53:28 STEP 765 H= 765:00 +CPU= 36.001
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:05:28 STEP 766 H= 766:00 +CPU= 26.285
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:13:42 STEP 767 H= 767:00 +CPU= 25.480
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:14:41 STEP 768 H= 768:00 +CPU= 43.306
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0840_1982050100_123_951_12168484_0_r1408553325_31.zip
Uploading the intermediate file: upload_file_31.zip
Uploading trickle at timestep: 2764800
16:15:09 STEP 769 H= 769:00 +CPU= 24.572
16:15:37 STEP 770 H= 770:00 +CPU= 25.107
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:16:30 STEP 771 H= 771:00 +CPU= 37.291
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:17:09 STEP 772 H= 772:00 +CPU= 25.386
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1982050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0840
(argv4) batchid: 951
(argv5) wuid: 12168484
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_dd2b5fd8177217c0cd5a2102ee7b2f7d to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0840_1982050100_123_951_12168484.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0840_1982050100_123_951_12168484.zip
ic_ancil_file: ic_ancil_12168484
ifsdata_file: ifsdata_12168484
climate_data_file: clim_data_12168484
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d45d9291126066868fb815ed51225022 to: /var/lib/boinc-client/slots/11/ic_ancil_12168484.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12168484.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_49b030913fe89c34b1e4c735e3705747 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12168484.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12168484.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12168484.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12168484.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12168484.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12168484.xml
-- Model is restarting --
Adjusting last_iter, 773, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12168484.xml
last_cpu_time: 21789
upload_file_number: 32
last_iter: 767
last_upload: 2764800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0840_1982050100_123_951_12168484_0_r1408553325
The child process has been launched with process id: 2944099
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:110556:1675382756:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
11:08:06 STEP 768 H= 768:00 +CPU= 87.501
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:09:21 STEP 769 H= 769:00 +CPU= 44.870
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:24:31 STEP 770 H= 770:00 +CPU= 47.574
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:27:41 STEP 771 H= 771:00 +CPU= 64.796
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:42:07 STEP 772 H= 772:00 +CPU= 43.720
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:46:03 STEP 773 H= 773:00 +CPU= 42.672
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:47:57 STEP 774 H= 774:00 +CPU= 57.450
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:50:35 STEP 775 H= 775:00 +CPU= 38.140
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
double free or corruption (top)
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.201] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3993MB (heap), 4403MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.201] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.201] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.201] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.201] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.201] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3993 MB (maxheap), 4403 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.202] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.202] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS_DRV
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-PHYSICS (1001)
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALLPAR
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CONVECTION_LAYER
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CUCALLN
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CUMASTRN
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115222:1675385542:2786.203] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CUFLXN
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115223:1675385543:2787.203] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Lian:1:1:2944099:2944099] [20230203:115223:1675385543:2787.203] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 00000000010DCDD1 cuflxn_ 306 cuflxn.F90
oifs_43r3_model.e 00000000010D3450 cumastrn_ 924 cumastrn.F90
oifs_43r3_model.e 00000000010C7792 cucalln_ 352 cucalln.F90
oifs_43r3_model.e 00000000010C4C15 convection_layer_ 121 convection_layer.F90
oifs_43r3_model.e 0000000000F11F5B callpar_ 1064 callpar.F90
oifs_43r3_model.e 0000000000E4940F ec_phys_ 670 ec_phys.F90
oifs_43r3_model.e 0000000000E3610C ec_phys_drv_ 599 ec_phys_drv.F90
oifs_43r3_model.e 0000000000BFA3D3 gp_model_ 613 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1982050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0840
(argv4) batchid: 951
(argv5) wuid: 12168484
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_dd2b5fd8177217c0cd5a2102ee7b2f7d to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0840_1982050100_123_951_12168484.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0840_1982050100_123_951_12168484.zip
ic_ancil_file: ic_ancil_12168484
ifsdata_file: ifsdata_12168484
climate_data_file: clim_data_12168484
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d45d9291126066868fb815ed51225022 to: /var/lib/boinc-client/slots/11/ic_ancil_12168484.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12168484.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_49b030913fe89c34b1e4c735e3705747 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12168484.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12168484.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12168484.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12168484.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12168484.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12168484.xml
-- Model is restarting --
Adjusting last_iter, 776, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12168484.xml
last_cpu_time: 22236
upload_file_number: 32
last_iter: 767
last_upload: 2764800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0840_1982050100_123_951_12168484_0_r1408553325
The child process has been launched with process id: 5316
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:5316:5316] [20230203:115457:1675385697:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:57:49 STEP 768 H= 768:00 +CPU= 79.183
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:00:01 STEP 769 H= 769:00 +CPU= 45.075
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:14:16 STEP 770 H= 770:00 +CPU= 47.558
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:16:48 STEP 771 H= 771:00 +CPU= 76.182
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:19:58 STEP 772 H= 772:00 +CPU= 49.484
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:22:45 STEP 773 H= 773:00 +CPU= 48.446
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:28:20 STEP 774 H= 774:00 +CPU= 73.755
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:31:56 STEP 775 H= 775:00 +CPU= 47.945
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:34:47 STEP 776 H= 776:00 +CPU= 50.305
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:38:22 STEP 777 H= 777:00 +CPU= 72.146
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:40:40 STEP 778 H= 778:00 +CPU= 50.787
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:44:24 STEP 779 H= 779:00 +CPU= 50.178
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:46:38 STEP 780 H= 780:00 +CPU= 89.083
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000780
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:48:11 STEP 781 H= 781:00 +CPU= 51.392
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:49:48 STEP 782 H= 782:00 +CPU= 51.226
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:53:37 STEP 783 H= 783:00 +CPU= 71.091
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:55:22 STEP 784 H= 784:00 +CPU= 50.126
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:57:43 STEP 785 H= 785:00 +CPU= 47.224
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:00:47 STEP 786 H= 786:00 +CPU= 73.761
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:03:25 STEP 787 H= 787:00 +CPU= 45.723
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:08:26 STEP 788 H= 788:00 +CPU= 47.263
13:10:00 STEP 789 H= 789:00 +CPU= 67.075
13:11:05 STEP 790 H= 790:00 +CPU= 45.870
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:12:34 STEP 791 H= 791:00 +CPU= 47.330
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:16:34 STEP 792 H= 792:00 +CPU= 86.978
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168484/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0840_1982050100_123_951_12168484_0_r1408553325_32.zip
Uploading the intermediate file: upload_file_32.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:18:15 STEP 793 H= 793:00 +CPU= 41.975
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:21:36 STEP 794 H= 794:00 +CPU= 49.442
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Abort request received from BOINC client, ending the child process
</stderr_txt>
]]>
