climateprediction.net home page
Task 22305668

Task 22305668

Name oifs_43r3_ps_0408_2010050100_123_979_12196052_2
Workunit 12196052
Created 4 Feb 2023, 23:18:56 UTC
Sent 4 Feb 2023, 23:19:06 UTC
Report deadline 5 Apr 2023, 23:19:06 UTC
Received 6 Feb 2023, 5:15:16 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1534455
Run time 10 hours 15 min 12 sec
CPU time 10 hours 15 min 12 sec
Validate state Invalid
Credit 6,531.84
Device peak FLOPS 4.57 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.56 MB
Peak swap size 4,974.80 MB
Peak disk usage 2,640.25 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
824:00 +CPU= 25.931
  06:44:43 STEP  825 H= 825:00 +CPU= 39.385
  06:45:09 STEP  826 H= 826:00 +CPU= 25.925
  06:45:34 STEP  827 H= 827:00 +CPU= 25.118
  06:46:18 STEP  828 H= 828:00 +CPU= 43.798
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000828
  06:46:43 STEP  829 H= 829:00 +CPU= 25.088
  06:47:09 STEP  830 H= 830:00 +CPU= 25.905
  06:47:48 STEP  831 H= 831:00 +CPU= 39.363
  06:48:14 STEP  832 H= 832:00 +CPU= 25.931
  06:48:39 STEP  833 H= 833:00 +CPU= 25.105
  06:49:19 STEP  834 H= 834:00 +CPU= 40.263
  06:49:44 STEP  835 H= 835:00 +CPU= 25.041
  06:50:10 STEP  836 H= 836:00 +CPU= 25.844
  06:50:49 STEP  837 H= 837:00 +CPU= 39.271
  06:51:15 STEP  838 H= 838:00 +CPU= 25.827
  06:51:41 STEP  839 H= 839:00 +CPU= 25.755
  06:52:27 STEP  840 H= 840:00 +CPU= 46.222
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_34.zip
Uploading the intermediate file: upload_file_34.zip
  06:52:52 STEP  841 H= 841:00 +CPU= 24.961
  06:53:18 STEP  842 H= 842:00 +CPU= 25.730
  06:53:57 STEP  843 H= 843:00 +CPU= 39.275
  06:54:23 STEP  844 H= 844:00 +CPU= 25.814
  06:54:48 STEP  845 H= 845:00 +CPU= 24.979
  06:55:28 STEP  846 H= 846:00 +CPU= 40.271
  06:55:53 STEP  847 H= 847:00 +CPU= 25.064
  06:56:19 STEP  848 H= 848:00 +CPU= 25.915
  06:56:59 STEP  849 H= 849:00 +CPU= 39.340
  06:57:25 STEP  850 H= 850:00 +CPU= 25.939
  06:57:50 STEP  851 H= 851:00 +CPU= 25.062
  06:58:33 STEP  852 H= 852:00 +CPU= 43.690
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000852
  06:58:58 STEP  853 H= 853:00 +CPU= 24.971
  06:59:24 STEP  854 H= 854:00 +CPU= 25.791
  07:00:03 STEP  855 H= 855:00 +CPU= 39.216
  07:00:29 STEP  856 H= 856:00 +CPU= 25.871
  07:00:54 STEP  857 H= 857:00 +CPU= 25.013
  07:01:34 STEP  858 H= 858:00 +CPU= 40.184
  07:01:59 STEP  859 H= 859:00 +CPU= 24.860
  07:02:25 STEP  860 H= 860:00 +CPU= 25.721
  07:03:04 STEP  861 H= 861:00 +CPU= 39.158
  07:03:30 STEP  862 H= 862:00 +CPU= 25.739
  07:03:56 STEP  863 H= 863:00 +CPU= 25.700
  07:04:42 STEP  864 H= 864:00 +CPU= 46.196
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_35.zip
Uploading the intermediate file: upload_file_35.zip
  07:05:07 STEP  865 H= 865:00 +CPU= 24.994
  07:05:33 STEP  866 H= 866:00 +CPU= 25.852
  07:06:12 STEP  867 H= 867:00 +CPU= 39.265
  07:06:38 STEP  868 H= 868:00 +CPU= 25.712
  07:07:03 STEP  869 H= 869:00 +CPU= 24.902
  07:07:43 STEP  870 H= 870:00 +CPU= 40.110
  07:08:08 STEP  871 H= 871:00 +CPU= 24.935
  07:08:33 STEP  872 H= 872:00 +CPU= 25.805
  07:09:13 STEP  873 H= 873:00 +CPU= 39.256
  07:09:38 STEP  874 H= 874:00 +CPU= 25.751
  07:10:03 STEP  875 H= 875:00 +CPU= 24.963
  07:10:47 STEP  876 H= 876:00 +CPU= 43.705
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000876
  07:11:12 STEP  877 H= 877:00 +CPU= 25.028
  07:11:38 STEP  878 H= 878:00 +CPU= 25.819
  07:12:17 STEP  879 H= 879:00 +CPU= 39.298
  07:12:43 STEP  880 H= 880:00 +CPU= 25.739
  07:13:08 STEP  881 H= 881:00 +CPU= 24.918
  07:13:48 STEP  882 H= 882:00 +CPU= 40.217
  07:14:13 STEP  883 H= 883:00 +CPU= 24.931
  07:14:39 STEP  884 H= 884:00 +CPU= 25.714
  07:15:18 STEP  885 H= 885:00 +CPU= 39.272
  07:15:44 STEP  886 H= 886:00 +CPU= 25.695
  07:16:09 STEP  887 H= 887:00 +CPU= 25.692
  07:16:56 STEP  888 H= 888:00 +CPU= 46.082
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_36.zip
Uploading the intermediate file: upload_file_36.zip
  07:17:21 STEP  889 H= 889:00 +CPU= 24.932
  07:17:46 STEP  890 H= 890:00 +CPU= 25.666
  07:18:25 STEP  891 H= 891:00 +CPU= 39.205
  07:18:51 STEP  892 H= 892:00 +CPU= 25.729
  07:19:16 STEP  893 H= 893:00 +CPU= 24.959
  07:19:56 STEP  894 H= 894:00 +CPU= 40.096
  07:20:21 STEP  895 H= 895:00 +CPU= 25.042
  07:20:47 STEP  896 H= 896:00 +CPU= 25.856
  07:21:26 STEP  897 H= 897:00 +CPU= 39.350
  07:21:52 STEP  898 H= 898:00 +CPU= 25.838
  07:22:17 STEP  899 H= 899:00 +CPU= 25.095
  07:22:50 STEP  900 H= 900:00 +CPU= 32.233
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000900
  07:23:05 STEP  901 H= 901:00 +CPU= 14.993
  07:23:20 STEP  902 H= 902:00 +CPU= 15.518
  07:23:44 STEP  903 H= 903:00 +CPU= 23.726
  07:23:59 STEP  904 H= 904:00 +CPU= 15.336
  07:24:14 STEP  905 H= 905:00 +CPU= 15.010
  07:24:39 STEP  906 H= 906:00 +CPU= 24.281
  07:24:53 STEP  907 H= 907:00 +CPU= 14.866
  07:25:09 STEP  908 H= 908:00 +CPU= 15.397
  07:25:33 STEP  909 H= 909:00 +CPU= 23.841
  07:25:48 STEP  910 H= 910:00 +CPU= 15.342
  07:26:03 STEP  911 H= 911:00 +CPU= 15.320
  07:26:31 STEP  912 H= 912:00 +CPU= 27.874
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000900
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_37.zip
Uploading the intermediate file: upload_file_37.zip
  07:26:46 STEP  913 H= 913:00 +CPU= 15.276
  07:27:02 STEP  914 H= 914:00 +CPU= 15.368
  07:27:26 STEP  915 H= 915:00 +CPU= 23.949
  07:27:41 STEP  916 H= 916:00 +CPU= 15.370
  07:27:56 STEP  917 H= 917:00 +CPU= 14.881
  07:28:20 STEP  918 H= 918:00 +CPU= 24.383
  07:28:35 STEP  919 H= 919:00 +CPU= 14.981
  07:28:51 STEP  920 H= 920:00 +CPU= 15.434
  07:29:15 STEP  921 H= 921:00 +CPU= 23.888
  07:29:30 STEP  922 H= 922:00 +CPU= 15.483
  07:29:45 STEP  923 H= 923:00 +CPU= 14.930
  07:30:11 STEP  924 H= 924:00 +CPU= 26.354
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000924
  07:30:27 STEP  925 H= 925:00 +CPU= 15.074
  07:30:42 STEP  926 H= 926:00 +CPU= 15.394
  07:31:06 STEP  927 H= 927:00 +CPU= 23.751
  07:31:21 STEP  928 H= 928:00 +CPU= 15.507
  07:31:36 STEP  929 H= 929:00 +CPU= 14.883
  07:32:00 STEP  930 H= 930:00 +CPU= 24.131
  07:32:15 STEP  931 H= 931:00 +CPU= 14.982
  07:32:30 STEP  932 H= 932:00 +CPU= 15.309
  07:32:54 STEP  933 H= 933:00 +CPU= 23.661
  07:33:10 STEP  934 H= 934:00 +CPU= 15.437
  07:33:25 STEP  935 H= 935:00 +CPU= 15.408
  07:33:53 STEP  936 H= 936:00 +CPU= 27.789
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000924
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_38.zip
Uploading the intermediate file: upload_file_38.zip
  07:34:08 STEP  937 H= 937:00 +CPU= 15.274
  07:34:24 STEP  938 H= 938:00 +CPU= 15.525
  07:34:47 STEP  939 H= 939:00 +CPU= 23.747
  07:35:03 STEP  940 H= 940:00 +CPU= 15.318
  07:35:18 STEP  941 H= 941:00 +CPU= 14.993
  07:35:42 STEP  942 H= 942:00 +CPU= 24.201
  07:35:57 STEP  943 H= 943:00 +CPU= 14.880
  07:36:12 STEP  944 H= 944:00 +CPU= 15.462
  07:36:36 STEP  945 H= 945:00 +CPU= 23.776
  07:36:51 STEP  946 H= 946:00 +CPU= 15.341
  07:37:06 STEP  947 H= 947:00 +CPU= 14.942
  07:37:33 STEP  948 H= 948:00 +CPU= 26.348
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000948
  07:37:48 STEP  949 H= 949:00 +CPU= 14.890
  07:38:03 STEP  950 H= 950:00 +CPU= 15.382
  07:38:27 STEP  951 H= 951:00 +CPU= 23.790
  07:38:45 STEP  952 H= 952:00 +CPU= 18.781
  07:39:11 STEP  953 H= 953:00 +CPU= 25.244
  07:39:51 STEP  954 H= 954:00 +CPU= 40.464
  07:40:16 STEP  955 H= 955:00 +CPU= 25.199
  07:40:42 STEP  956 H= 956:00 +CPU= 25.898
  07:41:22 STEP  957 H= 957:00 +CPU= 39.339
  07:41:48 STEP  958 H= 958:00 +CPU= 25.880
  07:42:13 STEP  959 H= 959:00 +CPU= 25.883
  07:43:00 STEP  960 H= 960:00 +CPU= 46.145
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_39.zip
Uploading the intermediate file: upload_file_39.zip
Uploading trickle at timestep: 3456000
  07:43:25 STEP  961 H= 961:00 +CPU= 25.097
  07:43:51 STEP  962 H= 962:00 +CPU= 25.878
  07:44:30 STEP  963 H= 963:00 +CPU= 39.301
  07:44:56 STEP  964 H= 964:00 +CPU= 25.884
  07:45:21 STEP  965 H= 965:00 +CPU= 25.100
  07:46:01 STEP  966 H= 966:00 +CPU= 40.259
  07:46:26 STEP  967 H= 967:00 +CPU= 25.132
  07:46:52 STEP  968 H= 968:00 +CPU= 25.934
  07:47:32 STEP  969 H= 969:00 +CPU= 39.423
  07:47:58 STEP  970 H= 970:00 +CPU= 26.044
  07:48:23 STEP  971 H= 971:00 +CPU= 25.225
  07:49:07 STEP  972 H= 972:00 +CPU= 43.899
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000972
  07:49:32 STEP  973 H= 973:00 +CPU= 25.165
  07:49:58 STEP  974 H= 974:00 +CPU= 25.984
  07:50:37 STEP  975 H= 975:00 +CPU= 39.385
  07:51:03 STEP  976 H= 976:00 +CPU= 25.953
  07:51:28 STEP  977 H= 977:00 +CPU= 25.159
  07:52:09 STEP  978 H= 978:00 +CPU= 40.264
  07:52:34 STEP  979 H= 979:00 +CPU= 25.165
  07:53:00 STEP  980 H= 980:00 +CPU= 25.930
  07:53:39 STEP  981 H= 981:00 +CPU= 39.364
  07:54:05 STEP  982 H= 982:00 +CPU= 25.894
  07:54:31 STEP  983 H= 983:00 +CPU= 25.934
  07:55:17 STEP  984 H= 984:00 +CPU= 45.926
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_40.zip
Uploading the intermediate file: upload_file_40.zip
  07:55:42 STEP  985 H= 985:00 +CPU= 24.851
  07:56:07 STEP  986 H= 986:00 +CPU= 25.639
  07:56:46 STEP  987 H= 987:00 +CPU= 39.050
  07:57:12 STEP  988 H= 988:00 +CPU= 25.665
  07:57:37 STEP  989 H= 989:00 +CPU= 24.903
  07:58:17 STEP  990 H= 990:00 +CPU= 40.072
  07:58:42 STEP  991 H= 991:00 +CPU= 24.958
  07:59:08 STEP  992 H= 992:00 +CPU= 25.801
  07:59:47 STEP  993 H= 993:00 +CPU= 39.167
  08:00:13 STEP  994 H= 994:00 +CPU= 25.849
  08:00:38 STEP  995 H= 995:00 +CPU= 25.066
  08:01:22 STEP  996 H= 996:00 +CPU= 43.718
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000996
  08:01:47 STEP  997 H= 997:00 +CPU= 25.065
  08:02:13 STEP  998 H= 998:00 +CPU= 25.854
  08:02:52 STEP  999 H= 999:00 +CPU= 39.242
  08:03:18 STEP 1000 H=1000:00 +CPU= 25.769
  08:03:43 STEP 1001 H=1001:00 +CPU= 24.956
  08:04:23 STEP 1002 H=1002:00 +CPU= 40.135
  08:04:48 STEP 1003 H=1003:00 +CPU= 24.953
  08:05:13 STEP 1004 H=1004:00 +CPU= 25.758
  08:05:53 STEP 1005 H=1005:00 +CPU= 39.202
  08:06:18 STEP 1006 H=1006:00 +CPU= 25.778
  08:06:44 STEP 1007 H=1007:00 +CPU= 25.707
  08:07:30 STEP 1008 H=1008:00 +CPU= 45.741
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_41.zip
Uploading the intermediate file: upload_file_41.zip
  08:07:45 STEP 1009 H=1009:00 +CPU= 15.146
  08:08:00 STEP 1010 H=1010:00 +CPU= 15.354
  08:08:24 STEP 1011 H=1011:00 +CPU= 23.880
  08:08:40 STEP 1012 H=1012:00 +CPU= 15.325
  08:08:54 STEP 1013 H=1013:00 +CPU= 14.877
  08:09:19 STEP 1014 H=1014:00 +CPU= 24.369
  08:09:34 STEP 1015 H=1015:00 +CPU= 14.934
  08:09:49 STEP 1016 H=1016:00 +CPU= 15.364
  08:10:13 STEP 1017 H=1017:00 +CPU= 23.880
  08:10:29 STEP 1018 H=1018:00 +CPU= 15.637
  08:10:44 STEP 1019 H=1019:00 +CPU= 15.018
  08:11:10 STEP 1020 H=1020:00 +CPU= 26.504
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001020
  08:11:25 STEP 1021 H=1021:00 +CPU= 15.067
  08:11:41 STEP 1022 H=1022:00 +CPU= 15.412
  08:12:04 STEP 1023 H=1023:00 +CPU= 23.821
  08:12:20 STEP 1024 H=1024:00 +CPU= 15.583
  08:12:35 STEP 1025 H=1025:00 +CPU= 14.956
  08:12:59 STEP 1026 H=1026:00 +CPU= 24.307
  08:13:14 STEP 1027 H=1027:00 +CPU= 15.077
  08:13:30 STEP 1028 H=1028:00 +CPU= 15.475
  08:13:54 STEP 1029 H=1029:00 +CPU= 23.803
  08:14:09 STEP 1030 H=1030:00 +CPU= 15.462
  08:14:25 STEP 1031 H=1031:00 +CPU= 15.473
  08:14:53 STEP 1032 H=1032:00 +CPU= 27.853
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_42.zip
Uploading the intermediate file: upload_file_42.zip
  08:15:16 STEP 1033 H=1033:00 +CPU= 23.617
  08:15:42 STEP 1034 H=1034:00 +CPU= 26.092
  08:16:22 STEP 1035 H=1035:00 +CPU= 39.654
  08:16:48 STEP 1036 H=1036:00 +CPU= 26.066
  08:17:13 STEP 1037 H=1037:00 +CPU= 25.252
  08:17:54 STEP 1038 H=1038:00 +CPU= 40.572
  08:18:19 STEP 1039 H=1039:00 +CPU= 25.298
  08:18:45 STEP 1040 H=1040:00 +CPU= 26.045
  08:19:25 STEP 1041 H=1041:00 +CPU= 39.644
  08:19:51 STEP 1042 H=1042:00 +CPU= 26.067
  08:20:16 STEP 1043 H=1043:00 +CPU= 25.289
  08:21:00 STEP 1044 H=1044:00 +CPU= 44.135
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001044
  08:21:26 STEP 1045 H=1045:00 +CPU= 25.291
  08:21:52 STEP 1046 H=1046:00 +CPU= 26.062
  08:22:31 STEP 1047 H=1047:00 +CPU= 39.567
  08:22:57 STEP 1048 H=1048:00 +CPU= 26.007
  08:23:22 STEP 1049 H=1049:00 +CPU= 25.216
  08:24:03 STEP 1050 H=1050:00 +CPU= 40.464
  08:24:28 STEP 1051 H=1051:00 +CPU= 25.191
  08:24:54 STEP 1052 H=1052:00 +CPU= 25.969
  08:25:34 STEP 1053 H=1053:00 +CPU= 39.605
  08:26:00 STEP 1054 H=1054:00 +CPU= 26.036
  08:26:26 STEP 1055 H=1055:00 +CPU= 26.024
  08:27:12 STEP 1056 H=1056:00 +CPU= 46.625
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_43.zip
Uploading the intermediate file: upload_file_43.zip
  08:27:38 STEP 1057 H=1057:00 +CPU= 25.196
  08:28:04 STEP 1058 H=1058:00 +CPU= 26.018
  08:28:43 STEP 1059 H=1059:00 +CPU= 39.592
  08:29:09 STEP 1060 H=1060:00 +CPU= 26.077
  08:29:34 STEP 1061 H=1061:00 +CPU= 25.039
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3992MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3992 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.204] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.205] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS CLDPP T/S    (1002) 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.205] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.205] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.205] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                CLDPRG_LAYER 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083003:1675582203:29431.205] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 CLDPP 
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083004:1675582204:29432.205] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:25877:25877] [20230205:083004:1675582204:29432.205] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000FDFCBC  cldpp_                    250  cldpp.F90
oifs_43r3_model.e  0000000000FDEE1A  cldprg_layer_              85  cldprg_layer.F90
oifs_43r3_model.e  0000000000F05C3F  callpar_                  707  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0408
(argv4) batchid: 979
(argv5) wuid: 12196052
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_33c965200637f65127800395778cbadc to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0408_2010050100_123_979_12196052.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0408_2010050100_123_979_12196052.zip
ic_ancil_file: ic_ancil_12196052
ifsdata_file: ifsdata_12196052
climate_data_file: clim_data_12196052
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_71fbc9a604cafe6666de5d28f4c9a20f to: /var/lib/boinc-client/slots/5/ic_ancil_12196052.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12196052.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9c283c1cf5ca0f42bdb2e1b9e9608733 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12196052.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12196052.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12196052.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12196052.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12196052.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12196052.xml
-- Model is restarting --
Adjusting last_iter, 1062, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12196052.xml
last_cpu_time: 29409
upload_file_number: 44
last_iter: 1055
last_upload: 3801600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739
The child process has been launched with process id: 1927
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRF:1:1:1927:1927] [20230205:083142:1675582302:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:32:48 STEP 1056 H=1056:00 +CPU= 44.714
  08:33:13 STEP 1057 H=1057:00 +CPU= 24.939
  08:33:38 STEP 1058 H=1058:00 +CPU= 25.726
  08:34:18 STEP 1059 H=1059:00 +CPU= 39.137
  08:34:43 STEP 1060 H=1060:00 +CPU= 25.739
  08:35:08 STEP 1061 H=1061:00 +CPU= 24.942
  08:35:48 STEP 1062 H=1062:00 +CPU= 40.086
  08:36:13 STEP 1063 H=1063:00 +CPU= 24.990
  08:36:39 STEP 1064 H=1064:00 +CPU= 25.811
  08:37:19 STEP 1065 H=1065:00 +CPU= 39.302
  08:37:45 STEP 1066 H=1066:00 +CPU= 25.892
  08:38:10 STEP 1067 H=1067:00 +CPU= 25.048
  08:38:53 STEP 1068 H=1068:00 +CPU= 43.641
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001068
  08:39:18 STEP 1069 H=1069:00 +CPU= 25.052
  08:39:44 STEP 1070 H=1070:00 +CPU= 25.847
  08:40:23 STEP 1071 H=1071:00 +CPU= 39.271
  08:40:49 STEP 1072 H=1072:00 +CPU= 25.840
  08:41:14 STEP 1073 H=1073:00 +CPU= 25.013
  08:41:54 STEP 1074 H=1074:00 +CPU= 40.202
  08:42:20 STEP 1075 H=1075:00 +CPU= 25.025
  08:42:45 STEP 1076 H=1076:00 +CPU= 25.850
  08:43:25 STEP 1077 H=1077:00 +CPU= 39.260
  08:43:50 STEP 1078 H=1078:00 +CPU= 25.795
  08:44:16 STEP 1079 H=1079:00 +CPU= 25.766
  08:45:02 STEP 1080 H=1080:00 +CPU= 45.934
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_44.zip
Uploading the intermediate file: upload_file_44.zip
  08:45:27 STEP 1081 H=1081:00 +CPU= 24.950
  08:45:53 STEP 1082 H=1082:00 +CPU= 25.748
  08:46:32 STEP 1083 H=1083:00 +CPU= 39.261
  08:46:58 STEP 1084 H=1084:00 +CPU= 25.792
  08:47:23 STEP 1085 H=1085:00 +CPU= 24.989
  08:48:03 STEP 1086 H=1086:00 +CPU= 40.211
  08:48:28 STEP 1087 H=1087:00 +CPU= 25.061
  08:48:54 STEP 1088 H=1088:00 +CPU= 25.859
  08:49:34 STEP 1089 H=1089:00 +CPU= 39.400
  08:50:00 STEP 1090 H=1090:00 +CPU= 25.896
  08:50:25 STEP 1091 H=1091:00 +CPU= 25.069
  08:51:08 STEP 1092 H=1092:00 +CPU= 43.646
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001092
  08:51:33 STEP 1093 H=1093:00 +CPU= 25.000
  08:51:59 STEP 1094 H=1094:00 +CPU= 25.835
  08:52:39 STEP 1095 H=1095:00 +CPU= 39.230
  08:53:04 STEP 1096 H=1096:00 +CPU= 25.769
  08:53:29 STEP 1097 H=1097:00 +CPU= 24.896
  08:54:09 STEP 1098 H=1098:00 +CPU= 40.069
  08:54:34 STEP 1099 H=1099:00 +CPU= 24.900
  08:55:00 STEP 1100 H=1100:00 +CPU= 25.706
  08:55:39 STEP 1101 H=1101:00 +CPU= 39.119
  08:56:05 STEP 1102 H=1102:00 +CPU= 25.651
  08:56:30 STEP 1103 H=1103:00 +CPU= 25.617
  08:57:16 STEP 1104 H=1104:00 +CPU= 45.970
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_45.zip
Uploading the intermediate file: upload_file_45.zip
  08:57:42 STEP 1105 H=1105:00 +CPU= 25.048
  08:58:08 STEP 1106 H=1106:00 +CPU= 25.968
  08:58:47 STEP 1107 H=1107:00 +CPU= 39.507
  08:59:13 STEP 1108 H=1108:00 +CPU= 25.944
  08:59:38 STEP 1109 H=1109:00 +CPU= 25.172
  09:00:19 STEP 1110 H=1110:00 +CPU= 40.724
  09:00:44 STEP 1111 H=1111:00 +CPU= 25.527
  09:01:11 STEP 1112 H=1112:00 +CPU= 26.349
  09:01:51 STEP 1113 H=1113:00 +CPU= 39.700
  09:02:17 STEP 1114 H=1114:00 +CPU= 26.045
  09:02:42 STEP 1115 H=1115:00 +CPU= 25.238
  09:03:26 STEP 1116 H=1116:00 +CPU= 44.329
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001116
  09:03:52 STEP 1117 H=1117:00 +CPU= 25.444
  09:04:18 STEP 1118 H=1118:00 +CPU= 26.262
  09:04:58 STEP 1119 H=1119:00 +CPU= 39.669
  09:05:24 STEP 1120 H=1120:00 +CPU= 26.185
  09:05:49 STEP 1121 H=1121:00 +CPU= 25.210
  09:06:30 STEP 1122 H=1122:00 +CPU= 40.578
  09:06:55 STEP 1123 H=1123:00 +CPU= 25.267
  09:07:21 STEP 1124 H=1124:00 +CPU= 26.060
  09:08:01 STEP 1125 H=1125:00 +CPU= 39.702
  09:08:27 STEP 1126 H=1126:00 +CPU= 26.271
  09:08:53 STEP 1127 H=1127:00 +CPU= 26.288
  09:09:40 STEP 1128 H=1128:00 +CPU= 46.682
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_46.zip
Uploading the intermediate file: upload_file_46.zip
  09:10:05 STEP 1129 H=1129:00 +CPU= 25.308
  09:10:32 STEP 1130 H=1130:00 +CPU= 26.363
  09:11:11 STEP 1131 H=1131:00 +CPU= 39.724
  09:11:38 STEP 1132 H=1132:00 +CPU= 26.160
  09:12:03 STEP 1133 H=1133:00 +CPU= 25.278
  09:12:44 STEP 1134 H=1134:00 +CPU= 40.862
  09:13:09 STEP 1135 H=1135:00 +CPU= 25.659
  09:13:36 STEP 1136 H=1136:00 +CPU= 26.496
  09:14:16 STEP 1137 H=1137:00 +CPU= 40.130
  09:14:42 STEP 1138 H=1138:00 +CPU= 26.317
  09:15:08 STEP 1139 H=1139:00 +CPU= 25.340
  09:15:52 STEP 1140 H=1140:00 +CPU= 44.257
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001140
  09:16:18 STEP 1141 H=1141:00 +CPU= 25.483
  09:16:44 STEP 1142 H=1142:00 +CPU= 26.501
  09:17:24 STEP 1143 H=1143:00 +CPU= 40.221
  09:17:51 STEP 1144 H=1144:00 +CPU= 26.555
  09:18:17 STEP 1145 H=1145:00 +CPU= 25.744
  09:18:58 STEP 1146 H=1146:00 +CPU= 41.108
  09:19:23 STEP 1147 H=1147:00 +CPU= 25.713
  09:19:50 STEP 1148 H=1148:00 +CPU= 26.370
  09:20:30 STEP 1149 H=1149:00 +CPU= 40.124
  09:20:56 STEP 1150 H=1150:00 +CPU= 26.500
  09:21:23 STEP 1151 H=1151:00 +CPU= 26.518
  09:22:10 STEP 1152 H=1152:00 +CPU= 47.025
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_47.zip
Uploading the intermediate file: upload_file_47.zip
  09:22:36 STEP 1153 H=1153:00 +CPU= 25.738
  09:23:02 STEP 1154 H=1154:00 +CPU= 26.566
  09:23:43 STEP 1155 H=1155:00 +CPU= 40.244
  09:24:09 STEP 1156 H=1156:00 +CPU= 26.551
  09:24:35 STEP 1157 H=1157:00 +CPU= 25.672
  09:25:16 STEP 1158 H=1158:00 +CPU= 41.151
  09:25:42 STEP 1159 H=1159:00 +CPU= 25.753
  09:26:08 STEP 1160 H=1160:00 +CPU= 26.575
  09:26:49 STEP 1161 H=1161:00 +CPU= 40.260
  09:27:15 STEP 1162 H=1162:00 +CPU= 26.547
  09:27:41 STEP 1163 H=1163:00 +CPU= 25.718
  09:28:26 STEP 1164 H=1164:00 +CPU= 44.579
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001164
  09:28:51 STEP 1165 H=1165:00 +CPU= 25.750
  09:29:18 STEP 1166 H=1166:00 +CPU= 26.547
  09:29:58 STEP 1167 H=1167:00 +CPU= 40.086
  09:30:25 STEP 1168 H=1168:00 +CPU= 26.391
  09:30:50 STEP 1169 H=1169:00 +CPU= 25.400
  09:31:31 STEP 1170 H=1170:00 +CPU= 40.768
  09:31:56 STEP 1171 H=1171:00 +CPU= 25.586
  09:32:23 STEP 1172 H=1172:00 +CPU= 26.532
  09:33:03 STEP 1173 H=1173:00 +CPU= 40.268
  09:33:30 STEP 1174 H=1174:00 +CPU= 26.510
  09:33:56 STEP 1175 H=1175:00 +CPU= 26.525
  09:34:43 STEP 1176 H=1176:00 +CPU= 47.008
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_48.zip
Uploading the intermediate file: upload_file_48.zip
  09:35:09 STEP 1177 H=1177:00 +CPU= 25.677
  09:35:36 STEP 1178 H=1178:00 +CPU= 26.470
  09:36:16 STEP 1179 H=1179:00 +CPU= 40.230
  09:36:43 STEP 1180 H=1180:00 +CPU= 26.549
  09:37:08 STEP 1181 H=1181:00 +CPU= 25.749
  09:37:50 STEP 1182 H=1182:00 +CPU= 41.219
  09:38:15 STEP 1183 H=1183:00 +CPU= 25.773
  09:38:42 STEP 1184 H=1184:00 +CPU= 26.616
  09:39:22 STEP 1185 H=1185:00 +CPU= 40.239
  09:39:49 STEP 1186 H=1186:00 +CPU= 26.281
  09:40:14 STEP 1187 H=1187:00 +CPU= 25.558
  09:40:59 STEP 1188 H=1188:00 +CPU= 44.456
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001188
  09:41:24 STEP 1189 H=1189:00 +CPU= 25.735
  09:41:51 STEP 1190 H=1190:00 +CPU= 26.614
  09:42:31 STEP 1191 H=1191:00 +CPU= 40.159
  09:42:58 STEP 1192 H=1192:00 +CPU= 26.361
  09:43:23 STEP 1193 H=1193:00 +CPU= 25.387
  09:44:04 STEP 1194 H=1194:00 +CPU= 40.804
  09:44:29 STEP 1195 H=1195:00 +CPU= 25.446
  09:44:56 STEP 1196 H=1196:00 +CPU= 26.307
  09:45:36 STEP 1197 H=1197:00 +CPU= 40.182
  09:46:02 STEP 1198 H=1198:00 +CPU= 26.471
  09:46:29 STEP 1199 H=1199:00 +CPU= 26.457
  09:47:15 STEP 1200 H=1200:00 +CPU= 46.541
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_49.zip
Uploading the intermediate file: upload_file_49.zip
  09:47:41 STEP 1201 H=1201:00 +CPU= 25.615
  09:48:08 STEP 1202 H=1202:00 +CPU= 26.548
  09:48:48 STEP 1203 H=1203:00 +CPU= 40.247
  09:49:14 STEP 1204 H=1204:00 +CPU= 26.229
  09:49:40 STEP 1205 H=1205:00 +CPU= 25.299
  09:50:20 STEP 1206 H=1206:00 +CPU= 40.478
  09:50:45 STEP 1207 H=1207:00 +CPU= 25.077
  09:51:11 STEP 1208 H=1208:00 +CPU= 26.153
  09:51:51 STEP 1209 H=1209:00 +CPU= 39.900
  09:52:18 STEP 1210 H=1210:00 +CPU= 26.186
  09:52:43 STEP 1211 H=1211:00 +CPU= 25.466
  09:53:27 STEP 1212 H=1212:00 +CPU= 44.240
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001212
  09:53:53 STEP 1213 H=1213:00 +CPU= 25.494
  09:54:19 STEP 1214 H=1214:00 +CPU= 26.325
  09:54:59 STEP 1215 H=1215:00 +CPU= 39.862
  09:55:25 STEP 1216 H=1216:00 +CPU= 26.379
  09:55:51 STEP 1217 H=1217:00 +CPU= 25.247
  09:56:31 STEP 1218 H=1218:00 +CPU= 40.495
  09:56:56 STEP 1219 H=1219:00 +CPU= 25.277
  09:57:23 STEP 1220 H=1220:00 +CPU= 26.082
  09:58:02 STEP 1221 H=1221:00 +CPU= 39.669
  09:58:28 STEP 1222 H=1222:00 +CPU= 26.057
  09:58:54 STEP 1223 H=1223:00 +CPU= 25.973
  09:59:41 STEP 1224 H=1224:00 +CPU= 46.299
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_50.zip
Uploading the intermediate file: upload_file_50.zip
  10:00:06 STEP 1225 H=1225:00 +CPU= 25.433
  10:00:33 STEP 1226 H=1226:00 +CPU= 26.448
  10:01:12 STEP 1227 H=1227:00 +CPU= 39.294
  10:01:38 STEP 1228 H=1228:00 +CPU= 25.940
  10:02:03 STEP 1229 H=1229:00 +CPU= 25.040
  10:02:43 STEP 1230 H=1230:00 +CPU= 40.126
  10:03:08 STEP 1231 H=1231:00 +CPU= 24.911
  10:03:34 STEP 1232 H=1232:00 +CPU= 25.741
  10:04:13 STEP 1233 H=1233:00 +CPU= 39.404
  10:04:39 STEP 1234 H=1234:00 +CPU= 25.940
  10:05:04 STEP 1235 H=1235:00 +CPU= 25.004
  10:05:48 STEP 1236 H=1236:00 +CPU= 43.581
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001236
  10:06:13 STEP 1237 H=1237:00 +CPU= 25.213
  10:06:39 STEP 1238 H=1238:00 +CPU= 25.866
  10:07:18 STEP 1239 H=1239:00 +CPU= 39.116
  10:07:44 STEP 1240 H=1240:00 +CPU= 25.658
  10:08:09 STEP 1241 H=1241:00 +CPU= 24.874
  10:08:49 STEP 1242 H=1242:00 +CPU= 39.997
  10:09:14 STEP 1243 H=1243:00 +CPU= 24.934
  10:09:39 STEP 1244 H=1244:00 +CPU= 25.710
  10:10:18 STEP 1245 H=1245:00 +CPU= 39.083
  10:10:44 STEP 1246 H=1246:00 +CPU= 25.726
  10:11:10 STEP 1247 H=1247:00 +CPU= 25.801
  10:11:56 STEP 1248 H=1248:00 +CPU= 46.105
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_51.zip
Uploading the intermediate file: upload_file_51.zip
  10:12:21 STEP 1249 H=1249:00 +CPU= 25.157
  10:12:47 STEP 1250 H=1250:00 +CPU= 25.745
  10:13:26 STEP 1251 H=1251:00 +CPU= 39.081
  10:13:52 STEP 1252 H=1252:00 +CPU= 25.826
  10:14:17 STEP 1253 H=1253:00 +CPU= 24.976
  10:14:58 STEP 1254 H=1254:00 +CPU= 40.049
  10:15:22 STEP 1255 H=1255:00 +CPU= 24.852
  10:15:48 STEP 1256 H=1256:00 +CPU= 25.625
  10:16:27 STEP 1257 H=1257:00 +CPU= 38.982
  10:16:53 STEP 1258 H=1258:00 +CPU= 25.923
  10:17:18 STEP 1259 H=1259:00 +CPU= 25.151
  10:18:02 STEP 1260 H=1260:00 +CPU= 43.907
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001260
  10:18:28 STEP 1261 H=1261:00 +CPU= 25.229
  10:18:54 STEP 1262 H=1262:00 +CPU= 26.207
  10:19:34 STEP 1263 H=1263:00 +CPU= 39.665
  10:20:00 STEP 1264 H=1264:00 +CPU= 25.805
  10:20:25 STEP 1265 H=1265:00 +CPU= 25.090
  10:21:05 STEP 1266 H=1266:00 +CPU= 40.169
  10:21:30 STEP 1267 H=1267:00 +CPU= 25.116
  10:21:56 STEP 1268 H=1268:00 +CPU= 25.994
  10:22:36 STEP 1269 H=1269:00 +CPU= 39.302
  10:23:01 STEP 1270 H=1270:00 +CPU= 25.625
  10:23:27 STEP 1271 H=1271:00 +CPU= 25.613
  10:24:13 STEP 1272 H=1272:00 +CPU= 45.878
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_52.zip
Uploading the intermediate file: upload_file_52.zip
  10:24:38 STEP 1273 H=1273:00 +CPU= 24.784
  10:25:04 STEP 1274 H=1274:00 +CPU= 25.769
  10:25:43 STEP 1275 H=1275:00 +CPU= 39.283
  10:26:09 STEP 1276 H=1276:00 +CPU= 25.703
  10:26:34 STEP 1277 H=1277:00 +CPU= 25.000
  10:27:14 STEP 1278 H=1278:00 +CPU= 40.203
  10:27:39 STEP 1279 H=1279:00 +CPU= 25.108
  10:28:05 STEP 1280 H=1280:00 +CPU= 25.953
  10:28:44 STEP 1281 H=1281:00 +CPU= 39.326
  10:29:10 STEP 1282 H=1282:00 +CPU= 25.828
  10:29:35 STEP 1283 H=1283:00 +CPU= 25.088
  10:30:19 STEP 1284 H=1284:00 +CPU= 43.700
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001284
  10:30:44 STEP 1285 H=1285:00 +CPU= 25.185
  10:31:10 STEP 1286 H=1286:00 +CPU= 25.852
  10:31:50 STEP 1287 H=1287:00 +CPU= 39.472
  10:32:16 STEP 1288 H=1288:00 +CPU= 26.215
  10:32:41 STEP 1289 H=1289:00 +CPU= 25.029
  10:33:22 STEP 1290 H=1290:00 +CPU= 40.456
  10:33:47 STEP 1291 H=1291:00 +CPU= 25.009
  10:34:13 STEP 1292 H=1292:00 +CPU= 25.833
  10:34:52 STEP 1293 H=1293:00 +CPU= 39.379
  10:35:18 STEP 1294 H=1294:00 +CPU= 25.814
  10:35:44 STEP 1295 H=1295:00 +CPU= 25.817
  10:36:30 STEP 1296 H=1296:00 +CPU= 46.212
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196052/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0408_2010050100_123_979_12196052_2_r681945739_53.zip
Uploading the intermediate file: upload_file_53.zip
  10:36:56 STEP 1297 H=1297:00 +CPU= 25.150
Uploading trickle at timestep: 4665600
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:48:09 1534455 22305668 oifs_43r3_ps_0408_2010050100_123_979_12196052_2 4,665,600 36,880 0.0079
09 May 2023 16:48:05 1534455 22305668 oifs_43r3_ps_0408_2010050100_123_979_12196052_2 3,456,000 26,599 0.0077
09 May 2023 16:48:02 1534455 22305668 oifs_43r3_ps_0408_2010050100_123_979_12196052_2 2,592,000 19,889 0.0077
09 May 2023 16:47:59 1534455 22305668 oifs_43r3_ps_0408_2010050100_123_979_12196052_2 1,728,000 12,869 0.0074
09 May 2023 16:47:58 1534455 22305668 oifs_43r3_ps_0408_2010050100_123_979_12196052_2 864,000 6,840 0.0079


©2024 climateprediction.net