climateprediction.net home page
Task 22261092

Task 22261092

Name oifs_43r3_ps_0700_1990050100_123_959_12176344_0
Workunit 12176344
Created 21 Dec 2022, 19:52:29 UTC
Sent 15 Jan 2023, 22:01:38 UTC
Report deadline 14 Feb 2023, 22:01:38 UTC
Received 17 Jan 2023, 3:47:58 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1517434
Run time 15 hours 40 min 46 sec
CPU time 15 hours 40 min 46 sec
Validate state Invalid
Credit 11,128.32
Device peak FLOPS 4.91 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.47 MB
Peak swap size 4,978.07 MB
Peak disk usage 3,637.64 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
ing to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001884
  02:27:32 STEP 1885 H=1885:00 +CPU= 21.186
  02:27:54 STEP 1886 H=1886:00 +CPU= 21.919
  02:28:28 STEP 1887 H=1887:00 +CPU= 33.293
  02:28:50 STEP 1888 H=1888:00 +CPU= 21.911
  02:29:11 STEP 1889 H=1889:00 +CPU= 21.234
  02:29:45 STEP 1890 H=1890:00 +CPU= 34.284
  02:30:06 STEP 1891 H=1891:00 +CPU= 21.200
  02:30:28 STEP 1892 H=1892:00 +CPU= 21.965
  02:31:02 STEP 1893 H=1893:00 +CPU= 33.316
  02:31:24 STEP 1894 H=1894:00 +CPU= 21.872
  02:31:44 STEP 1895 H=1895:00 +CPU= 20.428
  02:32:15 STEP 1896 H=1896:00 +CPU= 30.635
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_78.zip
Uploading the intermediate file: upload_file_78.zip
  02:32:29 STEP 1897 H=1897:00 +CPU= 14.513
  02:32:44 STEP 1898 H=1898:00 +CPU= 14.681
  02:33:07 STEP 1899 H=1899:00 +CPU= 22.779
  02:33:21 STEP 1900 H=1900:00 +CPU= 14.687
  02:33:36 STEP 1901 H=1901:00 +CPU= 14.248
  02:33:59 STEP 1902 H=1902:00 +CPU= 23.397
  02:34:13 STEP 1903 H=1903:00 +CPU= 14.275
  02:34:28 STEP 1904 H=1904:00 +CPU= 14.744
  02:34:51 STEP 1905 H=1905:00 +CPU= 23.466
  02:35:06 STEP 1906 H=1906:00 +CPU= 14.719
  02:35:20 STEP 1907 H=1907:00 +CPU= 14.266
  02:35:46 STEP 1908 H=1908:00 +CPU= 25.241
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001908
  02:36:00 STEP 1909 H=1909:00 +CPU= 14.430
  02:36:18 STEP 1910 H=1910:00 +CPU= 17.904
  02:36:51 STEP 1911 H=1911:00 +CPU= 33.237
  02:37:13 STEP 1912 H=1912:00 +CPU= 21.946
  02:37:34 STEP 1913 H=1913:00 +CPU= 21.149
  02:38:08 STEP 1914 H=1914:00 +CPU= 34.118
  02:38:30 STEP 1915 H=1915:00 +CPU= 21.132
  02:38:51 STEP 1916 H=1916:00 +CPU= 21.860
  02:39:24 STEP 1917 H=1917:00 +CPU= 33.026
  02:39:46 STEP 1918 H=1918:00 +CPU= 21.537
  02:40:08 STEP 1919 H=1919:00 +CPU= 21.520
  02:40:46 STEP 1920 H=1920:00 +CPU= 38.839
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_79.zip
Uploading the intermediate file: upload_file_79.zip
  02:41:07 STEP 1921 H=1921:00 +CPU= 20.857
  02:41:29 STEP 1922 H=1922:00 +CPU= 21.581
  02:42:02 STEP 1923 H=1923:00 +CPU= 32.844
  02:42:23 STEP 1924 H=1924:00 +CPU= 21.538
  02:42:44 STEP 1925 H=1925:00 +CPU= 20.852
  02:43:09 STEP 1926 H=1926:00 +CPU= 25.011
  02:43:24 STEP 1927 H=1927:00 +CPU= 14.427
  02:43:38 STEP 1928 H=1928:00 +CPU= 14.717
  02:44:01 STEP 1929 H=1929:00 +CPU= 22.749
  02:44:16 STEP 1930 H=1930:00 +CPU= 14.742
  02:44:30 STEP 1931 H=1931:00 +CPU= 14.292
  02:44:55 STEP 1932 H=1932:00 +CPU= 25.144
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001932
  02:45:10 STEP 1933 H=1933:00 +CPU= 14.296
  02:45:24 STEP 1934 H=1934:00 +CPU= 14.721
  02:45:47 STEP 1935 H=1935:00 +CPU= 22.890
  02:46:02 STEP 1936 H=1936:00 +CPU= 14.698
  02:46:16 STEP 1937 H=1937:00 +CPU= 14.252
  02:46:39 STEP 1938 H=1938:00 +CPU= 23.206
  02:46:54 STEP 1939 H=1939:00 +CPU= 14.243
  02:47:09 STEP 1940 H=1940:00 +CPU= 14.923
  02:47:31 STEP 1941 H=1941:00 +CPU= 22.718
  02:47:46 STEP 1942 H=1942:00 +CPU= 14.657
  02:48:01 STEP 1943 H=1943:00 +CPU= 14.690
  02:48:27 STEP 1944 H=1944:00 +CPU= 26.771
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_80.zip
Uploading the intermediate file: upload_file_80.zip
  02:48:42 STEP 1945 H=1945:00 +CPU= 14.525
  02:48:57 STEP 1946 H=1946:00 +CPU= 14.692
  02:49:19 STEP 1947 H=1947:00 +CPU= 22.727
  02:49:34 STEP 1948 H=1948:00 +CPU= 14.699
  02:49:48 STEP 1949 H=1949:00 +CPU= 14.247
  02:50:11 STEP 1950 H=1950:00 +CPU= 23.211
  02:50:26 STEP 1951 H=1951:00 +CPU= 14.253
  02:50:40 STEP 1952 H=1952:00 +CPU= 14.699
  02:51:03 STEP 1953 H=1953:00 +CPU= 22.704
  02:51:18 STEP 1954 H=1954:00 +CPU= 14.722
  02:51:32 STEP 1955 H=1955:00 +CPU= 14.270
  02:51:57 STEP 1956 H=1956:00 +CPU= 25.192
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001956
  02:52:18 STEP 1957 H=1957:00 +CPU= 20.543
  02:52:39 STEP 1958 H=1958:00 +CPU= 21.566
  02:53:12 STEP 1959 H=1959:00 +CPU= 32.761
  02:53:34 STEP 1960 H=1960:00 +CPU= 21.545
  02:53:52 STEP 1961 H=1961:00 +CPU= 18.493
  02:54:15 STEP 1962 H=1962:00 +CPU= 23.214
  02:54:30 STEP 1963 H=1963:00 +CPU= 14.261
  02:54:50 STEP 1964 H=1964:00 +CPU= 19.878
  02:55:17 STEP 1965 H=1965:00 +CPU= 27.868
  02:55:32 STEP 1966 H=1966:00 +CPU= 14.674
  02:55:47 STEP 1967 H=1967:00 +CPU= 14.696
  02:56:26 STEP 1968 H=1968:00 +CPU= 39.290
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_81.zip
Uploading the intermediate file: upload_file_81.zip
  02:56:47 STEP 1969 H=1969:00 +CPU= 21.160
Uploading trickle at timestep: 7084800
  02:57:06 STEP 1970 H=1970:00 +CPU= 18.654
  02:57:29 STEP 1971 H=1971:00 +CPU= 22.711
  02:57:44 STEP 1972 H=1972:00 +CPU= 14.794
  02:57:58 STEP 1973 H=1973:00 +CPU= 14.290
  02:58:24 STEP 1974 H=1974:00 +CPU= 25.827
  02:58:45 STEP 1975 H=1975:00 +CPU= 21.127
  02:59:07 STEP 1976 H=1976:00 +CPU= 21.854
  02:59:40 STEP 1977 H=1977:00 +CPU= 33.232
  03:00:02 STEP 1978 H=1978:00 +CPU= 21.852
  03:00:23 STEP 1979 H=1979:00 +CPU= 21.179
  03:01:00 STEP 1980 H=1980:00 +CPU= 37.024
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001980
  03:01:21 STEP 1981 H=1981:00 +CPU= 20.835
  03:01:42 STEP 1982 H=1982:00 +CPU= 21.536
  03:02:15 STEP 1983 H=1983:00 +CPU= 32.687
  03:02:37 STEP 1984 H=1984:00 +CPU= 21.664
  03:02:58 STEP 1985 H=1985:00 +CPU= 21.159
  03:03:32 STEP 1986 H=1986:00 +CPU= 33.984
  03:03:53 STEP 1987 H=1987:00 +CPU= 21.093
  03:04:15 STEP 1988 H=1988:00 +CPU= 21.816
  03:04:48 STEP 1989 H=1989:00 +CPU= 33.200
  03:05:10 STEP 1990 H=1990:00 +CPU= 21.780
  03:05:32 STEP 1991 H=1991:00 +CPU= 21.816
  03:06:11 STEP 1992 H=1992:00 +CPU= 39.181
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_82.zip
Uploading the intermediate file: upload_file_82.zip
  03:06:32 STEP 1993 H=1993:00 +CPU= 21.073
  03:06:54 STEP 1994 H=1994:00 +CPU= 21.779
  03:07:18 STEP 1995 H=1995:00 +CPU= 24.772
  03:07:33 STEP 1996 H=1996:00 +CPU= 14.706
  03:07:47 STEP 1997 H=1997:00 +CPU= 14.255
  03:08:11 STEP 1998 H=1998:00 +CPU= 23.245
  03:08:25 STEP 1999 H=1999:00 +CPU= 14.307
  03:08:40 STEP 2000 H=2000:00 +CPU= 14.809
  03:09:02 STEP 2001 H=2001:00 +CPU= 22.790
  03:09:17 STEP 2002 H=2002:00 +CPU= 14.748
  03:09:33 STEP 2003 H=2003:00 +CPU= 15.446
  03:10:06 STEP 2004 H=2004:00 +CPU= 33.105
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002004
  03:10:20 STEP 2005 H=2005:00 +CPU= 14.345
  03:10:35 STEP 2006 H=2006:00 +CPU= 14.748
  03:10:58 STEP 2007 H=2007:00 +CPU= 22.745
  03:11:12 STEP 2008 H=2008:00 +CPU= 14.764
  03:11:27 STEP 2009 H=2009:00 +CPU= 14.298
  03:11:50 STEP 2010 H=2010:00 +CPU= 23.277
  03:12:04 STEP 2011 H=2011:00 +CPU= 14.452
  03:12:19 STEP 2012 H=2012:00 +CPU= 14.733
  03:12:42 STEP 2013 H=2013:00 +CPU= 22.906
  03:12:57 STEP 2014 H=2014:00 +CPU= 14.707
  03:13:12 STEP 2015 H=2015:00 +CPU= 14.707
  03:13:50 STEP 2016 H=2016:00 +CPU= 38.064
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_83.zip
Uploading the intermediate file: upload_file_83.zip
  03:14:11 STEP 2017 H=2017:00 +CPU= 20.817
  03:14:31 STEP 2018 H=2018:00 +CPU= 20.385
  03:14:54 STEP 2019 H=2019:00 +CPU= 22.800
  03:15:09 STEP 2020 H=2020:00 +CPU= 14.783
  03:15:23 STEP 2021 H=2021:00 +CPU= 14.314
  03:15:46 STEP 2022 H=2022:00 +CPU= 23.353
  03:16:01 STEP 2023 H=2023:00 +CPU= 14.352
  03:16:15 STEP 2024 H=2024:00 +CPU= 14.765
  03:16:38 STEP 2025 H=2025:00 +CPU= 22.804
  03:16:53 STEP 2026 H=2026:00 +CPU= 14.762
  03:17:10 STEP 2027 H=2027:00 +CPU= 17.004
  03:17:46 STEP 2028 H=2028:00 +CPU= 35.691
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002028
  03:18:06 STEP 2029 H=2029:00 +CPU= 20.894
  03:18:28 STEP 2030 H=2030:00 +CPU= 21.598
  03:19:01 STEP 2031 H=2031:00 +CPU= 32.839
  03:19:23 STEP 2032 H=2032:00 +CPU= 21.723
  03:19:44 STEP 2033 H=2033:00 +CPU= 21.022
  03:20:08 STEP 2034 H=2034:00 +CPU= 24.121
  03:20:22 STEP 2035 H=2035:00 +CPU= 14.361
  03:20:37 STEP 2036 H=2036:00 +CPU= 14.759
  03:21:00 STEP 2037 H=2037:00 +CPU= 22.819
  03:21:14 STEP 2038 H=2038:00 +CPU= 14.709
  03:21:29 STEP 2039 H=2039:00 +CPU= 14.709
  03:21:56 STEP 2040 H=2040:00 +CPU= 26.855
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_84.zip
Uploading the intermediate file: upload_file_84.zip
  03:22:11 STEP 2041 H=2041:00 +CPU= 14.577
  03:22:26 STEP 2042 H=2042:00 +CPU= 15.098
  03:22:49 STEP 2043 H=2043:00 +CPU= 22.822
  03:23:03 STEP 2044 H=2044:00 +CPU= 14.744
  03:23:21 STEP 2045 H=2045:00 +CPU= 17.710
  03:23:55 STEP 2046 H=2046:00 +CPU= 34.183
  03:24:16 STEP 2047 H=2047:00 +CPU= 21.161
  03:24:38 STEP 2048 H=2048:00 +CPU= 21.979
  03:25:12 STEP 2049 H=2049:00 +CPU= 33.315
  03:25:29 STEP 2050 H=2050:00 +CPU= 17.411
  03:25:43 STEP 2051 H=2051:00 +CPU= 14.304
  03:26:09 STEP 2052 H=2052:00 +CPU= 25.222
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002052
  03:26:23 STEP 2053 H=2053:00 +CPU= 14.367
  03:26:38 STEP 2054 H=2054:00 +CPU= 14.809
  03:27:01 STEP 2055 H=2055:00 +CPU= 23.467
  03:27:22 STEP 2056 H=2056:00 +CPU= 20.365
  03:27:36 STEP 2057 H=2057:00 +CPU= 14.780
  03:28:00 STEP 2058 H=2058:00 +CPU= 23.341
  03:28:14 STEP 2059 H=2059:00 +CPU= 14.307
  03:28:29 STEP 2060 H=2060:00 +CPU= 14.729
  03:28:52 STEP 2061 H=2061:00 +CPU= 22.803
  03:29:06 STEP 2062 H=2062:00 +CPU= 14.752
  03:29:28 STEP 2063 H=2063:00 +CPU= 21.479
  03:30:07 STEP 2064 H=2064:00 +CPU= 39.492
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_85.zip
Uploading the intermediate file: upload_file_85.zip
  03:30:29 STEP 2065 H=2065:00 +CPU= 21.224
  03:30:51 STEP 2066 H=2066:00 +CPU= 21.930
  03:31:16 STEP 2067 H=2067:00 +CPU= 25.817
  03:31:31 STEP 2068 H=2068:00 +CPU= 14.737
  03:31:45 STEP 2069 H=2069:00 +CPU= 14.279
  03:32:09 STEP 2070 H=2070:00 +CPU= 23.409
  03:32:23 STEP 2071 H=2071:00 +CPU= 14.443
  03:32:38 STEP 2072 H=2072:00 +CPU= 14.735
  03:33:01 STEP 2073 H=2073:00 +CPU= 23.145
  03:33:16 STEP 2074 H=2074:00 +CPU= 14.743
  03:33:30 STEP 2075 H=2075:00 +CPU= 14.315
  03:34:04 STEP 2076 H=2076:00 +CPU= 33.417
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002076
  03:34:22 STEP 2077 H=2077:00 +CPU= 18.606
  03:34:42 STEP 2078 H=2078:00 +CPU= 19.496
  03:35:15 STEP 2079 H=2079:00 +CPU= 32.828
  03:35:36 STEP 2080 H=2080:00 +CPU= 21.624
  03:35:57 STEP 2081 H=2081:00 +CPU= 20.885
  03:36:31 STEP 2082 H=2082:00 +CPU= 33.612
  03:36:52 STEP 2083 H=2083:00 +CPU= 21.067
  03:37:13 STEP 2084 H=2084:00 +CPU= 21.098
  03:37:36 STEP 2085 H=2085:00 +CPU= 22.762
  03:37:50 STEP 2086 H=2086:00 +CPU= 14.752
  03:38:05 STEP 2087 H=2087:00 +CPU= 14.736
  03:38:42 STEP 2088 H=2088:00 +CPU= 37.044
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_86.zip
Uploading the intermediate file: upload_file_86.zip
  03:39:03 STEP 2089 H=2089:00 +CPU= 20.761
  03:39:24 STEP 2090 H=2090:00 +CPU= 21.474
  03:39:49 STEP 2091 H=2091:00 +CPU= 24.360
  03:40:03 STEP 2092 H=2092:00 +CPU= 14.735
  03:40:18 STEP 2093 H=2093:00 +CPU= 14.278
  03:40:41 STEP 2094 H=2094:00 +CPU= 23.318
  03:40:55 STEP 2095 H=2095:00 +CPU= 14.319
  03:41:10 STEP 2096 H=2096:00 +CPU= 14.799
  03:41:38 STEP 2097 H=2097:00 +CPU= 27.345
  03:41:59 STEP 2098 H=2098:00 +CPU= 21.488
  03:42:20 STEP 2099 H=2099:00 +CPU= 20.768
  03:42:56 STEP 2100 H=2100:00 +CPU= 36.388
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002100
  03:43:14 STEP 2101 H=2101:00 +CPU= 17.739
  03:43:29 STEP 2102 H=2102:00 +CPU= 14.802
  03:44:00 STEP 2103 H=2103:00 +CPU= 31.688
  03:44:22 STEP 2104 H=2104:00 +CPU= 21.469
  03:44:43 STEP 2105 H=2105:00 +CPU= 20.748
  03:45:17 STEP 2106 H=2106:00 +CPU= 33.962
  03:45:38 STEP 2107 H=2107:00 +CPU= 21.119
  03:46:00 STEP 2108 H=2108:00 +CPU= 21.827
  03:46:33 STEP 2109 H=2109:00 +CPU= 33.125
  03:46:52 STEP 2110 H=2110:00 +CPU= 18.893
  03:47:06 STEP 2111 H=2111:00 +CPU= 14.650
  03:47:45 STEP 2112 H=2112:00 +CPU= 38.653
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_87.zip
Uploading the intermediate file: upload_file_87.zip
  03:48:06 STEP 2113 H=2113:00 +CPU= 21.180
  03:48:28 STEP 2114 H=2114:00 +CPU= 21.882
  03:49:02 STEP 2115 H=2115:00 +CPU= 33.312
  03:49:21 STEP 2116 H=2116:00 +CPU= 19.075
  03:49:35 STEP 2117 H=2117:00 +CPU= 14.329
  03:49:58 STEP 2118 H=2118:00 +CPU= 23.284
  03:50:15 STEP 2119 H=2119:00 +CPU= 16.978
  03:50:37 STEP 2120 H=2120:00 +CPU= 21.565
  03:51:10 STEP 2121 H=2121:00 +CPU= 32.765
  03:51:31 STEP 2122 H=2122:00 +CPU= 21.561
  03:51:52 STEP 2123 H=2123:00 +CPU= 20.826
  03:52:27 STEP 2124 H=2124:00 +CPU= 35.256
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002124
  03:52:48 STEP 2125 H=2125:00 +CPU= 20.826
  03:53:10 STEP 2126 H=2126:00 +CPU= 21.555
  03:53:42 STEP 2127 H=2127:00 +CPU= 32.653
  03:53:57 STEP 2128 H=2128:00 +CPU= 14.740
  03:54:11 STEP 2129 H=2129:00 +CPU= 14.234
  03:54:35 STEP 2130 H=2130:00 +CPU= 23.282
  03:54:53 STEP 2131 H=2131:00 +CPU= 18.679
  03:55:15 STEP 2132 H=2132:00 +CPU= 21.395
  03:55:47 STEP 2133 H=2133:00 +CPU= 32.573
  03:56:08 STEP 2134 H=2134:00 +CPU= 21.151
  03:56:30 STEP 2135 H=2135:00 +CPU= 21.378
  03:57:08 STEP 2136 H=2136:00 +CPU= 38.570
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_88.zip
Uploading the intermediate file: upload_file_88.zip
  03:57:26 STEP 2137 H=2137:00 +CPU= 17.937
  03:57:41 STEP 2138 H=2138:00 +CPU= 14.671
  03:58:04 STEP 2139 H=2139:00 +CPU= 22.691
  03:58:18 STEP 2140 H=2140:00 +CPU= 14.674
  03:58:33 STEP 2141 H=2141:00 +CPU= 14.244
  03:58:56 STEP 2142 H=2142:00 +CPU= 23.589
  03:59:11 STEP 2143 H=2143:00 +CPU= 14.357
  03:59:29 STEP 2144 H=2144:00 +CPU= 18.877
  04:00:02 STEP 2145 H=2145:00 +CPU= 32.677
  04:00:24 STEP 2146 H=2146:00 +CPU= 21.487
  04:00:44 STEP 2147 H=2147:00 +CPU= 20.772
  04:01:21 STEP 2148 H=2148:00 +CPU= 36.921
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002148
  04:01:42 STEP 2149 H=2149:00 +CPU= 21.118
  04:02:04 STEP 2150 H=2150:00 +CPU= 21.846
  04:02:37 STEP 2151 H=2151:00 +CPU= 33.163
  04:02:59 STEP 2152 H=2152:00 +CPU= 21.790
  04:03:20 STEP 2153 H=2153:00 +CPU= 21.107
  04:03:54 STEP 2154 H=2154:00 +CPU= 34.011
  04:04:13 STEP 2155 H=2155:00 +CPU= 18.543
  04:04:35 STEP 2156 H=2156:00 +CPU= 21.775
  04:05:08 STEP 2157 H=2157:00 +CPU= 33.125
  04:05:30 STEP 2158 H=2158:00 +CPU= 21.799
  04:05:51 STEP 2159 H=2159:00 +CPU= 21.765
  04:06:31 STEP 2160 H=2160:00 +CPU= 39.184
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_89.zip
Uploading the intermediate file: upload_file_89.zip
  04:06:52 STEP 2161 H=2161:00 +CPU= 21.044
  04:07:14 STEP 2162 H=2162:00 +CPU= 21.865
  04:07:47 STEP 2163 H=2163:00 +CPU= 33.219
  04:08:09 STEP 2164 H=2164:00 +CPU= 21.779
  04:08:30 STEP 2165 H=2165:00 +CPU= 21.089
  04:09:04 STEP 2166 H=2166:00 +CPU= 34.060
  04:09:22 STEP 2167 H=2167:00 +CPU= 18.725
  04:09:41 STEP 2168 H=2168:00 +CPU= 18.292
  04:10:08 STEP 2169 H=2169:00 +CPU= 27.309
  04:10:23 STEP 2170 H=2170:00 +CPU= 14.732
  04:10:37 STEP 2171 H=2171:00 +CPU= 14.270
  04:11:02 STEP 2172 H=2172:00 +CPU= 25.202
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002172
  04:11:17 STEP 2173 H=2173:00 +CPU= 14.380
  04:11:31 STEP 2174 H=2174:00 +CPU= 14.854
  04:11:54 STEP 2175 H=2175:00 +CPU= 22.795
  04:12:09 STEP 2176 H=2176:00 +CPU= 14.717
  04:12:23 STEP 2177 H=2177:00 +CPU= 14.291
  04:12:47 STEP 2178 H=2178:00 +CPU= 23.284
  04:13:01 STEP 2179 H=2179:00 +CPU= 14.318
  04:13:16 STEP 2180 H=2180:00 +CPU= 14.700
  04:13:38 STEP 2181 H=2181:00 +CPU= 22.725
  04:13:53 STEP 2182 H=2182:00 +CPU= 14.685
  04:14:10 STEP 2183 H=2183:00 +CPU= 16.805
  04:14:49 STEP 2184 H=2184:00 +CPU= 38.870
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_90.zip
Uploading the intermediate file: upload_file_90.zip
  04:15:09 STEP 2185 H=2185:00 +CPU= 20.794
  04:15:31 STEP 2186 H=2186:00 +CPU= 21.521
  04:16:04 STEP 2187 H=2187:00 +CPU= 32.719
  04:16:25 STEP 2188 H=2188:00 +CPU= 21.551
  04:16:46 STEP 2189 H=2189:00 +CPU= 20.823
  04:17:20 STEP 2190 H=2190:00 +CPU= 33.578
  04:17:41 STEP 2191 H=2191:00 +CPU= 20.868
  04:18:02 STEP 2192 H=2192:00 +CPU= 21.559
  04:18:35 STEP 2193 H=2193:00 +CPU= 32.783
  04:18:56 STEP 2194 H=2194:00 +CPU= 21.549
  04:19:13 STEP 2195 H=2195:00 +CPU= 16.565
  04:19:39 STEP 2196 H=2196:00 +CPU= 25.508
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002196
  04:19:53 STEP 2197 H=2197:00 +CPU= 14.301
  04:20:08 STEP 2198 H=2198:00 +CPU= 14.726
  04:20:30 STEP 2199 H=2199:00 +CPU= 22.759
  04:20:45 STEP 2200 H=2200:00 +CPU= 14.709
  04:20:59 STEP 2201 H=2201:00 +CPU= 14.280
  04:21:26 STEP 2202 H=2202:00 +CPU= 27.140
  04:21:47 STEP 2203 H=2203:00 +CPU= 20.803
  04:22:03 STEP 2204 H=2204:00 +CPU= 15.881
  04:22:26 STEP 2205 H=2205:00 +CPU= 22.713
  04:22:41 STEP 2206 H=2206:00 +CPU= 14.688
  04:22:57 STEP 2207 H=2207:00 +CPU= 16.609
  04:23:36 STEP 2208 H=2208:00 +CPU= 38.729
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_91.zip
Uploading the intermediate file: upload_file_91.zip
  04:23:56 STEP 2209 H=2209:00 +CPU= 20.550
Uploading trickle at timestep: 7948800
  04:24:11 STEP 2210 H=2210:00 +CPU= 14.744
  04:24:36 STEP 2211 H=2211:00 +CPU= 25.299
  04:24:51 STEP 2212 H=2212:00 +CPU= 14.686
  04:25:05 STEP 2213 H=2213:00 +CPU= 14.285
  04:25:29 STEP 2214 H=2214:00 +CPU= 23.260
  04:25:43 STEP 2215 H=2215:00 +CPU= 14.293
  04:25:58 STEP 2216 H=2216:00 +CPU= 14.714
  04:26:20 STEP 2217 H=2217:00 +CPU= 22.724
  04:26:38 STEP 2218 H=2218:00 +CPU= 17.764
  04:26:59 STEP 2219 H=2219:00 +CPU= 20.765
  04:27:35 STEP 2220 H=2220:00 +CPU= 36.426
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002220
  04:27:56 STEP 2221 H=2221:00 +CPU= 20.754
  04:28:15 STEP 2222 H=2222:00 +CPU= 18.582
  04:28:37 STEP 2223 H=2223:00 +CPU= 22.674
  04:28:52 STEP 2224 H=2224:00 +CPU= 14.671
  04:29:06 STEP 2225 H=2225:00 +CPU= 14.213
  04:29:30 STEP 2226 H=2226:00 +CPU= 23.215
  04:29:48 STEP 2227 H=2227:00 +CPU= 18.057
  04:30:09 STEP 2228 H=2228:00 +CPU= 21.445
  04:30:42 STEP 2229 H=2229:00 +CPU= 32.623
  04:31:03 STEP 2230 H=2230:00 +CPU= 21.442
  04:31:24 STEP 2231 H=2231:00 +CPU= 20.986
  04:31:53 STEP 2232 H=2232:00 +CPU= 28.474
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_92.zip
Uploading the intermediate file: upload_file_92.zip
  04:32:13 STEP 2233 H=2233:00 +CPU= 20.744
  04:32:30 STEP 2234 H=2234:00 +CPU= 16.806
  04:32:53 STEP 2235 H=2235:00 +CPU= 22.772
  04:33:08 STEP 2236 H=2236:00 +CPU= 14.746
  04:33:22 STEP 2237 H=2237:00 +CPU= 14.425
  04:33:54 STEP 2238 H=2238:00 +CPU= 32.070
  04:34:15 STEP 2239 H=2239:00 +CPU= 20.781
  04:34:37 STEP 2240 H=2240:00 +CPU= 21.511
  04:35:04 STEP 2241 H=2241:00 +CPU= 27.881
  04:35:19 STEP 2242 H=2242:00 +CPU= 14.773
  04:35:34 STEP 2243 H=2243:00 +CPU= 14.382
  04:36:01 STEP 2244 H=2244:00 +CPU= 27.479
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002244
  04:36:22 STEP 2245 H=2245:00 +CPU= 20.823
  04:36:43 STEP 2246 H=2246:00 +CPU= 21.555
  04:37:16 STEP 2247 H=2247:00 +CPU= 32.753
  04:37:32 STEP 2248 H=2248:00 +CPU= 15.449
  04:37:46 STEP 2249 H=2249:00 +CPU= 14.263
  04:38:09 STEP 2250 H=2250:00 +CPU= 23.280
  04:38:24 STEP 2251 H=2251:00 +CPU= 14.622
  04:38:39 STEP 2252 H=2252:00 +CPU= 14.716
  04:39:05 STEP 2253 H=2253:00 +CPU= 26.905
  04:39:27 STEP 2254 H=2254:00 +CPU= 21.452
  04:39:48 STEP 2255 H=2255:00 +CPU= 21.426
  04:40:17 STEP 2256 H=2256:00 +CPU= 28.914
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_93.zip
Uploading the intermediate file: upload_file_93.zip
  04:40:32 STEP 2257 H=2257:00 +CPU= 14.277
  04:40:46 STEP 2258 H=2258:00 +CPU= 14.738
  04:41:09 STEP 2259 H=2259:00 +CPU= 22.706
  04:41:24 STEP 2260 H=2260:00 +CPU= 14.691
  04:41:38 STEP 2261 H=2261:00 +CPU= 14.424
  04:42:01 STEP 2262 H=2262:00 +CPU= 23.251
  04:42:16 STEP 2263 H=2263:00 +CPU= 14.518
  04:42:37 STEP 2264 H=2264:00 +CPU= 21.507
  04:43:10 STEP 2265 H=2265:00 +CPU= 32.738
  04:43:32 STEP 2266 H=2266:00 +CPU= 21.533
  04:43:52 STEP 2267 H=2267:00 +CPU= 20.822
  04:44:29 STEP 2268 H=2268:00 +CPU= 36.517
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002268
  04:44:50 STEP 2269 H=2269:00 +CPU= 20.826
  04:45:11 STEP 2270 H=2270:00 +CPU= 21.521
  04:45:44 STEP 2271 H=2271:00 +CPU= 32.707
  04:46:06 STEP 2272 H=2272:00 +CPU= 21.551
  04:46:27 STEP 2273 H=2273:00 +CPU= 20.844
  04:47:00 STEP 2274 H=2274:00 +CPU= 33.573
  04:47:21 STEP 2275 H=2275:00 +CPU= 20.817
  04:47:42 STEP 2276 H=2276:00 +CPU= 21.471
  04:48:15 STEP 2277 H=2277:00 +CPU= 32.697
  04:48:37 STEP 2278 H=2278:00 +CPU= 21.471
  04:48:58 STEP 2279 H=2279:00 +CPU= 21.420
  04:49:37 STEP 2280 H=2280:00 +CPU= 38.706
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_94.zip
Uploading the intermediate file: upload_file_94.zip
  04:49:58 STEP 2281 H=2281:00 +CPU= 20.768
  04:50:17 STEP 2282 H=2282:00 +CPU= 19.433
  04:50:40 STEP 2283 H=2283:00 +CPU= 22.800
  04:50:59 STEP 2284 H=2284:00 +CPU= 19.211
  04:51:20 STEP 2285 H=2285:00 +CPU= 20.722
  04:51:53 STEP 2286 H=2286:00 +CPU= 33.393
  04:52:14 STEP 2287 H=2287:00 +CPU= 20.697
  04:52:35 STEP 2288 H=2288:00 +CPU= 21.424
  04:53:08 STEP 2289 H=2289:00 +CPU= 32.585
  04:53:29 STEP 2290 H=2290:00 +CPU= 21.453
  04:53:50 STEP 2291 H=2291:00 +CPU= 20.737
  04:54:23 STEP 2292 H=2292:00 +CPU= 33.299
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002292
  04:54:44 STEP 2293 H=2293:00 +CPU= 20.719
  04:55:01 STEP 2294 H=2294:00 +CPU= 17.191
  04:55:24 STEP 2295 H=2295:00 +CPU= 22.731
  04:55:39 STEP 2296 H=2296:00 +CPU= 15.064
  04:55:53 STEP 2297 H=2297:00 +CPU= 14.294
  04:56:17 STEP 2298 H=2298:00 +CPU= 23.213
  04:56:31 STEP 2299 H=2299:00 +CPU= 14.433
  04:56:46 STEP 2300 H=2300:00 +CPU= 14.700
  04:57:09 STEP 2301 H=2301:00 +CPU= 22.873
  04:57:23 STEP 2302 H=2302:00 +CPU= 14.720
  04:57:38 STEP 2303 H=2303:00 +CPU= 14.744
  04:58:05 STEP 2304 H=2304:00 +CPU= 26.864
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_95.zip
Uploading the intermediate file: upload_file_95.zip
  04:58:20 STEP 2305 H=2305:00 +CPU= 14.533
  04:58:34 STEP 2306 H=2306:00 +CPU= 14.705
  04:59:02 STEP 2307 H=2307:00 +CPU= 27.246
  04:59:23 STEP 2308 H=2308:00 +CPU= 21.865
  04:59:45 STEP 2309 H=2309:00 +CPU= 21.183
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1990050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0700
(argv4) batchid: 959
(argv5) wuid: 12176344
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b92c60e39369b5fb6359c39fec487b6e to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0700_1990050100_123_959_12176344.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0700_1990050100_123_959_12176344.zip
ic_ancil_file: ic_ancil_12176344
ifsdata_file: ifsdata_12176344
climate_data_file: clim_data_12176344
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7d3bdf16a9a3bab11c21ca7e01dbb033 to: /var/lib/boinc-client/slots/7/ic_ancil_12176344.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12176344.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f6c8d4228cae0d90c656af698ec8d21a to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12176344.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12176344.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12176344.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12176344.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12176344.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12176344.xml
-- Model is restarting --
Adjusting last_iter, 2310, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12176344.xml
last_cpu_time: 53581
upload_file_number: 96
last_iter: 2303
last_upload: 8294400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095
The child process has been launched with process id: 25019
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ewen:1:1:25019:25019] [20230116:180005:1673920805:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:00:45 STEP 2304 H=2304:00 +CPU= 37.666
  18:01:07 STEP 2305 H=2305:00 +CPU= 21.156
  18:01:28 STEP 2306 H=2306:00 +CPU= 21.811
  18:02:02 STEP 2307 H=2307:00 +CPU= 33.233
  18:02:23 STEP 2308 H=2308:00 +CPU= 21.868
  18:02:45 STEP 2309 H=2309:00 +CPU= 21.130
  18:03:19 STEP 2310 H=2310:00 +CPU= 34.146
  18:03:40 STEP 2311 H=2311:00 +CPU= 21.185
  18:04:02 STEP 2312 H=2312:00 +CPU= 21.918
  18:04:35 STEP 2313 H=2313:00 +CPU= 33.269
  18:04:57 STEP 2314 H=2314:00 +CPU= 21.845
  18:05:18 STEP 2315 H=2315:00 +CPU= 20.768
  18:05:53 STEP 2316 H=2316:00 +CPU= 35.122
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002316
  18:06:08 STEP 2317 H=2317:00 +CPU= 15.213
  18:06:25 STEP 2318 H=2318:00 +CPU= 16.838
  18:06:48 STEP 2319 H=2319:00 +CPU= 23.161
  18:07:08 STEP 2320 H=2320:00 +CPU= 20.204
  18:07:24 STEP 2321 H=2321:00 +CPU= 15.324
  18:07:57 STEP 2322 H=2322:00 +CPU= 33.375
  18:08:16 STEP 2323 H=2323:00 +CPU= 19.243
  18:08:38 STEP 2324 H=2324:00 +CPU= 21.390
  18:09:10 STEP 2325 H=2325:00 +CPU= 32.336
  18:09:29 STEP 2326 H=2326:00 +CPU= 19.200
  18:09:51 STEP 2327 H=2327:00 +CPU= 21.542
  18:10:30 STEP 2328 H=2328:00 +CPU= 38.682
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_96.zip
Uploading the intermediate file: upload_file_96.zip
  18:10:50 STEP 2329 H=2329:00 +CPU= 20.840
  18:11:11 STEP 2330 H=2330:00 +CPU= 20.386
  18:11:44 STEP 2331 H=2331:00 +CPU= 33.291
  18:12:06 STEP 2332 H=2332:00 +CPU= 21.827
  18:12:27 STEP 2333 H=2333:00 +CPU= 21.126
  18:13:00 STEP 2334 H=2334:00 +CPU= 32.584
  18:13:20 STEP 2335 H=2335:00 +CPU= 20.817
  18:13:42 STEP 2336 H=2336:00 +CPU= 21.552
  18:14:15 STEP 2337 H=2337:00 +CPU= 32.549
  18:14:36 STEP 2338 H=2338:00 +CPU= 21.559
  18:14:57 STEP 2339 H=2339:00 +CPU= 20.848
  18:15:33 STEP 2340 H=2340:00 +CPU= 36.496
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002340
  18:15:54 STEP 2341 H=2341:00 +CPU= 20.872
  18:16:16 STEP 2342 H=2342:00 +CPU= 21.559
  18:16:49 STEP 2343 H=2343:00 +CPU= 32.714
  18:17:10 STEP 2344 H=2344:00 +CPU= 21.559
  18:17:31 STEP 2345 H=2345:00 +CPU= 20.822
  18:18:05 STEP 2346 H=2346:00 +CPU= 33.517
  18:18:25 STEP 2347 H=2347:00 +CPU= 20.834
  18:18:47 STEP 2348 H=2348:00 +CPU= 21.536
  18:19:20 STEP 2349 H=2349:00 +CPU= 32.727
  18:19:41 STEP 2350 H=2350:00 +CPU= 21.523
  18:20:03 STEP 2351 H=2351:00 +CPU= 21.507
  18:20:41 STEP 2352 H=2352:00 +CPU= 38.790
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_97.zip
Uploading the intermediate file: upload_file_97.zip
  18:21:02 STEP 2353 H=2353:00 +CPU= 20.826
  18:21:24 STEP 2354 H=2354:00 +CPU= 21.563
  18:21:57 STEP 2355 H=2355:00 +CPU= 32.833
  18:22:18 STEP 2356 H=2356:00 +CPU= 21.581
  18:22:39 STEP 2357 H=2357:00 +CPU= 20.867
  18:23:13 STEP 2358 H=2358:00 +CPU= 33.605
  18:23:28 STEP 2359 H=2359:00 +CPU= 15.701
  18:23:47 STEP 2360 H=2360:00 +CPU= 18.357
  18:24:20 STEP 2361 H=2361:00 +CPU= 32.816
  18:24:41 STEP 2362 H=2362:00 +CPU= 21.599
  18:25:01 STEP 2363 H=2363:00 +CPU= 19.395
  18:25:37 STEP 2364 H=2364:00 +CPU= 36.318
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002364
  18:25:58 STEP 2365 H=2365:00 +CPU= 20.755
  18:26:19 STEP 2366 H=2366:00 +CPU= 21.465
  18:26:52 STEP 2367 H=2367:00 +CPU= 32.574
  18:27:13 STEP 2368 H=2368:00 +CPU= 21.461
  18:27:34 STEP 2369 H=2369:00 +CPU= 20.740
  18:28:05 STEP 2370 H=2370:00 +CPU= 31.402
  18:28:20 STEP 2371 H=2371:00 +CPU= 14.306
  18:28:34 STEP 2372 H=2372:00 +CPU= 14.693
  18:28:57 STEP 2373 H=2373:00 +CPU= 22.703
  18:29:12 STEP 2374 H=2374:00 +CPU= 14.666
  18:29:26 STEP 2375 H=2375:00 +CPU= 14.653
  18:29:55 STEP 2376 H=2376:00 +CPU= 28.434
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_98.zip
Uploading the intermediate file: upload_file_98.zip
  18:30:10 STEP 2377 H=2377:00 +CPU= 14.849
  18:30:24 STEP 2378 H=2378:00 +CPU= 14.672
  18:30:47 STEP 2379 H=2379:00 +CPU= 22.670
  18:31:02 STEP 2380 H=2380:00 +CPU= 14.673
  18:31:16 STEP 2381 H=2381:00 +CPU= 14.224
  18:31:39 STEP 2382 H=2382:00 +CPU= 23.202
  18:31:54 STEP 2383 H=2383:00 +CPU= 14.238
  18:32:08 STEP 2384 H=2384:00 +CPU= 14.686
  18:32:31 STEP 2385 H=2385:00 +CPU= 22.752
  18:32:46 STEP 2386 H=2386:00 +CPU= 14.752
  18:33:00 STEP 2387 H=2387:00 +CPU= 14.285
  18:33:25 STEP 2388 H=2388:00 +CPU= 25.460
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002388
  18:33:40 STEP 2389 H=2389:00 +CPU= 14.314
  18:33:55 STEP 2390 H=2390:00 +CPU= 14.904
  18:34:17 STEP 2391 H=2391:00 +CPU= 22.813
  18:34:32 STEP 2392 H=2392:00 +CPU= 14.716
  18:34:46 STEP 2393 H=2393:00 +CPU= 14.260
  18:35:10 STEP 2394 H=2394:00 +CPU= 23.282
  18:35:26 STEP 2395 H=2395:00 +CPU= 16.299
  18:35:48 STEP 2396 H=2396:00 +CPU= 21.564
  18:36:20 STEP 2397 H=2397:00 +CPU= 32.744
  18:36:42 STEP 2398 H=2398:00 +CPU= 21.515
  18:37:03 STEP 2399 H=2399:00 +CPU= 21.503
  18:37:42 STEP 2400 H=2400:00 +CPU= 38.781
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_99.zip
Uploading the intermediate file: upload_file_99.zip
  18:38:03 STEP 2401 H=2401:00 +CPU= 20.840
  18:38:25 STEP 2402 H=2402:00 +CPU= 21.519
  18:38:58 STEP 2403 H=2403:00 +CPU= 32.785
  18:39:19 STEP 2404 H=2404:00 +CPU= 21.545
  18:39:40 STEP 2405 H=2405:00 +CPU= 21.004
  18:40:14 STEP 2406 H=2406:00 +CPU= 34.085
  18:40:33 STEP 2407 H=2407:00 +CPU= 19.099
  18:40:48 STEP 2408 H=2408:00 +CPU= 14.737
  18:41:11 STEP 2409 H=2409:00 +CPU= 22.764
  18:41:26 STEP 2410 H=2410:00 +CPU= 14.757
  18:41:40 STEP 2411 H=2411:00 +CPU= 14.290
  18:42:05 STEP 2412 H=2412:00 +CPU= 25.199
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002412
  18:42:19 STEP 2413 H=2413:00 +CPU= 14.305
  18:42:37 STEP 2414 H=2414:00 +CPU= 17.958
  18:43:09 STEP 2415 H=2415:00 +CPU= 31.638
  18:43:30 STEP 2416 H=2416:00 +CPU= 21.462
  18:43:51 STEP 2417 H=2417:00 +CPU= 20.756
  18:44:25 STEP 2418 H=2418:00 +CPU= 33.473
  18:44:45 STEP 2419 H=2419:00 +CPU= 20.754
  18:45:07 STEP 2420 H=2420:00 +CPU= 21.475
  18:45:40 STEP 2421 H=2421:00 +CPU= 32.622
  18:46:01 STEP 2422 H=2422:00 +CPU= 21.422
  18:46:22 STEP 2423 H=2423:00 +CPU= 21.411
  18:47:01 STEP 2424 H=2424:00 +CPU= 38.220
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176344/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0700_1990050100_123_959_12176344_0_r1452362095_100.zip
Uploading the intermediate file: upload_file_100.zip
  18:47:20 STEP 2425 H=2425:00 +CPU= 19.563
  18:47:38 STEP 2426 H=2426:00 +CPU= 17.643
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:22:20 1517434 22261092 oifs_43r3_ps_0700_1990050100_123_959_12176344_0 7,948,800 51,406 0.0065
09 May 2023 16:22:14 1517434 22261092 oifs_43r3_ps_0700_1990050100_123_959_12176344_0 7,084,800 46,183 0.0065
09 May 2023 16:22:08 1517434 22261092 oifs_43r3_ps_0700_1990050100_123_959_12176344_0 6,220,800 41,091 0.0066
09 May 2023 16:22:01 1517434 22261092 oifs_43r3_ps_0700_1990050100_123_959_12176344_0 5,184,000 34,228 0.0066
09 May 2023 16:21:54 1517434 22261092 oifs_43r3_ps_0700_1990050100_123_959_12176344_0 4,320,000 28,346 0.0066
09 May 2023 16:21:48 1517434 22261092 oifs_43r3_ps_0700_1990050100_123_959_12176344_0 3,456,000 22,432 0.0065
09 May 2023 16:21:41 1517434 22261092 oifs_43r3_ps_0700_1990050100_123_959_12176344_0 2,592,000 16,728 0.0065
09 May 2023 16:21:28 1517434 22261092 oifs_43r3_ps_0700_1990050100_123_959_12176344_0 864,000 5,350 0.0062


©2024 climateprediction.net