climateprediction.net home page
Task 22315742

Task 22315742

Name oifs_43r3_00py_2019110100_123_993_12214372_0
Workunit 12214372
Created 24 Feb 2023, 14:46:25 UTC
Sent 24 Feb 2023, 15:15:43 UTC
Report deadline 25 Apr 2023, 15:15:43 UTC
Received 25 Feb 2023, 9:41:13 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1515225
Run time 17 hours 35 min 58 sec
CPU time 17 hours 35 min 58 sec
Validate state Invalid
Credit 10,039.68
Device peak FLOPS 4.59 GFLOPS
Application version OpenIFS 43r3 v1.21
x86_64-pc-linux-gnu
Peak working set size 4,397.19 MB
Peak swap size 4,983.13 MB
Peak disk usage 1,265.36 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
c-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_69.zip
Uploading the intermediate file: upload_file_69.zip
  21:56:14 STEP 1681 H=1681:00 +CPU= 25.130
  21:56:40 STEP 1682 H=1682:00 +CPU= 26.069
  21:57:20 STEP 1683 H=1683:00 +CPU= 39.286
  21:57:46 STEP 1684 H=1684:00 +CPU= 26.333
  21:58:11 STEP 1685 H=1685:00 +CPU= 25.036
  21:58:51 STEP 1686 H=1686:00 +CPU= 39.710
  21:59:16 STEP 1687 H=1687:00 +CPU= 25.064
  21:59:42 STEP 1688 H=1688:00 +CPU= 26.405
  22:00:21 STEP 1689 H=1689:00 +CPU= 39.088
  22:00:47 STEP 1690 H=1690:00 +CPU= 25.830
  22:01:12 STEP 1691 H=1691:00 +CPU= 25.008
  22:01:55 STEP 1692 H=1692:00 +CPU= 42.764
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001692
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001692
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001692
  22:02:20 STEP 1693 H=1693:00 +CPU= 25.011
  22:02:46 STEP 1694 H=1694:00 +CPU= 26.084
  22:03:25 STEP 1695 H=1695:00 +CPU= 38.856
  22:03:51 STEP 1696 H=1696:00 +CPU= 25.834
  22:04:16 STEP 1697 H=1697:00 +CPU= 24.853
  22:04:56 STEP 1698 H=1698:00 +CPU= 39.942
  22:05:21 STEP 1699 H=1699:00 +CPU= 25.451
  22:05:47 STEP 1700 H=1700:00 +CPU= 25.698
  22:06:26 STEP 1701 H=1701:00 +CPU= 38.741
  22:06:51 STEP 1702 H=1702:00 +CPU= 25.599
  22:07:18 STEP 1703 H=1703:00 +CPU= 26.183
  22:08:03 STEP 1704 H=1704:00 +CPU= 44.630
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001704
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001704
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_70.zip
Uploading the intermediate file: upload_file_70.zip
  22:08:28 STEP 1705 H=1705:00 +CPU= 24.610
  22:08:54 STEP 1706 H=1706:00 +CPU= 25.846
  22:09:32 STEP 1707 H=1707:00 +CPU= 38.404
  22:09:58 STEP 1708 H=1708:00 +CPU= 25.720
  22:10:23 STEP 1709 H=1709:00 +CPU= 24.761
  22:11:02 STEP 1710 H=1710:00 +CPU= 39.578
  22:11:27 STEP 1711 H=1711:00 +CPU= 24.801
  22:11:53 STEP 1712 H=1712:00 +CPU= 26.086
  22:12:32 STEP 1713 H=1713:00 +CPU= 38.433
  22:12:50 STEP 1714 H=1714:00 +CPU= 18.219
  22:13:08 STEP 1715 H=1715:00 +CPU= 17.986
  22:14:07 STEP 1716 H=1716:00 +CPU= 48.041
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001716
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001716
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001716
  22:14:44 STEP 1717 H=1717:00 +CPU= 30.880
  22:15:16 STEP 1718 H=1718:00 +CPU= 30.145
  22:16:02 STEP 1719 H=1719:00 +CPU= 45.443
  22:16:34 STEP 1720 H=1720:00 +CPU= 32.022
  22:16:53 STEP 1721 H=1721:00 +CPU= 19.550
  22:17:23 STEP 1722 H=1722:00 +CPU= 29.510
  22:17:42 STEP 1723 H=1723:00 +CPU= 19.138
  22:18:02 STEP 1724 H=1724:00 +CPU= 19.858
  22:18:31 STEP 1725 H=1725:00 +CPU= 29.691
  22:18:50 STEP 1726 H=1726:00 +CPU= 19.002
  22:19:09 STEP 1727 H=1727:00 +CPU= 18.247
  22:19:47 STEP 1728 H=1728:00 +CPU= 33.752
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001728
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001728
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_71.zip
Uploading the intermediate file: upload_file_71.zip
  22:20:05 STEP 1729 H=1729:00 +CPU= 17.860
  22:20:24 STEP 1730 H=1730:00 +CPU= 19.121
  22:20:53 STEP 1731 H=1731:00 +CPU= 28.909
  22:21:12 STEP 1732 H=1732:00 +CPU= 18.583
  22:21:30 STEP 1733 H=1733:00 +CPU= 18.321
  22:22:00 STEP 1734 H=1734:00 +CPU= 29.789
  22:22:18 STEP 1735 H=1735:00 +CPU= 18.187
  22:22:36 STEP 1736 H=1736:00 +CPU= 18.399
  22:23:05 STEP 1737 H=1737:00 +CPU= 29.147
  22:23:25 STEP 1738 H=1738:00 +CPU= 19.140
  22:23:42 STEP 1739 H=1739:00 +CPU= 17.566
  22:24:14 STEP 1740 H=1740:00 +CPU= 31.512
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001740
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001740
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001740
  22:24:32 STEP 1741 H=1741:00 +CPU= 18.593
  22:24:50 STEP 1742 H=1742:00 +CPU= 18.200
  22:25:20 STEP 1743 H=1743:00 +CPU= 30.095
  22:25:40 STEP 1744 H=1744:00 +CPU= 19.244
  22:25:57 STEP 1745 H=1745:00 +CPU= 17.732
  22:26:27 STEP 1746 H=1746:00 +CPU= 29.561
  22:26:46 STEP 1747 H=1747:00 +CPU= 18.689
  22:27:04 STEP 1748 H=1748:00 +CPU= 18.581
  22:27:33 STEP 1749 H=1749:00 +CPU= 28.997
  22:27:52 STEP 1750 H=1750:00 +CPU= 19.054
  22:28:11 STEP 1751 H=1751:00 +CPU= 18.371
  22:28:48 STEP 1752 H=1752:00 +CPU= 33.826
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001752
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001752
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_72.zip
Uploading the intermediate file: upload_file_72.zip
  22:29:06 STEP 1753 H=1753:00 +CPU= 18.634
Uploading trickle at timestep: 6307200
  22:29:25 STEP 1754 H=1754:00 +CPU= 18.285
  22:29:54 STEP 1755 H=1755:00 +CPU= 28.990
  22:30:13 STEP 1756 H=1756:00 +CPU= 19.155
  22:30:31 STEP 1757 H=1757:00 +CPU= 17.891
  22:31:00 STEP 1758 H=1758:00 +CPU= 29.589
  22:31:19 STEP 1759 H=1759:00 +CPU= 18.749
  22:31:37 STEP 1760 H=1760:00 +CPU= 18.390
  22:32:07 STEP 1761 H=1761:00 +CPU= 29.176
  22:32:26 STEP 1762 H=1762:00 +CPU= 19.070
  22:32:43 STEP 1763 H=1763:00 +CPU= 17.800
  22:33:15 STEP 1764 H=1764:00 +CPU= 31.502
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001764
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001764
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001764
  22:33:34 STEP 1765 H=1765:00 +CPU= 18.664
  22:33:52 STEP 1766 H=1766:00 +CPU= 18.328
  22:34:20 STEP 1767 H=1767:00 +CPU= 28.451
  22:34:40 STEP 1768 H=1768:00 +CPU= 19.178
  22:34:58 STEP 1769 H=1769:00 +CPU= 18.375
  22:35:27 STEP 1770 H=1770:00 +CPU= 29.115
  22:35:46 STEP 1771 H=1771:00 +CPU= 18.598
  22:36:05 STEP 1772 H=1772:00 +CPU= 18.955
  22:36:33 STEP 1773 H=1773:00 +CPU= 28.200
  22:36:52 STEP 1774 H=1774:00 +CPU= 19.110
  22:37:11 STEP 1775 H=1775:00 +CPU= 19.173
  22:37:49 STEP 1776 H=1776:00 +CPU= 32.973
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001776
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001776
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_73.zip
Uploading the intermediate file: upload_file_73.zip
  22:38:07 STEP 1777 H=1777:00 +CPU= 18.683
  22:38:26 STEP 1778 H=1778:00 +CPU= 18.685
  22:38:54 STEP 1779 H=1779:00 +CPU= 28.414
  22:39:14 STEP 1780 H=1780:00 +CPU= 19.311
  22:39:32 STEP 1781 H=1781:00 +CPU= 18.595
  22:40:01 STEP 1782 H=1782:00 +CPU= 29.122
  22:40:20 STEP 1783 H=1783:00 +CPU= 18.705
  22:40:39 STEP 1784 H=1784:00 +CPU= 19.246
  22:41:08 STEP 1785 H=1785:00 +CPU= 28.389
  22:41:27 STEP 1786 H=1786:00 +CPU= 19.310
  22:41:46 STEP 1787 H=1787:00 +CPU= 18.766
  22:42:17 STEP 1788 H=1788:00 +CPU= 31.162
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001788
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001788
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001788
  22:42:36 STEP 1789 H=1789:00 +CPU= 18.675
  22:42:55 STEP 1790 H=1790:00 +CPU= 19.267
  22:43:24 STEP 1791 H=1791:00 +CPU= 28.561
  22:43:43 STEP 1792 H=1792:00 +CPU= 19.390
  22:44:02 STEP 1793 H=1793:00 +CPU= 18.863
  22:44:31 STEP 1794 H=1794:00 +CPU= 29.061
  22:44:50 STEP 1795 H=1795:00 +CPU= 18.793
  22:45:09 STEP 1796 H=1796:00 +CPU= 19.238
  22:45:38 STEP 1797 H=1797:00 +CPU= 28.732
  22:45:57 STEP 1798 H=1798:00 +CPU= 19.401
  22:46:16 STEP 1799 H=1799:00 +CPU= 19.304
  22:46:51 STEP 1800 H=1800:00 +CPU= 33.001
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001800
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001800
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_74.zip
Uploading the intermediate file: upload_file_74.zip
  22:47:10 STEP 1801 H=1801:00 +CPU= 18.698
  22:47:29 STEP 1802 H=1802:00 +CPU= 19.345
  22:47:58 STEP 1803 H=1803:00 +CPU= 28.738
  22:48:17 STEP 1804 H=1804:00 +CPU= 19.096
  22:48:36 STEP 1805 H=1805:00 +CPU= 18.606
  22:49:05 STEP 1806 H=1806:00 +CPU= 29.000
  22:49:23 STEP 1807 H=1807:00 +CPU= 18.702
  22:49:43 STEP 1808 H=1808:00 +CPU= 19.224
  22:50:11 STEP 1809 H=1809:00 +CPU= 28.364
  22:50:30 STEP 1810 H=1810:00 +CPU= 19.223
  22:50:49 STEP 1811 H=1811:00 +CPU= 18.633
  22:51:20 STEP 1812 H=1812:00 +CPU= 31.133
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001812
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001812
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001812
  22:51:39 STEP 1813 H=1813:00 +CPU= 18.609
  22:51:58 STEP 1814 H=1814:00 +CPU= 19.147
  22:52:26 STEP 1815 H=1815:00 +CPU= 28.284
  22:52:45 STEP 1816 H=1816:00 +CPU= 19.186
  22:53:04 STEP 1817 H=1817:00 +CPU= 18.677
  22:53:33 STEP 1818 H=1818:00 +CPU= 29.054
  22:53:52 STEP 1819 H=1819:00 +CPU= 18.800
  22:54:11 STEP 1820 H=1820:00 +CPU= 19.187
  22:54:39 STEP 1821 H=1821:00 +CPU= 28.243
  22:54:58 STEP 1822 H=1822:00 +CPU= 19.193
  22:55:18 STEP 1823 H=1823:00 +CPU= 19.366
  22:55:52 STEP 1824 H=1824:00 +CPU= 32.849
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001824
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001824
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_75.zip
Uploading the intermediate file: upload_file_75.zip
  22:56:11 STEP 1825 H=1825:00 +CPU= 18.466
  22:56:30 STEP 1826 H=1826:00 +CPU= 19.079
  22:56:58 STEP 1827 H=1827:00 +CPU= 28.348
  22:57:18 STEP 1828 H=1828:00 +CPU= 19.329
  22:57:36 STEP 1829 H=1829:00 +CPU= 18.834
  22:58:06 STEP 1830 H=1830:00 +CPU= 29.130
  22:58:24 STEP 1831 H=1831:00 +CPU= 18.604
  22:58:43 STEP 1832 H=1832:00 +CPU= 19.083
  22:59:12 STEP 1833 H=1833:00 +CPU= 28.377
  22:59:31 STEP 1834 H=1834:00 +CPU= 19.014
  22:59:49 STEP 1835 H=1835:00 +CPU= 18.697
  23:00:20 STEP 1836 H=1836:00 +CPU= 30.693
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001836
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001836
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001836
  23:00:39 STEP 1837 H=1837:00 +CPU= 18.513
  23:00:58 STEP 1838 H=1838:00 +CPU= 19.220
  23:01:27 STEP 1839 H=1839:00 +CPU= 28.809
  23:01:45 STEP 1840 H=1840:00 +CPU= 18.885
  23:02:04 STEP 1841 H=1841:00 +CPU= 18.745
  23:02:34 STEP 1842 H=1842:00 +CPU= 29.509
  23:02:51 STEP 1843 H=1843:00 +CPU= 17.678
  23:03:11 STEP 1844 H=1844:00 +CPU= 19.135
  23:03:40 STEP 1845 H=1845:00 +CPU= 29.019
  23:03:58 STEP 1846 H=1846:00 +CPU= 18.144
  23:04:17 STEP 1847 H=1847:00 +CPU= 19.246
  23:04:52 STEP 1848 H=1848:00 +CPU= 33.342
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001848
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001848
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_76.zip
Uploading the intermediate file: upload_file_76.zip
  23:05:10 STEP 1849 H=1849:00 +CPU= 17.715
  23:05:29 STEP 1850 H=1850:00 +CPU= 19.091
  23:05:58 STEP 1851 H=1851:00 +CPU= 28.991
  23:06:16 STEP 1852 H=1852:00 +CPU= 18.142
  23:06:35 STEP 1853 H=1853:00 +CPU= 18.726
  23:07:04 STEP 1854 H=1854:00 +CPU= 29.511
  23:07:22 STEP 1855 H=1855:00 +CPU= 17.817
  23:07:41 STEP 1856 H=1856:00 +CPU= 19.022
  23:08:10 STEP 1857 H=1857:00 +CPU= 28.858
  23:08:29 STEP 1858 H=1858:00 +CPU= 18.499
  23:08:47 STEP 1859 H=1859:00 +CPU= 18.826
  23:09:19 STEP 1860 H=1860:00 +CPU= 31.888
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001860
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001860
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001860
  23:09:37 STEP 1861 H=1861:00 +CPU= 17.938
  23:09:56 STEP 1862 H=1862:00 +CPU= 19.162
  23:10:26 STEP 1863 H=1863:00 +CPU= 29.124
  23:10:44 STEP 1864 H=1864:00 +CPU= 18.513
  23:11:03 STEP 1865 H=1865:00 +CPU= 18.653
  23:11:32 STEP 1866 H=1866:00 +CPU= 29.443
  23:11:50 STEP 1867 H=1867:00 +CPU= 17.875
  23:12:09 STEP 1868 H=1868:00 +CPU= 19.134
  23:12:38 STEP 1869 H=1869:00 +CPU= 29.230
  23:12:57 STEP 1870 H=1870:00 +CPU= 18.568
  23:13:16 STEP 1871 H=1871:00 +CPU= 18.783
  23:13:51 STEP 1872 H=1872:00 +CPU= 33.683
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001872
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001872
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_77.zip
Uploading the intermediate file: upload_file_77.zip
  23:14:09 STEP 1873 H=1873:00 +CPU= 18.189
  23:14:29 STEP 1874 H=1874:00 +CPU= 19.199
  23:14:58 STEP 1875 H=1875:00 +CPU= 29.099
  23:15:16 STEP 1876 H=1876:00 +CPU= 18.383
  23:15:35 STEP 1877 H=1877:00 +CPU= 18.540
  23:16:04 STEP 1878 H=1878:00 +CPU= 29.490
  23:16:22 STEP 1879 H=1879:00 +CPU= 18.119
  23:16:41 STEP 1880 H=1880:00 +CPU= 18.591
  23:17:10 STEP 1881 H=1881:00 +CPU= 28.819
  23:17:29 STEP 1882 H=1882:00 +CPU= 19.158
  23:17:47 STEP 1883 H=1883:00 +CPU= 17.912
  23:18:19 STEP 1884 H=1884:00 +CPU= 32.016
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001884
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001884
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001884
  23:18:37 STEP 1885 H=1885:00 +CPU= 18.647
  23:18:56 STEP 1886 H=1886:00 +CPU= 18.289
  23:19:25 STEP 1887 H=1887:00 +CPU= 29.074
  23:19:44 STEP 1888 H=1888:00 +CPU= 19.072
  23:20:02 STEP 1889 H=1889:00 +CPU= 17.734
  23:20:31 STEP 1890 H=1890:00 +CPU= 29.573
  23:20:50 STEP 1891 H=1891:00 +CPU= 18.710
  23:21:08 STEP 1892 H=1892:00 +CPU= 18.274
  23:21:37 STEP 1893 H=1893:00 +CPU= 29.171
  23:21:57 STEP 1894 H=1894:00 +CPU= 19.184
  23:22:15 STEP 1895 H=1895:00 +CPU= 18.463
  23:22:51 STEP 1896 H=1896:00 +CPU= 33.890
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001896
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001896
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_78.zip
Uploading the intermediate file: upload_file_78.zip
  23:23:09 STEP 1897 H=1897:00 +CPU= 18.513
  23:23:28 STEP 1898 H=1898:00 +CPU= 18.214
  23:23:57 STEP 1899 H=1899:00 +CPU= 29.260
  23:24:16 STEP 1900 H=1900:00 +CPU= 19.375
  23:24:34 STEP 1901 H=1901:00 +CPU= 17.852
  23:25:04 STEP 1902 H=1902:00 +CPU= 29.769
  23:25:23 STEP 1903 H=1903:00 +CPU= 18.829
  23:25:41 STEP 1904 H=1904:00 +CPU= 18.641
  23:26:11 STEP 1905 H=1905:00 +CPU= 29.211
  23:26:30 STEP 1906 H=1906:00 +CPU= 19.280
  23:26:48 STEP 1907 H=1907:00 +CPU= 17.985
  23:27:20 STEP 1908 H=1908:00 +CPU= 31.950
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001908
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001908
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001908
  23:27:39 STEP 1909 H=1909:00 +CPU= 18.748
  23:27:57 STEP 1910 H=1910:00 +CPU= 18.356
  23:28:26 STEP 1911 H=1911:00 +CPU= 29.289
  23:28:46 STEP 1912 H=1912:00 +CPU= 19.377
  23:29:03 STEP 1913 H=1913:00 +CPU= 17.890
  23:29:33 STEP 1914 H=1914:00 +CPU= 29.780
  23:29:52 STEP 1915 H=1915:00 +CPU= 18.791
  23:30:11 STEP 1916 H=1916:00 +CPU= 18.578
  23:30:40 STEP 1917 H=1917:00 +CPU= 28.900
  23:30:59 STEP 1918 H=1918:00 +CPU= 19.099
  23:31:17 STEP 1919 H=1919:00 +CPU= 18.772
  23:31:53 STEP 1920 H=1920:00 +CPU= 33.616
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001920
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001920
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_79.zip
Uploading the intermediate file: upload_file_79.zip
  23:32:12 STEP 1921 H=1921:00 +CPU= 18.882
  23:32:31 STEP 1922 H=1922:00 +CPU= 18.410
  23:33:00 STEP 1923 H=1923:00 +CPU= 29.271
  23:33:19 STEP 1924 H=1924:00 +CPU= 19.239
  23:33:37 STEP 1925 H=1925:00 +CPU= 18.167
  23:34:07 STEP 1926 H=1926:00 +CPU= 29.620
  23:34:26 STEP 1927 H=1927:00 +CPU= 18.803
  23:34:45 STEP 1928 H=1928:00 +CPU= 18.792
  23:35:13 STEP 1929 H=1929:00 +CPU= 28.767
  23:35:33 STEP 1930 H=1930:00 +CPU= 19.193
  23:35:51 STEP 1931 H=1931:00 +CPU= 18.495
  23:36:22 STEP 1932 H=1932:00 +CPU= 30.985
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001932
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001932
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001932
  23:36:41 STEP 1933 H=1933:00 +CPU= 18.878
  23:37:00 STEP 1934 H=1934:00 +CPU= 19.190
  23:37:29 STEP 1935 H=1935:00 +CPU= 28.551
  23:37:48 STEP 1936 H=1936:00 +CPU= 19.232
  23:38:07 STEP 1937 H=1937:00 +CPU= 18.854
  23:38:36 STEP 1938 H=1938:00 +CPU= 28.936
  23:38:54 STEP 1939 H=1939:00 +CPU= 18.746
  23:39:14 STEP 1940 H=1940:00 +CPU= 19.269
  23:39:42 STEP 1941 H=1941:00 +CPU= 28.481
  23:40:01 STEP 1942 H=1942:00 +CPU= 19.170
  23:40:21 STEP 1943 H=1943:00 +CPU= 19.301
  23:40:55 STEP 1944 H=1944:00 +CPU= 32.941
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001944
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001944
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_80.zip
Uploading the intermediate file: upload_file_80.zip
  23:41:14 STEP 1945 H=1945:00 +CPU= 18.606
  23:41:33 STEP 1946 H=1946:00 +CPU= 19.209
  23:42:02 STEP 1947 H=1947:00 +CPU= 28.319
  23:42:21 STEP 1948 H=1948:00 +CPU= 19.211
  23:42:39 STEP 1949 H=1949:00 +CPU= 18.722
  23:43:09 STEP 1950 H=1950:00 +CPU= 29.278
  23:43:27 STEP 1951 H=1951:00 +CPU= 18.683
  23:43:47 STEP 1952 H=1952:00 +CPU= 19.254
  23:44:15 STEP 1953 H=1953:00 +CPU= 28.701
  23:44:35 STEP 1954 H=1954:00 +CPU= 19.278
  23:44:53 STEP 1955 H=1955:00 +CPU= 18.687
  23:45:24 STEP 1956 H=1956:00 +CPU= 30.999
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001956
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001956
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001956
  23:45:43 STEP 1957 H=1957:00 +CPU= 18.746
  23:46:02 STEP 1958 H=1958:00 +CPU= 19.197
  23:46:31 STEP 1959 H=1959:00 +CPU= 28.688
  23:46:50 STEP 1960 H=1960:00 +CPU= 19.217
  23:47:09 STEP 1961 H=1961:00 +CPU= 18.658
  23:47:38 STEP 1962 H=1962:00 +CPU= 29.057
  23:47:57 STEP 1963 H=1963:00 +CPU= 18.634
  23:48:16 STEP 1964 H=1964:00 +CPU= 19.161
  23:48:44 STEP 1965 H=1965:00 +CPU= 28.420
  23:49:03 STEP 1966 H=1966:00 +CPU= 19.266
  23:49:23 STEP 1967 H=1967:00 +CPU= 19.160
  23:49:58 STEP 1968 H=1968:00 +CPU= 33.140
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001968
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001968
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_81.zip
Uploading the intermediate file: upload_file_81.zip
  23:50:16 STEP 1969 H=1969:00 +CPU= 18.642
  23:50:35 STEP 1970 H=1970:00 +CPU= 19.219
  23:51:04 STEP 1971 H=1971:00 +CPU= 28.639
  23:51:23 STEP 1972 H=1972:00 +CPU= 19.317
  23:51:42 STEP 1973 H=1973:00 +CPU= 18.682
  23:52:11 STEP 1974 H=1974:00 +CPU= 29.059
  23:52:30 STEP 1975 H=1975:00 +CPU= 18.808
  23:52:49 STEP 1976 H=1976:00 +CPU= 19.228
  23:53:18 STEP 1977 H=1977:00 +CPU= 28.480
  23:53:37 STEP 1978 H=1978:00 +CPU= 19.409
  23:53:56 STEP 1979 H=1979:00 +CPU= 18.732
  23:54:27 STEP 1980 H=1980:00 +CPU= 30.905
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001980
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001980
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001980
  23:54:45 STEP 1981 H=1981:00 +CPU= 18.606
  23:55:04 STEP 1982 H=1982:00 +CPU= 19.095
  23:55:33 STEP 1983 H=1983:00 +CPU= 28.584
  23:55:52 STEP 1984 H=1984:00 +CPU= 19.234
  23:56:11 STEP 1985 H=1985:00 +CPU= 18.546
  23:56:40 STEP 1986 H=1986:00 +CPU= 28.929
  23:56:58 STEP 1987 H=1987:00 +CPU= 18.798
  23:57:18 STEP 1988 H=1988:00 +CPU= 19.265
  23:57:46 STEP 1989 H=1989:00 +CPU= 28.151
  23:58:05 STEP 1990 H=1990:00 +CPU= 19.157
  23:58:24 STEP 1991 H=1991:00 +CPU= 19.355
  23:58:59 STEP 1992 H=1992:00 +CPU= 33.098
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001992
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001992
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_82.zip
Uploading the intermediate file: upload_file_82.zip
  23:59:19 STEP 1993 H=1993:00 +CPU= 19.159
Uploading trickle at timestep: 7171200
  23:59:38 STEP 1994 H=1994:00 +CPU= 19.230
  00:00:06 STEP 1995 H=1995:00 +CPU= 28.442
  00:00:25 STEP 1996 H=1996:00 +CPU= 19.206
  00:00:44 STEP 1997 H=1997:00 +CPU= 18.788
  00:01:13 STEP 1998 H=1998:00 +CPU= 28.930
  00:01:32 STEP 1999 H=1999:00 +CPU= 18.705
  00:01:51 STEP 2000 H=2000:00 +CPU= 19.283
  00:02:19 STEP 2001 H=2001:00 +CPU= 28.359
  00:02:38 STEP 2002 H=2002:00 +CPU= 19.017
  00:02:57 STEP 2003 H=2003:00 +CPU= 18.641
  00:03:28 STEP 2004 H=2004:00 +CPU= 30.997
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+002004
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+002004
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+002004
  00:03:47 STEP 2005 H=2005:00 +CPU= 18.628
  00:04:06 STEP 2006 H=2006:00 +CPU= 19.178
  00:04:35 STEP 2007 H=2007:00 +CPU= 28.705
  00:04:53 STEP 2008 H=2008:00 +CPU= 18.595
  00:05:12 STEP 2009 H=2009:00 +CPU= 18.632
  00:05:41 STEP 2010 H=2010:00 +CPU= 29.508
  00:05:59 STEP 2011 H=2011:00 +CPU= 17.942
  00:06:18 STEP 2012 H=2012:00 +CPU= 19.130
  00:06:48 STEP 2013 H=2013:00 +CPU= 29.101
  00:07:06 STEP 2014 H=2014:00 +CPU= 18.291
  00:07:25 STEP 2015 H=2015:00 +CPU= 19.379
  00:08:01 STEP 2016 H=2016:00 +CPU= 33.023
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+002016
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+002016
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_83.zip
Uploading the intermediate file: upload_file_83.zip
  00:08:19 STEP 2017 H=2017:00 +CPU= 18.535
  00:08:38 STEP 2018 H=2018:00 +CPU= 19.014
  00:09:07 STEP 2019 H=2019:00 +CPU= 28.924
  00:09:26 STEP 2020 H=2020:00 +CPU= 18.415
  00:09:44 STEP 2021 H=2021:00 +CPU= 18.721
  00:10:14 STEP 2022 H=2022:00 +CPU= 29.585
  00:10:32 STEP 2023 H=2023:00 +CPU= 17.743
  00:10:51 STEP 2024 H=2024:00 +CPU= 19.448
  00:11:20 STEP 2025 H=2025:00 +CPU= 29.205
  00:11:39 STEP 2026 H=2026:00 +CPU= 18.493
  00:11:58 STEP 2027 H=2027:00 +CPU= 18.807
  00:12:29 STEP 2028 H=2028:00 +CPU= 31.775
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+002028
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+002028
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+002028
  00:12:47 STEP 2029 H=2029:00 +CPU= 17.781
  00:13:06 STEP 2030 H=2030:00 +CPU= 19.239
  00:13:35 STEP 2031 H=2031:00 +CPU= 28.958
  00:13:54 STEP 2032 H=2032:00 +CPU= 18.493
  00:14:13 STEP 2033 H=2033:00 +CPU= 18.707
  00:14:42 STEP 2034 H=2034:00 +CPU= 29.606
  00:15:00 STEP 2035 H=2035:00 +CPU= 17.973
  00:15:20 STEP 2036 H=2036:00 +CPU= 19.468
  00:15:49 STEP 2037 H=2037:00 +CPU= 29.638
  00:16:08 STEP 2038 H=2038:00 +CPU= 18.326
  00:16:27 STEP 2039 H=2039:00 +CPU= 19.340
  00:17:03 STEP 2040 H=2040:00 +CPU= 34.061
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+002040
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+002040
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_84.zip
Uploading the intermediate file: upload_file_84.zip
  00:17:21 STEP 2041 H=2041:00 +CPU= 17.965
  00:17:40 STEP 2042 H=2042:00 +CPU= 19.271
  00:18:09 STEP 2043 H=2043:00 +CPU= 29.335
  00:18:28 STEP 2044 H=2044:00 +CPU= 18.539
  00:18:47 STEP 2045 H=2045:00 +CPU= 18.746
  00:19:17 STEP 2046 H=2046:00 +CPU= 29.962
  00:19:35 STEP 2047 H=2047:00 +CPU= 18.269
  00:19:53 STEP 2048 H=2048:00 +CPU= 18.531
  00:20:25 STEP 2049 H=2049:00 +CPU= 31.671
  00:20:47 STEP 2050 H=2050:00 +CPU= 22.146
  00:21:08 STEP 2051 H=2051:00 +CPU= 20.731
  00:21:45 STEP 2052 H=2052:00 +CPU= 36.705
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+002052
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+002052
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+002052
  00:22:06 STEP 2053 H=2053:00 +CPU= 21.672
  00:22:28 STEP 2054 H=2054:00 +CPU= 21.419
  00:23:00 STEP 2055 H=2055:00 +CPU= 32.236
  00:23:19 STEP 2056 H=2056:00 +CPU= 19.163
  00:23:37 STEP 2057 H=2057:00 +CPU= 17.905
  00:24:07 STEP 2058 H=2058:00 +CPU= 29.673
  00:24:26 STEP 2059 H=2059:00 +CPU= 18.764
  00:24:44 STEP 2060 H=2060:00 +CPU= 18.455
  00:25:13 STEP 2061 H=2061:00 +CPU= 28.991
  00:25:32 STEP 2062 H=2062:00 +CPU= 19.308
  00:25:51 STEP 2063 H=2063:00 +CPU= 18.993
  00:26:27 STEP 2064 H=2064:00 +CPU= 33.429
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+002064
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+002064
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_85.zip
Uploading the intermediate file: upload_file_85.zip
  00:26:45 STEP 2065 H=2065:00 +CPU= 18.852
  00:27:04 STEP 2066 H=2066:00 +CPU= 18.633
  00:27:33 STEP 2067 H=2067:00 +CPU= 28.969
  00:27:54 STEP 2068 H=2068:00 +CPU= 20.503
  00:28:15 STEP 2069 H=2069:00 +CPU= 21.541
  00:28:48 STEP 2070 H=2070:00 +CPU= 32.637
  00:29:10 STEP 2071 H=2071:00 +CPU= 21.812
  00:29:32 STEP 2072 H=2072:00 +CPU= 22.150
  00:30:04 STEP 2073 H=2073:00 +CPU= 32.304
  00:30:26 STEP 2074 H=2074:00 +CPU= 22.149
  00:30:46 STEP 2075 H=2075:00 +CPU= 19.412
  00:31:17 STEP 2076 H=2076:00 +CPU= 31.513
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+002076
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+002076
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+002076
  00:31:36 STEP 2077 H=2077:00 +CPU= 18.829
  00:31:55 STEP 2078 H=2078:00 +CPU= 19.436
  00:32:24 STEP 2079 H=2079:00 +CPU= 28.297
  00:32:43 STEP 2080 H=2080:00 +CPU= 19.328
  00:33:02 STEP 2081 H=2081:00 +CPU= 18.835
  00:33:31 STEP 2082 H=2082:00 +CPU= 28.974
  00:33:50 STEP 2083 H=2083:00 +CPU= 18.766
  00:34:09 STEP 2084 H=2084:00 +CPU= 19.216
  00:34:37 STEP 2085 H=2085:00 +CPU= 28.576
  00:34:57 STEP 2086 H=2086:00 +CPU= 19.316
  00:35:16 STEP 2087 H=2087:00 +CPU= 19.339
  00:35:55 STEP 2088 H=2088:00 +CPU= 37.560
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+002088
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+002088
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_86.zip
Uploading the intermediate file: upload_file_86.zip
  00:36:18 STEP 2089 H=2089:00 +CPU= 22.154
  00:36:40 STEP 2090 H=2090:00 +CPU= 22.485
  00:37:13 STEP 2091 H=2091:00 +CPU= 32.778
  00:37:36 STEP 2092 H=2092:00 +CPU= 22.847
  00:37:57 STEP 2093 H=2093:00 +CPU= 21.471
  00:38:26 STEP 2094 H=2094:00 +CPU= 28.738
  00:38:45 STEP 2095 H=2095:00 +CPU= 18.728
  00:39:04 STEP 2096 H=2096:00 +CPU= 19.217
  00:39:33 STEP 2097 H=2097:00 +CPU= 28.787
  00:39:51 STEP 2098 H=2098:00 +CPU= 18.841
  00:40:10 STEP 2099 H=2099:00 +CPU= 18.836
  00:40:42 STEP 2100 H=2100:00 +CPU= 31.430
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+002100
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+002100
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+002100
  00:41:00 STEP 2101 H=2101:00 +CPU= 18.343
  00:41:20 STEP 2102 H=2102:00 +CPU= 19.598
  00:41:49 STEP 2103 H=2103:00 +CPU= 29.755
  00:42:08 STEP 2104 H=2104:00 +CPU= 18.457
  00:42:27 STEP 2105 H=2105:00 +CPU= 19.055
  00:42:58 STEP 2106 H=2106:00 +CPU= 31.346
  00:43:19 STEP 2107 H=2107:00 +CPU= 20.824
  00:43:41 STEP 2108 H=2108:00 +CPU= 22.287
  00:44:15 STEP 2109 H=2109:00 +CPU= 33.426
  00:44:36 STEP 2110 H=2110:00 +CPU= 21.435
  00:44:59 STEP 2111 H=2111:00 +CPU= 22.611
  00:45:36 STEP 2112 H=2112:00 +CPU= 35.355
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+002112
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+002112
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_87.zip
Uploading the intermediate file: upload_file_87.zip
  00:45:53 STEP 2113 H=2113:00 +CPU= 17.869
  00:46:13 STEP 2114 H=2114:00 +CPU= 19.441
  00:46:42 STEP 2115 H=2115:00 +CPU= 29.355
  00:47:01 STEP 2116 H=2116:00 +CPU= 18.414
  00:47:19 STEP 2117 H=2117:00 +CPU= 18.748
  00:47:49 STEP 2118 H=2118:00 +CPU= 29.802
  00:48:07 STEP 2119 H=2119:00 +CPU= 18.217
  00:48:26 STEP 2120 H=2120:00 +CPU= 19.001
  00:48:56 STEP 2121 H=2121:00 +CPU= 29.374
  00:49:15 STEP 2122 H=2122:00 +CPU= 19.126
  00:49:33 STEP 2123 H=2123:00 +CPU= 18.225
  00:50:05 STEP 2124 H=2124:00 +CPU= 31.963
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+002124
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+002124
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+002124
  00:50:25 STEP 2125 H=2125:00 +CPU= 20.247
  00:50:47 STEP 2126 H=2126:00 +CPU= 21.215
  00:51:20 STEP 2127 H=2127:00 +CPU= 33.711
  00:51:43 STEP 2128 H=2128:00 +CPU= 22.716
  00:52:04 STEP 2129 H=2129:00 +CPU= 20.752
  00:52:38 STEP 2130 H=2130:00 +CPU= 34.282
  00:52:58 STEP 2131 H=2131:00 +CPU= 19.844
  00:53:16 STEP 2132 H=2132:00 +CPU= 18.569
  00:53:46 STEP 2133 H=2133:00 +CPU= 29.482
  00:54:05 STEP 2134 H=2134:00 +CPU= 19.408
  00:54:24 STEP 2135 H=2135:00 +CPU= 18.518
  00:54:59 STEP 2136 H=2136:00 +CPU= 33.978
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+002136
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+002136
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_88.zip
Uploading the intermediate file: upload_file_88.zip
  00:55:18 STEP 2137 H=2137:00 +CPU= 19.046
  00:55:37 STEP 2138 H=2138:00 +CPU= 18.750
  00:56:07 STEP 2139 H=2139:00 +CPU= 29.838
  00:56:27 STEP 2140 H=2140:00 +CPU= 19.634
  00:56:45 STEP 2141 H=2141:00 +CPU= 18.094
  00:57:15 STEP 2142 H=2142:00 +CPU= 30.208
  00:57:34 STEP 2143 H=2143:00 +CPU= 19.022
  00:57:54 STEP 2144 H=2144:00 +CPU= 20.129
  00:58:28 STEP 2145 H=2145:00 +CPU= 33.539
  00:58:51 STEP 2146 H=2146:00 +CPU= 23.782
  00:59:24 STEP 2147 H=2147:00 +CPU= 28.756
  00:59:59 STEP 2148 H=2148:00 +CPU= 34.961
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+002148
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+002148
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+002148
  01:00:18 STEP 2149 H=2149:00 +CPU= 19.536
  01:00:40 STEP 2150 H=2150:00 +CPU= 21.462
  01:01:39 STEP 2151 H=2151:00 +CPU= 42.555
  01:02:08 STEP 2152 H=2152:00 +CPU= 24.422
  01:02:31 STEP 2153 H=2153:00 +CPU= 23.328
  01:03:10 STEP 2154 H=2154:00 +CPU= 38.863
  01:03:31 STEP 2155 H=2155:00 +CPU= 21.233
  01:03:55 STEP 2156 H=2156:00 +CPU= 22.476
  01:04:29 STEP 2157 H=2157:00 +CPU= 34.040
  01:04:56 STEP 2158 H=2158:00 +CPU= 26.939
  01:05:22 STEP 2159 H=2159:00 +CPU= 26.515
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 00py
(argv4) batchid: 993
(argv5) wuid: 12214372
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_67fd427e55e559a176316300f3ea3277 to: /var/lib/boinc-client/slots/7/oifs_43r3_00py_2019110100_123_993_12214372.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_00py_2019110100_123_993_12214372.zip
ic_ancil_file: ic_ancil_12214372
ifsdata_file: ifsdata_12214372
climate_data_file: clim_data_12214372
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ffe39dc3af1f9f4afb1096496d02c0b1 to: /var/lib/boinc-client/slots/7/ic_ancil_12214372.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12214372.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2b2243968649088da0ce85053c7a67b8 to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12214372.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12214372.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12214372.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12214372.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12214372.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12214372.xml
-- Model is restarting --
Adjusting last_iter, 2160, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12214372.xml
last_cpu_time: 63342
upload_file_number: 89
last_iter: 2161
last_upload: 7689600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685
The child process has been launched with process id: 18978
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:18978:18978] [20230225:010713:1677316033:0.001] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMGGhnw4+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMSHhnw4+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214372/ICMUAhnw4+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00py_2019110100_123_993_12214372_0_r1867799685_89.zip
Uploading the intermediate file: upload_file_89.zip
..The child process has been killed with signal: 9
>>> Printing last 70 lines from file: NODE.001_01

  UPDO3CH: IMONTH2=           2  IMT2=           2
  UPDO3CH: IPNCH1=           9  NVCLIS=           8
 ICHF1  1 ILEVF1  60 ILATF1 160
 ICHF2  1 ILEVF2  60 ILATF2 160
 ICHF1  2 ILEVF1  60 ILATF1 160
 ICHF2  2 ILEVF2  60 ILATF2 160
 ICHF1  3 ILEVF1  60 ILATF1 160
 ICHF2  3 ILEVF2  60 ILATF2 160
 ICHF1  4 ILEVF1  60 ILATF1 160
 ICHF2  4 ILEVF2  60 ILATF2 160
 ICHF1  5 ILEVF1  60 ILATF1 160
 ICHF2  5 ILEVF2  60 ILATF2 160
 ICHF1  6 ILEVF1  60 ILATF1 160
 ICHF2  6 ILEVF2  60 ILATF2 160
 ICHF1  7 ILEVF1  60 ILATF1 160
 ICHF2  7 ILEVF2  60 ILATF2 160
 ICHF1  8 ILEVF1  60 ILATF1 160
 ICHF2  8 ILEVF2  60 ILATF2 160
FREE_LUN returns unit  57 to the pool
FREE_LUN returns unit  58 to the pool
  DATE= 2020    1   30
 NORMS AT NSTEP CNT4 2160
 SPECTRAL NORMS -    LOG(PREHYDS)    0.114957609339613E+02     OROGRAPHY      0.646549856393636E+04
 LEV     VORTICITY             DIVERGENCE           TEMPERATURE           KINETIC ENERGY    
 AVE 0.284080616437830E-04 0.115590550414132E-04 0.248646296968745E+03 0.265215817408264E+03
 GPNORM  HUMIDITY            AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.212399714230920E-02 0.999999999972377E-08 0.212721177454037E-01
 GPNORM  LIQUID WATER        AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.348279053926312E-05 0.000000000000000E+00 0.108952899724581E-02
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.121881994594547E-05 0.000000000000000E+00 0.723006952887407E-03
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.252915121464962E-05 0.000000000000000E+00 0.163840210758973E-02
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.645479838097855E-06 0.000000000000000E+00 0.278041130460228E-03
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.625018268197979E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM  OZONE               AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.307723555367513E-05 0.249832975298580E-07 0.164465473795330E-04
 GPNORM TENUNOGW             AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.104941273791279E-05 -.244680670170636E-02 0.959667925045675E-03
 GPNORM TENVNOGW             AVERAGE               MINIMUM               MAXIMUM        
         AVE   -.146203741724758E-06 -.692964870678276E-03 0.244660573791834E-02
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.349176391401789E-05 0.000000000000000E+00 0.156527337054449E-02
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.124145088812800E-05 0.000000000000000E+00 0.856339912173973E-03
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.624032720277059E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.268269717426006E-05 0.000000000000000E+00 0.126127879846472E-02
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.197213667916007E-05 0.000000000000000E+00 0.101222504630830E-02
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.473121540134422E-05 0.000000000000000E+00 0.229098614483853E-02
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.131961259288231E-05 0.000000000000000E+00 0.163794606818235E-02
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.134204802281487E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  2160 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 5.071418935362E-01 1.856115388811E-01 6.448207689312E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.380454538696E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -3.46902946831430E-02 -1.99721502809826E+00  1.88830912221965E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -3.38492344119358E-02 -1.00000000000000E+00  1.00000000000000E+00
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 2 lines from file: ifs.stat
 01:07:13 000000000 CNT3      -999     0.514    0.514    0.515      0:00      0:00 0.00000000000000E+00       1GB       0MB
 01:07:22 0AAA00AAA STEPO     2160     7.999    7.999    9.236      0:08      0:09 0.11559055041413E-04       1GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    2160",
 CTIME   = "00900000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 5*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =   98321.8737566107     ,
 GMASSI  =   98321.4536438319     
 /
------------------------------------------------
>>> Printing last 17 lines from file: waminfo

PARAM=10U/10V,                                                          
REPRES=GG,                                                              
ANALYSIS PERIOD =       0                                               
FORECAST PERIOD =    8294400                                            
WIND TIME STEP IN SECONDS =    3600                                     
ANALYSIS FROM 20191101000000 TO 20191101000000                          
ASSIMILATION NO                                                         
NEW FORECAST NO                                                         
STATUS STORM1= UNFINISHED                                               
STATUS STORM2= UNFINISHED                                               
STATUS STORM3= UNFINISHED                                               
DATE FOR OUTPUT OF BOTH RESTART FILES = 00000000000000                  
LAST DATE FOR SPECTRA FILE OUTPUT = 20200303000000                      
BEGIN DATE FOR USING SURFACE CURRENT = 20191101000000                   
PARALLEL RESTART = YES                                                  
NUMBER OF MPI TASKS USED =          1                                   
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/7/progress_file_12214372.xml

<running_values>
  <last_cpu_time>63349.770000</last_cpu_time>
  <upload_file_number>90</upload_file_number>
  <last_iter>2160</last_iter>
  <last_upload>7779600</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:53:23 1515225 22315742 oifs_43r3_00py_2019110100_123_993_12214372_0 7,171,200 59,368 0.0083
09 May 2023 16:53:20 1515225 22315742 oifs_43r3_00py_2019110100_123_993_12214372_0 6,307,200 53,981 0.0086
09 May 2023 16:53:05 1515225 22315742 oifs_43r3_00py_2019110100_123_993_12214372_0 4,838,400 41,073 0.0085
09 May 2023 16:52:59 1515225 22315742 oifs_43r3_00py_2019110100_123_993_12214372_0 3,974,400 33,595 0.0085
09 May 2023 16:52:54 1515225 22315742 oifs_43r3_00py_2019110100_123_993_12214372_0 3,110,400 26,325 0.0085
09 May 2023 16:52:49 1515225 22315742 oifs_43r3_00py_2019110100_123_993_12214372_0 2,246,400 18,995 0.0085
09 May 2023 16:52:45 1515225 22315742 oifs_43r3_00py_2019110100_123_993_12214372_0 1,382,400 11,676 0.0084


©2024 climateprediction.net