climateprediction.net home page
Task 22304754

Task 22304754

Name oifs_43r3_ps_0586_1999050100_123_968_12185230_2
Workunit 12185230
Created 29 Jan 2023, 23:41:32 UTC
Sent 29 Jan 2023, 23:41:43 UTC
Report deadline 30 Mar 2023, 23:41:43 UTC
Received 30 Jan 2023, 22:56:31 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1537854
Run time 22 hours 14 min 45 sec
CPU time 21 hours 12 min 23 sec
Validate state Invalid
Credit 7,257.60
Device peak FLOPS 3.60 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,403.39 MB
Peak swap size 4,974.20 MB
Peak disk usage 2,217.63 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
t/oifs_43r3_ps_12185230/ICMSHhq0f+000744
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000744
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000756
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000756
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000756
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_31.zip
Uploading the intermediate file: upload_file_31.zip
  13:35:04 STEP  769 H= 769:00 +CPU= 43.164
  13:35:51 STEP  770 H= 770:00 +CPU= 44.485
  13:37:02 STEP  771 H= 771:00 +CPU= 67.760
  13:37:48 STEP  772 H= 772:00 +CPU= 44.261
  13:38:34 STEP  773 H= 773:00 +CPU= 43.001
  13:39:46 STEP  774 H= 774:00 +CPU= 69.373
  13:40:31 STEP  775 H= 775:00 +CPU= 42.965
  13:41:18 STEP  776 H= 776:00 +CPU= 44.522
  13:42:29 STEP  777 H= 777:00 +CPU= 68.053
  13:43:15 STEP  778 H= 778:00 +CPU= 44.564
  13:44:01 STEP  779 H= 779:00 +CPU= 43.315
  13:45:20 STEP  780 H= 780:00 +CPU= 76.279
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000780
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000780
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000780
  13:46:06 STEP  781 H= 781:00 +CPU= 43.139
  13:46:52 STEP  782 H= 782:00 +CPU= 44.546
  13:48:04 STEP  783 H= 783:00 +CPU= 67.919
  13:48:50 STEP  784 H= 784:00 +CPU= 44.636
  13:49:36 STEP  785 H= 785:00 +CPU= 43.262
  13:50:49 STEP  786 H= 786:00 +CPU= 69.308
  13:51:34 STEP  787 H= 787:00 +CPU= 43.049
  13:52:20 STEP  788 H= 788:00 +CPU= 44.472
  13:53:31 STEP  789 H= 789:00 +CPU= 67.875
  13:54:17 STEP  790 H= 790:00 +CPU= 44.209
  13:55:04 STEP  791 H= 791:00 +CPU= 44.584
  13:56:29 STEP  792 H= 792:00 +CPU= 79.585
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000792
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000792
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000792
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000768
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000768
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000768
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000780
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000780
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000780
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_32.zip
Uploading the intermediate file: upload_file_32.zip
  13:57:15 STEP  793 H= 793:00 +CPU= 43.326
  13:58:01 STEP  794 H= 794:00 +CPU= 44.197
  13:59:12 STEP  795 H= 795:00 +CPU= 67.933
  13:59:59 STEP  796 H= 796:00 +CPU= 44.539
  14:00:44 STEP  797 H= 797:00 +CPU= 43.177
  14:01:57 STEP  798 H= 798:00 +CPU= 69.210
  14:02:42 STEP  799 H= 799:00 +CPU= 43.407
  14:03:28 STEP  800 H= 800:00 +CPU= 44.552
  14:04:39 STEP  801 H= 801:00 +CPU= 67.969
  14:05:26 STEP  802 H= 802:00 +CPU= 44.519
  14:06:11 STEP  803 H= 803:00 +CPU= 43.222
  14:07:31 STEP  804 H= 804:00 +CPU= 75.963
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000804
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000804
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000804
  14:08:16 STEP  805 H= 805:00 +CPU= 43.254
  14:09:03 STEP  806 H= 806:00 +CPU= 44.546
  14:10:14 STEP  807 H= 807:00 +CPU= 68.004
  14:11:00 STEP  808 H= 808:00 +CPU= 44.641
  14:11:45 STEP  809 H= 809:00 +CPU= 43.220
  14:12:58 STEP  810 H= 810:00 +CPU= 69.525
  14:13:43 STEP  811 H= 811:00 +CPU= 43.177
  14:14:30 STEP  812 H= 812:00 +CPU= 44.395
  14:15:41 STEP  813 H= 813:00 +CPU= 67.964
  14:16:28 STEP  814 H= 814:00 +CPU= 44.111
  14:17:15 STEP  815 H= 815:00 +CPU= 44.523
  14:18:40 STEP  816 H= 816:00 +CPU= 79.852
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000816
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000816
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000816
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000792
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000792
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000792
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000804
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000804
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000804
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_33.zip
Uploading the intermediate file: upload_file_33.zip
  14:19:25 STEP  817 H= 817:00 +CPU= 43.030
  14:20:12 STEP  818 H= 818:00 +CPU= 44.223
  14:21:23 STEP  819 H= 819:00 +CPU= 67.990
  14:22:09 STEP  820 H= 820:00 +CPU= 44.522
  14:22:55 STEP  821 H= 821:00 +CPU= 43.120
  14:24:07 STEP  822 H= 822:00 +CPU= 69.316
  14:24:52 STEP  823 H= 823:00 +CPU= 43.257
  14:25:39 STEP  824 H= 824:00 +CPU= 44.630
  14:26:50 STEP  825 H= 825:00 +CPU= 68.107
  14:27:37 STEP  826 H= 826:00 +CPU= 44.479
  14:28:22 STEP  827 H= 827:00 +CPU= 43.098
  14:29:41 STEP  828 H= 828:00 +CPU= 76.204
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000828
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000828
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000828
  14:30:27 STEP  829 H= 829:00 +CPU= 43.202
  14:31:13 STEP  830 H= 830:00 +CPU= 44.574
  14:32:25 STEP  831 H= 831:00 +CPU= 67.969
  14:33:11 STEP  832 H= 832:00 +CPU= 44.559
  14:33:56 STEP  833 H= 833:00 +CPU= 43.159
  14:35:09 STEP  834 H= 834:00 +CPU= 69.545
  14:35:54 STEP  835 H= 835:00 +CPU= 43.176
  14:36:41 STEP  836 H= 836:00 +CPU= 44.212
  14:37:52 STEP  837 H= 837:00 +CPU= 68.074
  14:38:39 STEP  838 H= 838:00 +CPU= 44.194
  14:39:26 STEP  839 H= 839:00 +CPU= 44.576
  14:40:52 STEP  840 H= 840:00 +CPU= 79.848
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000840
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000840
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000840
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000816
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000816
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000816
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000828
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000828
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000828
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_34.zip
Uploading the intermediate file: upload_file_34.zip
  14:41:37 STEP  841 H= 841:00 +CPU= 43.033
  14:42:23 STEP  842 H= 842:00 +CPU= 44.417
  14:43:34 STEP  843 H= 843:00 +CPU= 68.184
  14:44:21 STEP  844 H= 844:00 +CPU= 44.468
  14:45:06 STEP  845 H= 845:00 +CPU= 43.200
  14:46:19 STEP  846 H= 846:00 +CPU= 69.641
  14:47:04 STEP  847 H= 847:00 +CPU= 43.318
  14:47:51 STEP  848 H= 848:00 +CPU= 44.615
  14:49:02 STEP  849 H= 849:00 +CPU= 68.225
  14:49:49 STEP  850 H= 850:00 +CPU= 44.568
  14:50:34 STEP  851 H= 851:00 +CPU= 43.278
  14:51:54 STEP  852 H= 852:00 +CPU= 76.256
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000852
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000852
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000852
  14:52:40 STEP  853 H= 853:00 +CPU= 43.206
  14:53:26 STEP  854 H= 854:00 +CPU= 44.526
  14:54:38 STEP  855 H= 855:00 +CPU= 68.118
  14:55:25 STEP  856 H= 856:00 +CPU= 44.617
  14:56:10 STEP  857 H= 857:00 +CPU= 42.947
  14:57:23 STEP  858 H= 858:00 +CPU= 69.711
  14:58:08 STEP  859 H= 859:00 +CPU= 42.938
  14:58:54 STEP  860 H= 860:00 +CPU= 44.474
  15:00:05 STEP  861 H= 861:00 +CPU= 67.974
  15:00:52 STEP  862 H= 862:00 +CPU= 44.393
  15:01:38 STEP  863 H= 863:00 +CPU= 44.578
  15:03:04 STEP  864 H= 864:00 +CPU= 80.293
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000864
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000864
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000864
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000840
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000840
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000840
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000852
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000852
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000852
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_35.zip
Uploading the intermediate file: upload_file_35.zip
  15:03:49 STEP  865 H= 865:00 +CPU= 43.095
  15:04:35 STEP  866 H= 866:00 +CPU= 44.373
  15:05:47 STEP  867 H= 867:00 +CPU= 68.126
  15:06:33 STEP  868 H= 868:00 +CPU= 44.493
  15:07:19 STEP  869 H= 869:00 +CPU= 43.280
  15:08:32 STEP  870 H= 870:00 +CPU= 69.339
  15:09:17 STEP  871 H= 871:00 +CPU= 43.332
  15:10:04 STEP  872 H= 872:00 +CPU= 44.445
  15:11:15 STEP  873 H= 873:00 +CPU= 68.103
  15:12:01 STEP  874 H= 874:00 +CPU= 44.560
  15:12:47 STEP  875 H= 875:00 +CPU= 43.373
  15:14:07 STEP  876 H= 876:00 +CPU= 76.209
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000876
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000876
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000876
  15:14:52 STEP  877 H= 877:00 +CPU= 43.423
  15:15:39 STEP  878 H= 878:00 +CPU= 44.524
  15:16:51 STEP  879 H= 879:00 +CPU= 67.879
  15:17:37 STEP  880 H= 880:00 +CPU= 44.511
  15:18:23 STEP  881 H= 881:00 +CPU= 42.938
  15:19:36 STEP  882 H= 882:00 +CPU= 69.510
  15:20:22 STEP  883 H= 883:00 +CPU= 42.840
  15:21:08 STEP  884 H= 884:00 +CPU= 44.511
  15:22:20 STEP  885 H= 885:00 +CPU= 68.162
  15:23:06 STEP  886 H= 886:00 +CPU= 44.554
  15:23:53 STEP  887 H= 887:00 +CPU= 44.611
  15:25:19 STEP  888 H= 888:00 +CPU= 80.340
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000888
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000888
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000888
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000864
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000864
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000864
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000876
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000876
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000876
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_36.zip
Uploading the intermediate file: upload_file_36.zip
  15:26:04 STEP  889 H= 889:00 +CPU= 43.018
  15:26:51 STEP  890 H= 890:00 +CPU= 44.514
  15:28:02 STEP  891 H= 891:00 +CPU= 67.950
  15:28:48 STEP  892 H= 892:00 +CPU= 44.500
  15:29:33 STEP  893 H= 893:00 +CPU= 43.115
  15:30:46 STEP  894 H= 894:00 +CPU= 69.580
  15:31:31 STEP  895 H= 895:00 +CPU= 43.328
  15:32:18 STEP  896 H= 896:00 +CPU= 44.477
  15:33:29 STEP  897 H= 897:00 +CPU= 68.175
  15:34:16 STEP  898 H= 898:00 +CPU= 44.400
  15:35:01 STEP  899 H= 899:00 +CPU= 43.209
  15:36:21 STEP  900 H= 900:00 +CPU= 75.854
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000900
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000900
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000900
  15:37:06 STEP  901 H= 901:00 +CPU= 43.322
  15:37:53 STEP  902 H= 902:00 +CPU= 44.643
  15:39:05 STEP  903 H= 903:00 +CPU= 68.119
  15:39:51 STEP  904 H= 904:00 +CPU= 44.435
  15:40:37 STEP  905 H= 905:00 +CPU= 43.050
  15:41:50 STEP  906 H= 906:00 +CPU= 69.482
  15:42:35 STEP  907 H= 907:00 +CPU= 43.327
  15:43:21 STEP  908 H= 908:00 +CPU= 44.454
  15:44:33 STEP  909 H= 909:00 +CPU= 68.129
  15:45:19 STEP  910 H= 910:00 +CPU= 44.424
  15:46:06 STEP  911 H= 911:00 +CPU= 44.439
  15:47:31 STEP  912 H= 912:00 +CPU= 80.183
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000912
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000912
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000912
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000888
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000888
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000888
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000900
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000900
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000900
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_37.zip
Uploading the intermediate file: upload_file_37.zip
  15:48:17 STEP  913 H= 913:00 +CPU= 43.210
  15:49:03 STEP  914 H= 914:00 +CPU= 44.431
  15:50:14 STEP  915 H= 915:00 +CPU= 67.931
  15:51:01 STEP  916 H= 916:00 +CPU= 44.468
  15:51:46 STEP  917 H= 917:00 +CPU= 43.321
  15:52:58 STEP  918 H= 918:00 +CPU= 69.344
  15:53:44 STEP  919 H= 919:00 +CPU= 43.198
  15:54:30 STEP  920 H= 920:00 +CPU= 44.356
  15:55:42 STEP  921 H= 921:00 +CPU= 68.249
  15:56:28 STEP  922 H= 922:00 +CPU= 44.127
  15:57:14 STEP  923 H= 923:00 +CPU= 43.271
  15:58:34 STEP  924 H= 924:00 +CPU= 76.055
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000924
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000924
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000924
  15:59:19 STEP  925 H= 925:00 +CPU= 43.432
  16:00:06 STEP  926 H= 926:00 +CPU= 44.304
  16:01:17 STEP  927 H= 927:00 +CPU= 68.158
  16:02:04 STEP  928 H= 928:00 +CPU= 44.491
  16:02:49 STEP  929 H= 929:00 +CPU= 43.212
  16:04:02 STEP  930 H= 930:00 +CPU= 69.615
  16:04:48 STEP  931 H= 931:00 +CPU= 43.263
  16:05:34 STEP  932 H= 932:00 +CPU= 44.336
  16:06:45 STEP  933 H= 933:00 +CPU= 68.076
  16:07:31 STEP  934 H= 934:00 +CPU= 44.313
  16:08:18 STEP  935 H= 935:00 +CPU= 44.484
  16:09:44 STEP  936 H= 936:00 +CPU= 80.351
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000936
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000936
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000936
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000912
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000912
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000912
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000924
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000924
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000924
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_38.zip
Uploading the intermediate file: upload_file_38.zip
  16:10:29 STEP  937 H= 937:00 +CPU= 42.944
  16:11:15 STEP  938 H= 938:00 +CPU= 44.447
  16:12:27 STEP  939 H= 939:00 +CPU= 68.073
  16:13:13 STEP  940 H= 940:00 +CPU= 44.471
  16:13:58 STEP  941 H= 941:00 +CPU= 43.147
  16:15:11 STEP  942 H= 942:00 +CPU= 69.316
  16:15:57 STEP  943 H= 943:00 +CPU= 43.209
  16:16:44 STEP  944 H= 944:00 +CPU= 44.463
  16:17:55 STEP  945 H= 945:00 +CPU= 68.013
  16:18:42 STEP  946 H= 946:00 +CPU= 44.427
  16:19:27 STEP  947 H= 947:00 +CPU= 43.348
  16:20:47 STEP  948 H= 948:00 +CPU= 76.089
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000948
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000948
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000948
  16:21:32 STEP  949 H= 949:00 +CPU= 43.273
  16:22:19 STEP  950 H= 950:00 +CPU= 44.499
  16:23:30 STEP  951 H= 951:00 +CPU= 68.250
  16:24:17 STEP  952 H= 952:00 +CPU= 44.452
  16:25:02 STEP  953 H= 953:00 +CPU= 43.195
  16:26:15 STEP  954 H= 954:00 +CPU= 69.697
  16:27:00 STEP  955 H= 955:00 +CPU= 43.162
  16:27:46 STEP  956 H= 956:00 +CPU= 44.465
  16:28:57 STEP  957 H= 957:00 +CPU= 67.425
  16:29:43 STEP  958 H= 958:00 +CPU= 44.380
  16:30:30 STEP  959 H= 959:00 +CPU= 44.580
  16:31:56 STEP  960 H= 960:00 +CPU= 80.021
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000960
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000960
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000960
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000936
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000936
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000936
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000948
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000948
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000948
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_39.zip
Uploading the intermediate file: upload_file_39.zip
Uploading trickle at timestep: 3456000
  16:32:41 STEP  961 H= 961:00 +CPU= 43.117
  16:33:27 STEP  962 H= 962:00 +CPU= 44.535
  16:34:39 STEP  963 H= 963:00 +CPU= 67.926
  16:35:25 STEP  964 H= 964:00 +CPU= 44.346
  16:36:11 STEP  965 H= 965:00 +CPU= 42.877
  16:37:23 STEP  966 H= 966:00 +CPU= 69.513
  16:38:09 STEP  967 H= 967:00 +CPU= 43.009
  16:38:55 STEP  968 H= 968:00 +CPU= 44.512
  16:40:07 STEP  969 H= 969:00 +CPU= 67.996
  16:40:53 STEP  970 H= 970:00 +CPU= 44.559
  16:41:38 STEP  971 H= 971:00 +CPU= 43.159
  16:42:58 STEP  972 H= 972:00 +CPU= 76.126
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000972
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000972
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000972
  16:43:43 STEP  973 H= 973:00 +CPU= 43.232
  16:44:30 STEP  974 H= 974:00 +CPU= 44.463
  16:45:41 STEP  975 H= 975:00 +CPU= 67.998
  16:46:27 STEP  976 H= 976:00 +CPU= 44.455
  16:47:13 STEP  977 H= 977:00 +CPU= 43.220
  16:48:25 STEP  978 H= 978:00 +CPU= 69.488
  16:49:10 STEP  979 H= 979:00 +CPU= 43.140
  16:49:57 STEP  980 H= 980:00 +CPU= 44.509
  16:51:08 STEP  981 H= 981:00 +CPU= 68.064
  16:51:55 STEP  982 H= 982:00 +CPU= 44.385
  16:52:41 STEP  983 H= 983:00 +CPU= 44.609
  16:54:07 STEP  984 H= 984:00 +CPU= 79.964
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000984
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000984
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000984
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000960
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000960
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000960
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000972
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000972
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000972
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_40.zip
Uploading the intermediate file: upload_file_40.zip
  16:54:52 STEP  985 H= 985:00 +CPU= 43.174
  16:55:39 STEP  986 H= 986:00 +CPU= 44.495
  16:56:50 STEP  987 H= 987:00 +CPU= 67.860
  16:57:37 STEP  988 H= 988:00 +CPU= 44.593
  16:58:22 STEP  989 H= 989:00 +CPU= 43.255
  16:59:35 STEP  990 H= 990:00 +CPU= 69.321
  17:00:20 STEP  991 H= 991:00 +CPU= 43.096
  17:01:07 STEP  992 H= 992:00 +CPU= 44.597
  17:02:18 STEP  993 H= 993:00 +CPU= 68.036
  17:03:05 STEP  994 H= 994:00 +CPU= 44.604
  17:03:50 STEP  995 H= 995:00 +CPU= 43.243
  17:05:09 STEP  996 H= 996:00 +CPU= 76.107
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000996
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000996
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000996
  17:05:55 STEP  997 H= 997:00 +CPU= 43.251
  17:06:41 STEP  998 H= 998:00 +CPU= 44.429
  17:07:53 STEP  999 H= 999:00 +CPU= 68.274
  17:08:39 STEP 1000 H=1000:00 +CPU= 44.486
  17:09:24 STEP 1001 H=1001:00 +CPU= 43.114
  17:10:37 STEP 1002 H=1002:00 +CPU= 69.336
  17:11:22 STEP 1003 H=1003:00 +CPU= 43.243
  17:12:09 STEP 1004 H=1004:00 +CPU= 44.381
  17:13:20 STEP 1005 H=1005:00 +CPU= 68.080
  17:14:06 STEP 1006 H=1006:00 +CPU= 44.284
  17:14:53 STEP 1007 H=1007:00 +CPU= 44.653
  17:16:18 STEP 1008 H=1008:00 +CPU= 80.079
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001008
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001008
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001008
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000984
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000984
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000984
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+000996
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+000996
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+000996
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_41.zip
Uploading the intermediate file: upload_file_41.zip
  17:17:03 STEP 1009 H=1009:00 +CPU= 43.062
  17:17:50 STEP 1010 H=1010:00 +CPU= 44.341
  17:19:01 STEP 1011 H=1011:00 +CPU= 68.060
  17:19:48 STEP 1012 H=1012:00 +CPU= 44.628
  17:20:33 STEP 1013 H=1013:00 +CPU= 43.200
  17:21:46 STEP 1014 H=1014:00 +CPU= 69.602
  17:22:31 STEP 1015 H=1015:00 +CPU= 43.410
  17:23:18 STEP 1016 H=1016:00 +CPU= 44.537
  17:24:29 STEP 1017 H=1017:00 +CPU= 68.073
  17:25:16 STEP 1018 H=1018:00 +CPU= 44.581
  17:26:01 STEP 1019 H=1019:00 +CPU= 43.381
  17:27:20 STEP 1020 H=1020:00 +CPU= 76.055
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001020
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001020
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001020
  17:28:05 STEP 1021 H=1021:00 +CPU= 43.096
  17:28:52 STEP 1022 H=1022:00 +CPU= 44.569
  17:30:03 STEP 1023 H=1023:00 +CPU= 68.114
  17:30:50 STEP 1024 H=1024:00 +CPU= 44.589
  17:31:35 STEP 1025 H=1025:00 +CPU= 43.189
  17:32:47 STEP 1026 H=1026:00 +CPU= 69.482
  17:33:32 STEP 1027 H=1027:00 +CPU= 43.192
  17:34:19 STEP 1028 H=1028:00 +CPU= 44.299
  17:35:30 STEP 1029 H=1029:00 +CPU= 68.112
  17:36:17 STEP 1030 H=1030:00 +CPU= 44.450
  17:37:04 STEP 1031 H=1031:00 +CPU= 44.734
  17:38:29 STEP 1032 H=1032:00 +CPU= 80.096
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001032
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001032
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001032
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001008
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001008
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001008
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001020
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001020
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001020
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_42.zip
Uploading the intermediate file: upload_file_42.zip
  17:39:15 STEP 1033 H=1033:00 +CPU= 43.145
  17:40:01 STEP 1034 H=1034:00 +CPU= 44.378
  17:41:12 STEP 1035 H=1035:00 +CPU= 68.001
  17:41:59 STEP 1036 H=1036:00 +CPU= 44.479
  17:42:44 STEP 1037 H=1037:00 +CPU= 43.215
  17:43:56 STEP 1038 H=1038:00 +CPU= 69.383
  17:44:41 STEP 1039 H=1039:00 +CPU= 43.291
  17:45:28 STEP 1040 H=1040:00 +CPU= 44.539
  17:46:39 STEP 1041 H=1041:00 +CPU= 68.050
  17:47:26 STEP 1042 H=1042:00 +CPU= 44.540
  17:48:11 STEP 1043 H=1043:00 +CPU= 43.271
  17:49:31 STEP 1044 H=1044:00 +CPU= 76.156
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001044
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001044
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001044
  17:50:16 STEP 1045 H=1045:00 +CPU= 43.346
  17:51:03 STEP 1046 H=1046:00 +CPU= 44.536
  17:52:14 STEP 1047 H=1047:00 +CPU= 67.923
  17:53:00 STEP 1048 H=1048:00 +CPU= 44.452
  17:53:45 STEP 1049 H=1049:00 +CPU= 43.181
  17:54:58 STEP 1050 H=1050:00 +CPU= 69.414
  17:55:43 STEP 1051 H=1051:00 +CPU= 43.180
  17:56:29 STEP 1052 H=1052:00 +CPU= 44.385
  17:57:40 STEP 1053 H=1053:00 +CPU= 68.043
  17:58:27 STEP 1054 H=1054:00 +CPU= 44.341
  17:59:14 STEP 1055 H=1055:00 +CPU= 44.745
  18:00:39 STEP 1056 H=1056:00 +CPU= 79.970
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001056
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001056
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001056
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001032
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001032
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001032
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001044
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001044
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001044
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_43.zip
Uploading the intermediate file: upload_file_43.zip
  18:01:24 STEP 1057 H=1057:00 +CPU= 43.065
  18:02:11 STEP 1058 H=1058:00 +CPU= 44.387
  18:03:22 STEP 1059 H=1059:00 +CPU= 67.813
  18:04:08 STEP 1060 H=1060:00 +CPU= 44.353
  18:04:53 STEP 1061 H=1061:00 +CPU= 43.277
  18:06:06 STEP 1062 H=1062:00 +CPU= 69.525
  18:06:51 STEP 1063 H=1063:00 +CPU= 43.268
  18:07:38 STEP 1064 H=1064:00 +CPU= 44.524
  18:08:49 STEP 1065 H=1065:00 +CPU= 68.155
  18:09:35 STEP 1066 H=1066:00 +CPU= 44.500
  18:10:21 STEP 1067 H=1067:00 +CPU= 43.264
  18:11:40 STEP 1068 H=1068:00 +CPU= 76.152
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001068
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001068
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001068
  18:12:25 STEP 1069 H=1069:00 +CPU= 43.298
  18:13:12 STEP 1070 H=1070:00 +CPU= 44.492
  18:14:23 STEP 1071 H=1071:00 +CPU= 67.829
  18:15:09 STEP 1072 H=1072:00 +CPU= 44.384
  18:15:54 STEP 1073 H=1073:00 +CPU= 43.157
  18:17:07 STEP 1074 H=1074:00 +CPU= 69.495
  18:17:52 STEP 1075 H=1075:00 +CPU= 43.254
  18:18:39 STEP 1076 H=1076:00 +CPU= 44.441
  18:19:50 STEP 1077 H=1077:00 +CPU= 68.067
  18:20:36 STEP 1078 H=1078:00 +CPU= 44.389
  18:21:23 STEP 1079 H=1079:00 +CPU= 44.569
  18:22:48 STEP 1080 H=1080:00 +CPU= 80.106
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001080
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001080
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001080
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001056
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001056
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001056
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001068
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001068
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001068
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_44.zip
Uploading the intermediate file: upload_file_44.zip
  18:23:33 STEP 1081 H=1081:00 +CPU= 43.215
  18:24:20 STEP 1082 H=1082:00 +CPU= 44.534
  18:25:31 STEP 1083 H=1083:00 +CPU= 68.075
  18:26:17 STEP 1084 H=1084:00 +CPU= 44.472
  18:27:02 STEP 1085 H=1085:00 +CPU= 43.215
  18:28:15 STEP 1086 H=1086:00 +CPU= 69.526
  18:29:00 STEP 1087 H=1087:00 +CPU= 43.315
  18:29:47 STEP 1088 H=1088:00 +CPU= 44.532
  18:30:58 STEP 1089 H=1089:00 +CPU= 68.087
  18:31:44 STEP 1090 H=1090:00 +CPU= 44.561
  18:32:30 STEP 1091 H=1091:00 +CPU= 43.236
  18:33:49 STEP 1092 H=1092:00 +CPU= 76.050
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001092
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001092
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001092
  18:34:34 STEP 1093 H=1093:00 +CPU= 43.245
  18:35:21 STEP 1094 H=1094:00 +CPU= 44.423
  18:36:32 STEP 1095 H=1095:00 +CPU= 68.157
  18:37:19 STEP 1096 H=1096:00 +CPU= 44.540
  18:38:04 STEP 1097 H=1097:00 +CPU= 43.071
  18:39:16 STEP 1098 H=1098:00 +CPU= 69.525
  18:40:02 STEP 1099 H=1099:00 +CPU= 43.210
  18:40:48 STEP 1100 H=1100:00 +CPU= 44.371
  18:41:59 STEP 1101 H=1101:00 +CPU= 68.016
  18:42:45 STEP 1102 H=1102:00 +CPU= 44.412
  18:43:32 STEP 1103 H=1103:00 +CPU= 44.525
  18:44:57 STEP 1104 H=1104:00 +CPU= 80.135
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001104
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001104
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001104
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001080
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001080
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001080
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001092
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001092
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001092
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_45.zip
Uploading the intermediate file: upload_file_45.zip
  18:45:42 STEP 1105 H=1105:00 +CPU= 43.212
  18:46:29 STEP 1106 H=1106:00 +CPU= 44.310
  18:47:40 STEP 1107 H=1107:00 +CPU= 68.126
  18:48:26 STEP 1108 H=1108:00 +CPU= 44.461
  18:49:12 STEP 1109 H=1109:00 +CPU= 43.384
  18:50:24 STEP 1110 H=1110:00 +CPU= 69.342
  18:51:09 STEP 1111 H=1111:00 +CPU= 43.328
  18:51:56 STEP 1112 H=1112:00 +CPU= 44.465
  18:53:07 STEP 1113 H=1113:00 +CPU= 68.196
  18:53:53 STEP 1114 H=1114:00 +CPU= 44.424
  18:54:39 STEP 1115 H=1115:00 +CPU= 43.355
  18:55:58 STEP 1116 H=1116:00 +CPU= 76.131
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001116
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001116
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001116
  18:56:44 STEP 1117 H=1117:00 +CPU= 43.271
  18:57:30 STEP 1118 H=1118:00 +CPU= 44.441
  18:58:41 STEP 1119 H=1119:00 +CPU= 68.037
  18:59:28 STEP 1120 H=1120:00 +CPU= 44.462
  19:00:13 STEP 1121 H=1121:00 +CPU= 43.200
  19:01:26 STEP 1122 H=1122:00 +CPU= 69.437
  19:02:11 STEP 1123 H=1123:00 +CPU= 43.257
  19:02:57 STEP 1124 H=1124:00 +CPU= 44.409
  19:04:08 STEP 1125 H=1125:00 +CPU= 67.886
  19:04:55 STEP 1126 H=1126:00 +CPU= 44.451
  19:05:42 STEP 1127 H=1127:00 +CPU= 44.757
  19:07:07 STEP 1128 H=1128:00 +CPU= 79.929
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001128
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001128
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001128
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001104
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001104
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001104
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001116
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001116
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001116
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_46.zip
Uploading the intermediate file: upload_file_46.zip
  19:07:52 STEP 1129 H=1129:00 +CPU= 43.101
  19:08:38 STEP 1130 H=1130:00 +CPU= 44.457
  19:09:49 STEP 1131 H=1131:00 +CPU= 67.967
  19:10:36 STEP 1132 H=1132:00 +CPU= 44.547
  19:11:21 STEP 1133 H=1133:00 +CPU= 43.157
  19:12:34 STEP 1134 H=1134:00 +CPU= 69.512
  19:13:19 STEP 1135 H=1135:00 +CPU= 43.272
  19:14:06 STEP 1136 H=1136:00 +CPU= 44.592
  19:15:17 STEP 1137 H=1137:00 +CPU= 68.081
  19:16:04 STEP 1138 H=1138:00 +CPU= 44.569
  19:16:49 STEP 1139 H=1139:00 +CPU= 43.311
  19:18:09 STEP 1140 H=1140:00 +CPU= 76.000
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001140
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001140
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001140
  19:18:54 STEP 1141 H=1141:00 +CPU= 43.271
  19:19:40 STEP 1142 H=1142:00 +CPU= 44.460
  19:20:52 STEP 1143 H=1143:00 +CPU= 68.140
  19:21:38 STEP 1144 H=1144:00 +CPU= 44.529
  19:22:24 STEP 1145 H=1145:00 +CPU= 43.274
  19:23:36 STEP 1146 H=1146:00 +CPU= 69.572
  19:24:22 STEP 1147 H=1147:00 +CPU= 43.243
  19:25:08 STEP 1148 H=1148:00 +CPU= 44.394
  19:26:19 STEP 1149 H=1149:00 +CPU= 68.191
  19:27:06 STEP 1150 H=1150:00 +CPU= 44.366
  19:27:52 STEP 1151 H=1151:00 +CPU= 44.488
  19:29:17 STEP 1152 H=1152:00 +CPU= 79.974
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001152
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001152
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001152
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001128
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001128
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001128
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001140
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001140
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001140
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_47.zip
Uploading the intermediate file: upload_file_47.zip
  19:30:03 STEP 1153 H=1153:00 +CPU= 43.189
  19:30:49 STEP 1154 H=1154:00 +CPU= 44.598
  19:32:01 STEP 1155 H=1155:00 +CPU= 68.046
  19:32:47 STEP 1156 H=1156:00 +CPU= 44.416
  19:33:32 STEP 1157 H=1157:00 +CPU= 43.174
  19:34:45 STEP 1158 H=1158:00 +CPU= 69.448
  19:35:30 STEP 1159 H=1159:00 +CPU= 43.292
  19:36:17 STEP 1160 H=1160:00 +CPU= 44.601
  19:37:28 STEP 1161 H=1161:00 +CPU= 68.241
  19:38:15 STEP 1162 H=1162:00 +CPU= 44.505
  19:39:00 STEP 1163 H=1163:00 +CPU= 43.378
  19:40:20 STEP 1164 H=1164:00 +CPU= 76.185
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001164
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001164
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001164
  19:41:05 STEP 1165 H=1165:00 +CPU= 43.297
  19:41:52 STEP 1166 H=1166:00 +CPU= 44.570
  19:43:03 STEP 1167 H=1167:00 +CPU= 68.123
  19:43:50 STEP 1168 H=1168:00 +CPU= 44.542
  19:44:35 STEP 1169 H=1169:00 +CPU= 43.309
  19:45:48 STEP 1170 H=1170:00 +CPU= 69.370
  19:46:33 STEP 1171 H=1171:00 +CPU= 43.062
  19:47:20 STEP 1172 H=1172:00 +CPU= 44.512
  19:48:31 STEP 1173 H=1173:00 +CPU= 68.081
  19:49:17 STEP 1174 H=1174:00 +CPU= 44.310
  19:50:04 STEP 1175 H=1175:00 +CPU= 44.571
  19:51:29 STEP 1176 H=1176:00 +CPU= 79.928
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001176
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001176
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001176
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001152
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001152
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001152
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001164
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001164
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001164
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_48.zip
Uploading the intermediate file: upload_file_48.zip
  19:52:15 STEP 1177 H=1177:00 +CPU= 43.032
  19:53:01 STEP 1178 H=1178:00 +CPU= 44.371
  19:54:12 STEP 1179 H=1179:00 +CPU= 68.030
  19:54:59 STEP 1180 H=1180:00 +CPU= 44.534
  19:55:44 STEP 1181 H=1181:00 +CPU= 43.156
  19:56:56 STEP 1182 H=1182:00 +CPU= 69.491
  19:57:42 STEP 1183 H=1183:00 +CPU= 43.291
  19:58:28 STEP 1184 H=1184:00 +CPU= 44.526
  19:59:40 STEP 1185 H=1185:00 +CPU= 68.081
  20:00:26 STEP 1186 H=1186:00 +CPU= 44.448
  20:01:11 STEP 1187 H=1187:00 +CPU= 43.286
  20:02:31 STEP 1188 H=1188:00 +CPU= 76.253
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001188
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001188
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001188
  20:03:16 STEP 1189 H=1189:00 +CPU= 43.142
  20:04:03 STEP 1190 H=1190:00 +CPU= 44.516
  20:05:14 STEP 1191 H=1191:00 +CPU= 68.208
  20:06:01 STEP 1192 H=1192:00 +CPU= 44.358
  20:06:46 STEP 1193 H=1193:00 +CPU= 43.178
  20:07:59 STEP 1194 H=1194:00 +CPU= 69.566
  20:08:44 STEP 1195 H=1195:00 +CPU= 43.191
  20:09:31 STEP 1196 H=1196:00 +CPU= 44.528
  20:10:42 STEP 1197 H=1197:00 +CPU= 68.160
  20:11:28 STEP 1198 H=1198:00 +CPU= 44.473
  20:12:15 STEP 1199 H=1199:00 +CPU= 44.454
  20:13:40 STEP 1200 H=1200:00 +CPU= 80.179
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001200
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001200
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001200
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001176
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001176
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001176
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001188
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001188
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001188
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_49.zip
Uploading the intermediate file: upload_file_49.zip
Uploading trickle at timestep: 4320000
  20:14:25 STEP 1201 H=1201:00 +CPU= 43.025
  20:15:12 STEP 1202 H=1202:00 +CPU= 44.376
  20:16:23 STEP 1203 H=1203:00 +CPU= 67.968
  20:17:09 STEP 1204 H=1204:00 +CPU= 44.451
  20:17:55 STEP 1205 H=1205:00 +CPU= 43.261
  20:19:08 STEP 1206 H=1206:00 +CPU= 69.527
  20:19:53 STEP 1207 H=1207:00 +CPU= 43.204
  20:20:39 STEP 1208 H=1208:00 +CPU= 44.641
  20:21:51 STEP 1209 H=1209:00 +CPU= 68.003
  20:22:37 STEP 1210 H=1210:00 +CPU= 44.675
  20:23:23 STEP 1211 H=1211:00 +CPU= 43.271
  20:24:42 STEP 1212 H=1212:00 +CPU= 76.156
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001212
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001212
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001212
  20:25:28 STEP 1213 H=1213:00 +CPU= 43.291
  20:26:14 STEP 1214 H=1214:00 +CPU= 44.494
  20:27:26 STEP 1215 H=1215:00 +CPU= 67.991
  20:28:12 STEP 1216 H=1216:00 +CPU= 44.460
  20:28:58 STEP 1217 H=1217:00 +CPU= 43.259
  20:30:10 STEP 1218 H=1218:00 +CPU= 69.544
  20:30:56 STEP 1219 H=1219:00 +CPU= 43.261
  20:31:42 STEP 1220 H=1220:00 +CPU= 44.416
  20:32:53 STEP 1221 H=1221:00 +CPU= 67.992
  20:33:40 STEP 1222 H=1222:00 +CPU= 44.322
  20:34:26 STEP 1223 H=1223:00 +CPU= 44.405
  20:35:52 STEP 1224 H=1224:00 +CPU= 80.074
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001224
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001224
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001224
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001200
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001200
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001200
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001212
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001212
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001212
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_50.zip
Uploading the intermediate file: upload_file_50.zip
  20:36:37 STEP 1225 H=1225:00 +CPU= 43.107
  20:37:23 STEP 1226 H=1226:00 +CPU= 44.422
  20:38:34 STEP 1227 H=1227:00 +CPU= 67.996
  20:39:21 STEP 1228 H=1228:00 +CPU= 44.356
  20:40:06 STEP 1229 H=1229:00 +CPU= 43.199
  20:41:18 STEP 1230 H=1230:00 +CPU= 69.423
  20:42:04 STEP 1231 H=1231:00 +CPU= 43.264
  20:42:50 STEP 1232 H=1232:00 +CPU= 44.412
  20:44:01 STEP 1233 H=1233:00 +CPU= 68.147
  20:44:48 STEP 1234 H=1234:00 +CPU= 44.420
  20:45:33 STEP 1235 H=1235:00 +CPU= 43.176
  20:46:52 STEP 1236 H=1236:00 +CPU= 76.165
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001236
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001236
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001236
  20:47:37 STEP 1237 H=1237:00 +CPU= 43.135
  20:48:24 STEP 1238 H=1238:00 +CPU= 44.460
  20:49:35 STEP 1239 H=1239:00 +CPU= 68.059
  20:50:22 STEP 1240 H=1240:00 +CPU= 44.405
  20:51:07 STEP 1241 H=1241:00 +CPU= 43.068
  20:52:19 STEP 1242 H=1242:00 +CPU= 69.414
  20:53:04 STEP 1243 H=1243:00 +CPU= 43.057
  20:53:51 STEP 1244 H=1244:00 +CPU= 44.497
  20:55:02 STEP 1245 H=1245:00 +CPU= 68.002
  20:55:48 STEP 1246 H=1246:00 +CPU= 44.406
  20:56:35 STEP 1247 H=1247:00 +CPU= 44.507
  20:58:00 STEP 1248 H=1248:00 +CPU= 79.994
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001248
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001248
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001248
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001224
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001224
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001224
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001236
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001236
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001236
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_51.zip
Uploading the intermediate file: upload_file_51.zip
  20:58:45 STEP 1249 H=1249:00 +CPU= 43.151
  20:59:32 STEP 1250 H=1250:00 +CPU= 44.535
  21:00:43 STEP 1251 H=1251:00 +CPU= 67.922
  21:01:30 STEP 1252 H=1252:00 +CPU= 44.448
  21:02:15 STEP 1253 H=1253:00 +CPU= 43.072
  21:03:27 STEP 1254 H=1254:00 +CPU= 69.480
  21:04:12 STEP 1255 H=1255:00 +CPU= 43.069
  21:04:59 STEP 1256 H=1256:00 +CPU= 44.576
  21:06:10 STEP 1257 H=1257:00 +CPU= 68.000
  21:06:57 STEP 1258 H=1258:00 +CPU= 44.568
  21:07:42 STEP 1259 H=1259:00 +CPU= 43.181
  21:09:01 STEP 1260 H=1260:00 +CPU= 76.090
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001260
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001260
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001260
  21:09:46 STEP 1261 H=1261:00 +CPU= 43.186
  21:10:33 STEP 1262 H=1262:00 +CPU= 44.466
  21:11:44 STEP 1263 H=1263:00 +CPU= 67.951
  21:12:30 STEP 1264 H=1264:00 +CPU= 44.341
  21:13:16 STEP 1265 H=1265:00 +CPU= 43.161
  21:14:28 STEP 1266 H=1266:00 +CPU= 69.449
  21:15:14 STEP 1267 H=1267:00 +CPU= 43.389
  21:16:00 STEP 1268 H=1268:00 +CPU= 44.269
  21:17:11 STEP 1269 H=1269:00 +CPU= 68.013
  21:17:58 STEP 1270 H=1270:00 +CPU= 44.437
  21:18:44 STEP 1271 H=1271:00 +CPU= 44.466
  21:20:10 STEP 1272 H=1272:00 +CPU= 80.137
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001272
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001272
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001272
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001248
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001248
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001248
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001260
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001260
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001260
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_52.zip
Uploading the intermediate file: upload_file_52.zip
  21:20:55 STEP 1273 H=1273:00 +CPU= 43.074
  21:21:41 STEP 1274 H=1274:00 +CPU= 44.368
  21:22:52 STEP 1275 H=1275:00 +CPU= 67.969
  21:23:39 STEP 1276 H=1276:00 +CPU= 44.417
  21:24:24 STEP 1277 H=1277:00 +CPU= 43.251
  21:25:37 STEP 1278 H=1278:00 +CPU= 69.432
  21:26:22 STEP 1279 H=1279:00 +CPU= 43.132
  21:27:08 STEP 1280 H=1280:00 +CPU= 44.534
  21:28:20 STEP 1281 H=1281:00 +CPU= 68.079
  21:29:06 STEP 1282 H=1282:00 +CPU= 44.502
  21:29:51 STEP 1283 H=1283:00 +CPU= 43.244
  21:31:11 STEP 1284 H=1284:00 +CPU= 76.026
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001284
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001284
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001284
  21:31:56 STEP 1285 H=1285:00 +CPU= 43.269
  21:32:43 STEP 1286 H=1286:00 +CPU= 44.435
  21:33:53 STEP 1287 H=1287:00 +CPU= 67.948
  21:34:40 STEP 1288 H=1288:00 +CPU= 44.462
  21:35:25 STEP 1289 H=1289:00 +CPU= 43.108
  21:36:37 STEP 1290 H=1290:00 +CPU= 69.360
  21:37:23 STEP 1291 H=1291:00 +CPU= 43.314
  21:38:09 STEP 1292 H=1292:00 +CPU= 44.418
  21:39:20 STEP 1293 H=1293:00 +CPU= 67.960
  21:40:07 STEP 1294 H=1294:00 +CPU= 44.389
  21:40:53 STEP 1295 H=1295:00 +CPU= 44.501
  21:42:19 STEP 1296 H=1296:00 +CPU= 80.071
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001296
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001296
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001296
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001272
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001272
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001272
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001284
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001284
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001284
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_53.zip
Uploading the intermediate file: upload_file_53.zip
  21:43:04 STEP 1297 H=1297:00 +CPU= 43.177
  21:43:50 STEP 1298 H=1298:00 +CPU= 44.340
  21:45:01 STEP 1299 H=1299:00 +CPU= 68.129
  21:45:48 STEP 1300 H=1300:00 +CPU= 44.526
  21:46:33 STEP 1301 H=1301:00 +CPU= 43.290
  21:47:46 STEP 1302 H=1302:00 +CPU= 69.387
  21:48:31 STEP 1303 H=1303:00 +CPU= 43.212
  21:49:18 STEP 1304 H=1304:00 +CPU= 44.655
  21:50:29 STEP 1305 H=1305:00 +CPU= 67.923
  21:51:15 STEP 1306 H=1306:00 +CPU= 44.572
  21:52:01 STEP 1307 H=1307:00 +CPU= 43.327
  21:53:20 STEP 1308 H=1308:00 +CPU= 76.059
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001308
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001308
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001308
  21:54:05 STEP 1309 H=1309:00 +CPU= 43.245
  21:54:52 STEP 1310 H=1310:00 +CPU= 44.433
  21:56:03 STEP 1311 H=1311:00 +CPU= 68.183
  21:56:49 STEP 1312 H=1312:00 +CPU= 44.416
  21:57:34 STEP 1313 H=1313:00 +CPU= 43.102
  21:58:47 STEP 1314 H=1314:00 +CPU= 69.486
  21:59:32 STEP 1315 H=1315:00 +CPU= 43.248
  22:00:19 STEP 1316 H=1316:00 +CPU= 44.426
  22:01:30 STEP 1317 H=1317:00 +CPU= 68.022
  22:02:16 STEP 1318 H=1318:00 +CPU= 44.369
  22:03:03 STEP 1319 H=1319:00 +CPU= 44.549
  22:04:28 STEP 1320 H=1320:00 +CPU= 79.906
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001320
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001320
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001320
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001296
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001296
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001296
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001308
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001308
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001308
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_54.zip
Uploading the intermediate file: upload_file_54.zip
  22:05:13 STEP 1321 H=1321:00 +CPU= 43.100
  22:06:00 STEP 1322 H=1322:00 +CPU= 44.319
  22:07:11 STEP 1323 H=1323:00 +CPU= 67.975
  22:07:57 STEP 1324 H=1324:00 +CPU= 44.327
  22:08:42 STEP 1325 H=1325:00 +CPU= 43.150
  22:09:55 STEP 1326 H=1326:00 +CPU= 69.363
  22:10:40 STEP 1327 H=1327:00 +CPU= 43.097
  22:11:26 STEP 1328 H=1328:00 +CPU= 44.420
  22:12:37 STEP 1329 H=1329:00 +CPU= 68.072
  22:13:24 STEP 1330 H=1330:00 +CPU= 44.311
  22:14:09 STEP 1331 H=1331:00 +CPU= 43.207
  22:15:28 STEP 1332 H=1332:00 +CPU= 75.998
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001332
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001332
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001332
  22:16:13 STEP 1333 H=1333:00 +CPU= 43.211
  22:17:00 STEP 1334 H=1334:00 +CPU= 44.685
  22:18:11 STEP 1335 H=1335:00 +CPU= 68.112
  22:18:58 STEP 1336 H=1336:00 +CPU= 44.528
  22:19:43 STEP 1337 H=1337:00 +CPU= 43.195
  22:20:56 STEP 1338 H=1338:00 +CPU= 69.472
  22:21:41 STEP 1339 H=1339:00 +CPU= 43.087
  22:22:27 STEP 1340 H=1340:00 +CPU= 44.469
  22:23:38 STEP 1341 H=1341:00 +CPU= 67.960
  22:24:25 STEP 1342 H=1342:00 +CPU= 44.430
  22:25:11 STEP 1343 H=1343:00 +CPU= 44.537
  22:26:37 STEP 1344 H=1344:00 +CPU= 80.144
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001344
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001344
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001344
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001320
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001320
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001320
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001332
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001332
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001332
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_55.zip
Uploading the intermediate file: upload_file_55.zip
  22:27:22 STEP 1345 H=1345:00 +CPU= 42.940
  22:28:08 STEP 1346 H=1346:00 +CPU= 44.184
  22:29:19 STEP 1347 H=1347:00 +CPU= 68.021
  22:30:06 STEP 1348 H=1348:00 +CPU= 44.381
  22:30:51 STEP 1349 H=1349:00 +CPU= 43.308
  22:32:04 STEP 1350 H=1350:00 +CPU= 69.256
  22:32:49 STEP 1351 H=1351:00 +CPU= 43.310
  22:33:36 STEP 1352 H=1352:00 +CPU= 44.635
  22:34:47 STEP 1353 H=1353:00 +CPU= 68.051
  22:35:34 STEP 1354 H=1354:00 +CPU= 44.452
  22:36:19 STEP 1355 H=1355:00 +CPU= 43.293
  22:37:38 STEP 1356 H=1356:00 +CPU= 76.033
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001356
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001356
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001356
  22:38:23 STEP 1357 H=1357:00 +CPU= 43.142
  22:39:10 STEP 1358 H=1358:00 +CPU= 44.546
  22:40:21 STEP 1359 H=1359:00 +CPU= 67.969
  22:41:08 STEP 1360 H=1360:00 +CPU= 44.414
  22:41:53 STEP 1361 H=1361:00 +CPU= 43.113
  22:43:05 STEP 1362 H=1362:00 +CPU= 69.472
  22:43:50 STEP 1363 H=1363:00 +CPU= 43.173
  22:44:37 STEP 1364 H=1364:00 +CPU= 44.469
  22:45:48 STEP 1365 H=1365:00 +CPU= 67.948
  22:46:35 STEP 1366 H=1366:00 +CPU= 44.571
  22:47:21 STEP 1367 H=1367:00 +CPU= 44.489
  22:48:47 STEP 1368 H=1368:00 +CPU= 80.376
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001368
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001368
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001368
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001344
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001344
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001344
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001356
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001356
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001356
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_56.zip
Uploading the intermediate file: upload_file_56.zip
  22:49:32 STEP 1369 H=1369:00 +CPU= 42.980
  22:50:18 STEP 1370 H=1370:00 +CPU= 44.367
  22:51:29 STEP 1371 H=1371:00 +CPU= 67.951
  22:52:16 STEP 1372 H=1372:00 +CPU= 44.419
  22:53:01 STEP 1373 H=1373:00 +CPU= 43.223
  22:54:13 STEP 1374 H=1374:00 +CPU= 69.438
  22:54:59 STEP 1375 H=1375:00 +CPU= 43.295
  22:55:45 STEP 1376 H=1376:00 +CPU= 44.577
  22:56:56 STEP 1377 H=1377:00 +CPU= 68.081
  22:57:43 STEP 1378 H=1378:00 +CPU= 44.629
  22:58:29 STEP 1379 H=1379:00 +CPU= 43.289
  22:59:48 STEP 1380 H=1380:00 +CPU= 76.089
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001380
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001380
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001380
  23:00:33 STEP 1381 H=1381:00 +CPU= 43.159
  23:01:20 STEP 1382 H=1382:00 +CPU= 44.437
  23:02:31 STEP 1383 H=1383:00 +CPU= 68.215
  23:03:18 STEP 1384 H=1384:00 +CPU= 44.470
  23:04:03 STEP 1385 H=1385:00 +CPU= 43.158
  23:05:15 STEP 1386 H=1386:00 +CPU= 69.313
  23:06:01 STEP 1387 H=1387:00 +CPU= 43.065
  23:06:47 STEP 1388 H=1388:00 +CPU= 44.442
  23:07:58 STEP 1389 H=1389:00 +CPU= 68.030
  23:08:45 STEP 1390 H=1390:00 +CPU= 44.474
  23:09:32 STEP 1391 H=1391:00 +CPU= 44.607
  23:10:57 STEP 1392 H=1392:00 +CPU= 80.295
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001392
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001392
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001392
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001368
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001368
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001368
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001380
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001380
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001380
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_57.zip
Uploading the intermediate file: upload_file_57.zip
  23:11:42 STEP 1393 H=1393:00 +CPU= 43.006
  23:12:29 STEP 1394 H=1394:00 +CPU= 44.433
  23:13:40 STEP 1395 H=1395:00 +CPU= 67.977
  23:14:26 STEP 1396 H=1396:00 +CPU= 44.440
  23:15:11 STEP 1397 H=1397:00 +CPU= 43.302
  23:16:24 STEP 1398 H=1398:00 +CPU= 69.689
  23:17:09 STEP 1399 H=1399:00 +CPU= 43.239
  23:17:56 STEP 1400 H=1400:00 +CPU= 44.626
  23:19:07 STEP 1401 H=1401:00 +CPU= 68.223
  23:19:54 STEP 1402 H=1402:00 +CPU= 44.590
  23:20:39 STEP 1403 H=1403:00 +CPU= 43.237
  23:21:59 STEP 1404 H=1404:00 +CPU= 76.263
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001404
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001404
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001404
  23:22:44 STEP 1405 H=1405:00 +CPU= 43.291
  23:23:31 STEP 1406 H=1406:00 +CPU= 44.504
  23:24:42 STEP 1407 H=1407:00 +CPU= 67.893
  23:25:28 STEP 1408 H=1408:00 +CPU= 44.482
  23:26:13 STEP 1409 H=1409:00 +CPU= 43.226
  23:27:26 STEP 1410 H=1410:00 +CPU= 69.375
  23:28:11 STEP 1411 H=1411:00 +CPU= 43.155
  23:28:57 STEP 1412 H=1412:00 +CPU= 44.562
  23:30:09 STEP 1413 H=1413:00 +CPU= 68.081
  23:30:55 STEP 1414 H=1414:00 +CPU= 44.346
  23:31:41 STEP 1415 H=1415:00 +CPU= 44.563
  23:33:07 STEP 1416 H=1416:00 +CPU= 80.230
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001416
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001416
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001416
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001392
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001392
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001392
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001404
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001404
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001404
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_58.zip
Uploading the intermediate file: upload_file_58.zip
  23:33:52 STEP 1417 H=1417:00 +CPU= 43.163
  23:34:39 STEP 1418 H=1418:00 +CPU= 44.472
  23:35:50 STEP 1419 H=1419:00 +CPU= 68.072
  23:36:36 STEP 1420 H=1420:00 +CPU= 44.412
  23:37:21 STEP 1421 H=1421:00 +CPU= 43.226
  23:38:34 STEP 1422 H=1422:00 +CPU= 69.596
  23:39:20 STEP 1423 H=1423:00 +CPU= 43.321
  23:40:06 STEP 1424 H=1424:00 +CPU= 44.579
  23:41:17 STEP 1425 H=1425:00 +CPU= 68.331
  23:42:04 STEP 1426 H=1426:00 +CPU= 44.550
  23:42:49 STEP 1427 H=1427:00 +CPU= 43.277
  23:44:09 STEP 1428 H=1428:00 +CPU= 76.235
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001428
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001428
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001428
  23:44:55 STEP 1429 H=1429:00 +CPU= 43.324
  23:45:41 STEP 1430 H=1430:00 +CPU= 44.524
  23:46:52 STEP 1431 H=1431:00 +CPU= 67.966
  23:47:39 STEP 1432 H=1432:00 +CPU= 44.597
  23:48:24 STEP 1433 H=1433:00 +CPU= 43.216
  23:49:37 STEP 1434 H=1434:00 +CPU= 69.548
  23:50:22 STEP 1435 H=1435:00 +CPU= 43.275
  23:51:09 STEP 1436 H=1436:00 +CPU= 44.441
  23:52:20 STEP 1437 H=1437:00 +CPU= 68.007
  23:53:06 STEP 1438 H=1438:00 +CPU= 44.437
  23:53:53 STEP 1439 H=1439:00 +CPU= 44.494
  23:55:18 STEP 1440 H=1440:00 +CPU= 80.087
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001440
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001440
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001440
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001416
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001416
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001416
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMGGhq0f+001428
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMSHhq0f+001428
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12185230/ICMUAhq0f+001428
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0586_1999050100_123_968_12185230_2_r775257057_59.zip
Uploading the intermediate file: upload_file_59.zip
Uploading trickle at timestep: 5184000
  23:56:04 STEP 1441 H=1441:00 +CPU= 43.071
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:45:43 1537854 22304754 oifs_43r3_ps_0586_1999050100_123_968_12185230_2 5,184,000 76,272 0.0147
09 May 2023 16:45:38 1537854 22304754 oifs_43r3_ps_0586_1999050100_123_968_12185230_2 4,320,000 63,576 0.0147
09 May 2023 16:45:33 1537854 22304754 oifs_43r3_ps_0586_1999050100_123_968_12185230_2 3,456,000 50,875 0.0147
09 May 2023 16:45:28 1537854 22304754 oifs_43r3_ps_0586_1999050100_123_968_12185230_2 2,592,000 38,176 0.0147
09 May 2023 16:45:23 1537854 22304754 oifs_43r3_ps_0586_1999050100_123_968_12185230_2 1,728,000 25,479 0.0147
09 May 2023 16:45:18 1537854 22304754 oifs_43r3_ps_0586_1999050100_123_968_12185230_2 864,000 12,781 0.0148


©2024 climateprediction.net