climateprediction.net home page
Task 22311240

Task 22311240

Name oifs_43r3_bl_a394_2016092300_15_991_12211003_0
Workunit 12211003
Created 13 Feb 2023, 19:46:12 UTC
Sent 14 Feb 2023, 5:20:45 UTC
Report deadline 15 Apr 2023, 5:20:45 UTC
Received 15 Feb 2023, 14:44:32 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1518460
Run time 18 hours 53 min 48 sec
CPU time 18 hours 52 min 25 sec
Validate state Invalid
Credit 1,209.60
Device peak FLOPS 4.96 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 5,067.20 MB
Peak swap size 5,930.78 MB
Peak disk usage 1,334.31 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
p: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a394_2016092300_15_991_12211003_0_r1080860046_7.zip
Uploading the intermediate file: upload_file_7.zip
  08:19:40 STEP  769 H= 192:15 +CPU= 55.147
  08:20:36 STEP  770 H= 192:30 +CPU= 56.264
  08:21:32 STEP  771 H= 192:45 +CPU= 55.994
  08:22:32 STEP  772 H= 193:00 +CPU= 60.006
  08:23:32 STEP  773 H= 193:15 +CPU= 59.771
  08:24:25 STEP  774 H= 193:30 +CPU= 53.407
  08:25:20 STEP  775 H= 193:45 +CPU= 54.373
  08:26:13 STEP  776 H= 194:00 +CPU= 53.042
  08:27:08 STEP  777 H= 194:15 +CPU= 55.789
  08:28:10 STEP  778 H= 194:30 +CPU= 61.037
  08:29:07 STEP  779 H= 194:45 +CPU= 57.090
  08:30:11 STEP  780 H= 195:00 +CPU= 64.155
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000780
  08:31:05 STEP  781 H= 195:15 +CPU= 54.275
  08:32:03 STEP  782 H= 195:30 +CPU= 58.377
  08:32:59 STEP  783 H= 195:45 +CPU= 55.759
  08:33:56 STEP  784 H= 196:00 +CPU= 56.673
  08:34:48 STEP  785 H= 196:15 +CPU= 52.411
  08:35:50 STEP  786 H= 196:30 +CPU= 61.765
  08:36:50 STEP  787 H= 196:45 +CPU= 59.654
  08:37:45 STEP  788 H= 197:00 +CPU= 54.958
  08:38:39 STEP  789 H= 197:15 +CPU= 54.146
  08:39:33 STEP  790 H= 197:30 +CPU= 53.656
  08:40:27 STEP  791 H= 197:45 +CPU= 54.498
  08:41:36 STEP  792 H= 198:00 +CPU= 69.272
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000792
  08:42:34 STEP  793 H= 198:15 +CPU= 57.205
  08:43:29 STEP  794 H= 198:30 +CPU= 55.040
  08:44:28 STEP  795 H= 198:45 +CPU= 59.775
  08:45:29 STEP  796 H= 199:00 +CPU= 60.398
  08:46:28 STEP  797 H= 199:15 +CPU= 59.428
  08:47:29 STEP  798 H= 199:30 +CPU= 61.040
  08:48:28 STEP  799 H= 199:45 +CPU= 58.906
  08:49:26 STEP  800 H= 200:00 +CPU= 57.658
  08:50:23 STEP  801 H= 200:15 +CPU= 57.230
  08:51:19 STEP  802 H= 200:30 +CPU= 55.522
  08:52:18 STEP  803 H= 200:45 +CPU= 59.524
  08:53:25 STEP  804 H= 201:00 +CPU= 66.478
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000804
  08:54:21 STEP  805 H= 201:15 +CPU= 55.940
  08:55:18 STEP  806 H= 201:30 +CPU= 56.888
  08:56:16 STEP  807 H= 201:45 +CPU= 58.206
  08:57:16 STEP  808 H= 202:00 +CPU= 59.900
  08:58:15 STEP  809 H= 202:15 +CPU= 59.543
  08:59:12 STEP  810 H= 202:30 +CPU= 56.916
  09:00:09 STEP  811 H= 202:45 +CPU= 56.640
  09:01:05 STEP  812 H= 203:00 +CPU= 56.191
  09:02:04 STEP  813 H= 203:15 +CPU= 58.917
  09:03:03 STEP  814 H= 203:30 +CPU= 59.527
  09:04:03 STEP  815 H= 203:45 +CPU= 59.438
  09:05:25 STEP  816 H= 204:00 +CPU= 81.812
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000816
  09:06:23 STEP  817 H= 204:15 +CPU= 58.472
  09:07:24 STEP  818 H= 204:30 +CPU= 60.358
  09:08:23 STEP  819 H= 204:45 +CPU= 59.524
  09:09:24 STEP  820 H= 205:00 +CPU= 60.894
  09:10:21 STEP  821 H= 205:15 +CPU= 57.194
  09:11:18 STEP  822 H= 205:30 +CPU= 56.749
  09:12:13 STEP  823 H= 205:45 +CPU= 54.518
  09:13:10 STEP  824 H= 206:00 +CPU= 57.004
  09:14:05 STEP  825 H= 206:15 +CPU= 54.932
  09:15:01 STEP  826 H= 206:30 +CPU= 56.376
  09:15:56 STEP  827 H= 206:45 +CPU= 54.877
  09:17:02 STEP  828 H= 207:00 +CPU= 66.162
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000828
  09:17:59 STEP  829 H= 207:15 +CPU= 57.465
  09:18:56 STEP  830 H= 207:30 +CPU= 56.900
  09:19:55 STEP  831 H= 207:45 +CPU= 59.019
  09:20:58 STEP  832 H= 208:00 +CPU= 62.852
  09:21:58 STEP  833 H= 208:15 +CPU= 60.205
  09:22:59 STEP  834 H= 208:30 +CPU= 60.719
  09:23:54 STEP  835 H= 208:45 +CPU= 55.174
  09:24:50 STEP  836 H= 209:00 +CPU= 55.621
  09:25:47 STEP  837 H= 209:15 +CPU= 56.640
  09:26:43 STEP  838 H= 209:30 +CPU= 56.071
  09:27:38 STEP  839 H= 209:45 +CPU= 55.706
  09:28:45 STEP  840 H= 210:00 +CPU= 66.681
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000840
  09:29:43 STEP  841 H= 210:15 +CPU= 57.406
  09:30:39 STEP  842 H= 210:30 +CPU= 56.889
  09:31:37 STEP  843 H= 210:45 +CPU= 57.122
  09:32:31 STEP  844 H= 211:00 +CPU= 54.764
  09:33:27 STEP  845 H= 211:15 +CPU= 55.212
  09:34:21 STEP  846 H= 211:30 +CPU= 54.867
  09:35:17 STEP  847 H= 211:45 +CPU= 55.238
  09:36:16 STEP  848 H= 212:00 +CPU= 59.748
  09:37:18 STEP  849 H= 212:15 +CPU= 61.463
  09:38:19 STEP  850 H= 212:30 +CPU= 61.239
  09:39:18 STEP  851 H= 212:45 +CPU= 59.194
  09:40:29 STEP  852 H= 213:00 +CPU= 70.146
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000852
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000852
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000852
  09:41:27 STEP  853 H= 213:15 +CPU= 58.839
  09:42:30 STEP  854 H= 213:30 +CPU= 62.221
  09:43:28 STEP  855 H= 213:45 +CPU= 57.924
  09:44:24 STEP  856 H= 214:00 +CPU= 55.917
  09:45:22 STEP  857 H= 214:15 +CPU= 58.224
  09:46:17 STEP  858 H= 214:30 +CPU= 55.499
  09:47:10 STEP  859 H= 214:45 +CPU= 52.365
  09:48:10 STEP  860 H= 215:00 +CPU= 60.123
  09:49:11 STEP  861 H= 215:15 +CPU= 61.382
  09:50:12 STEP  862 H= 215:30 +CPU= 60.517
  09:51:11 STEP  863 H= 215:45 +CPU= 59.693
  09:52:30 STEP  864 H= 216:00 +CPU= 78.439
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000864
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000864
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a394_2016092300_15_991_12211003_0_r1080860046_8.zip
Uploading the intermediate file: upload_file_8.zip
  09:53:27 STEP  865 H= 216:15 +CPU= 57.166
  09:54:25 STEP  866 H= 216:30 +CPU= 57.812
  09:55:23 STEP  867 H= 216:45 +CPU= 58.094
  09:56:24 STEP  868 H= 217:00 +CPU= 60.855
  09:57:19 STEP  869 H= 217:15 +CPU= 55.311
  09:58:19 STEP  870 H= 217:30 +CPU= 59.695
  09:59:19 STEP  871 H= 217:45 +CPU= 60.520
  10:00:20 STEP  872 H= 218:00 +CPU= 61.027
  10:01:19 STEP  873 H= 218:15 +CPU= 58.356
  10:02:18 STEP  874 H= 218:30 +CPU= 59.551
  10:03:15 STEP  875 H= 218:45 +CPU= 56.314
  10:04:21 STEP  876 H= 219:00 +CPU= 66.265
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000876
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000876
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000876
  10:05:20 STEP  877 H= 219:15 +CPU= 58.681
  10:06:18 STEP  878 H= 219:30 +CPU= 58.601
  10:07:17 STEP  879 H= 219:45 +CPU= 58.297
  10:08:14 STEP  880 H= 220:00 +CPU= 57.886
  10:09:14 STEP  881 H= 220:15 +CPU= 59.180
  10:10:12 STEP  882 H= 220:30 +CPU= 58.505
  10:11:13 STEP  883 H= 220:45 +CPU= 60.577
  10:12:15 STEP  884 H= 221:00 +CPU= 62.400
  10:13:14 STEP  885 H= 221:15 +CPU= 58.633
  10:14:12 STEP  886 H= 221:30 +CPU= 58.338
  10:15:11 STEP  887 H= 221:45 +CPU= 59.237
  10:16:20 STEP  888 H= 222:00 +CPU= 68.938
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000888
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000888
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000888
  10:17:18 STEP  889 H= 222:15 +CPU= 57.524
  10:18:14 STEP  890 H= 222:30 +CPU= 56.350
  10:19:17 STEP  891 H= 222:45 +CPU= 62.679
  10:20:17 STEP  892 H= 223:00 +CPU= 60.008
  10:21:16 STEP  893 H= 223:15 +CPU= 59.041
  10:22:15 STEP  894 H= 223:30 +CPU= 58.517
  10:23:13 STEP  895 H= 223:45 +CPU= 58.035
  10:24:10 STEP  896 H= 224:00 +CPU= 57.838
  10:25:07 STEP  897 H= 224:15 +CPU= 56.710
  10:26:01 STEP  898 H= 224:30 +CPU= 53.434
  10:26:56 STEP  899 H= 224:45 +CPU= 55.398
  10:28:06 STEP  900 H= 225:00 +CPU= 69.684
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000900
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000900
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000900
  10:29:04 STEP  901 H= 225:15 +CPU= 58.485
  10:30:03 STEP  902 H= 225:30 +CPU= 58.976
  10:31:05 STEP  903 H= 225:45 +CPU= 62.145
  10:32:01 STEP  904 H= 226:00 +CPU= 55.404
  10:32:56 STEP  905 H= 226:15 +CPU= 55.391
  10:33:53 STEP  906 H= 226:30 +CPU= 56.801
  10:34:52 STEP  907 H= 226:45 +CPU= 58.595
  10:35:48 STEP  908 H= 227:00 +CPU= 56.581
  10:36:44 STEP  909 H= 227:15 +CPU= 55.506
  10:37:40 STEP  910 H= 227:30 +CPU= 56.532
  10:38:37 STEP  911 H= 227:45 +CPU= 57.243
  10:39:56 STEP  912 H= 228:00 +CPU= 78.256
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000912
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000912
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000912
  10:40:55 STEP  913 H= 228:15 +CPU= 59.502
  10:41:57 STEP  914 H= 228:30 +CPU= 61.441
  10:42:55 STEP  915 H= 228:45 +CPU= 58.308
  10:43:56 STEP  916 H= 229:00 +CPU= 60.910
  10:44:56 STEP  917 H= 229:15 +CPU= 59.620
  10:45:51 STEP  918 H= 229:30 +CPU= 55.434
  10:46:46 STEP  919 H= 229:45 +CPU= 55.295
  10:47:43 STEP  920 H= 230:00 +CPU= 56.483
  10:48:36 STEP  921 H= 230:15 +CPU= 53.358
  10:49:33 STEP  922 H= 230:30 +CPU= 56.734
  10:50:32 STEP  923 H= 230:45 +CPU= 58.758
  10:51:39 STEP  924 H= 231:00 +CPU= 67.424
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000924
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000924
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000924
  10:52:35 STEP  925 H= 231:15 +CPU= 55.831
  10:53:35 STEP  926 H= 231:30 +CPU= 60.160
  10:54:30 STEP  927 H= 231:45 +CPU= 55.121
  10:55:27 STEP  928 H= 232:00 +CPU= 56.541
  10:56:22 STEP  929 H= 232:15 +CPU= 55.165
  10:57:21 STEP  930 H= 232:30 +CPU= 59.327
  10:58:23 STEP  931 H= 232:45 +CPU= 61.707
  10:59:23 STEP  932 H= 233:00 +CPU= 60.149
  11:00:19 STEP  933 H= 233:15 +CPU= 55.999
  11:01:16 STEP  934 H= 233:30 +CPU= 56.637
  11:02:09 STEP  935 H= 233:45 +CPU= 53.489
  11:03:16 STEP  936 H= 234:00 +CPU= 66.331
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000936
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000936
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000936
  11:04:11 STEP  937 H= 234:15 +CPU= 54.858
  11:05:10 STEP  938 H= 234:30 +CPU= 59.479
  11:06:06 STEP  939 H= 234:45 +CPU= 55.602
  11:07:01 STEP  940 H= 235:00 +CPU= 55.764
  11:07:55 STEP  941 H= 235:15 +CPU= 53.596
  11:08:48 STEP  942 H= 235:30 +CPU= 53.125
  11:09:45 STEP  943 H= 235:45 +CPU= 57.083
  11:10:42 STEP  944 H= 236:00 +CPU= 56.081
  11:11:36 STEP  945 H= 236:15 +CPU= 53.963
  11:12:30 STEP  946 H= 236:30 +CPU= 54.052
  11:13:24 STEP  947 H= 236:45 +CPU= 54.373
  11:14:35 STEP  948 H= 237:00 +CPU= 70.760
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000948
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000948
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000948
  11:15:31 STEP  949 H= 237:15 +CPU= 55.776
  11:16:23 STEP  950 H= 237:30 +CPU= 52.157
  11:17:17 STEP  951 H= 237:45 +CPU= 54.441
  11:18:15 STEP  952 H= 238:00 +CPU= 57.543
  11:19:11 STEP  953 H= 238:15 +CPU= 56.198
  11:20:09 STEP  954 H= 238:30 +CPU= 57.717
  11:21:08 STEP  955 H= 238:45 +CPU= 59.602
  11:22:02 STEP  956 H= 239:00 +CPU= 53.542
  11:23:01 STEP  957 H= 239:15 +CPU= 58.692
  11:23:56 STEP  958 H= 239:30 +CPU= 55.330
  11:24:51 STEP  959 H= 239:45 +CPU= 54.985
  11:26:10 STEP  960 H= 240:00 +CPU= 79.372
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000960
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000960
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a394_2016092300_15_991_12211003_0_r1080860046_9.zip
Uploading the intermediate file: upload_file_9.zip
Uploading trickle at timestep: 864000
  11:27:10 STEP  961 H= 240:15 +CPU= 59.127
  11:28:08 STEP  962 H= 240:30 +CPU= 58.735
  11:29:07 STEP  963 H= 240:45 +CPU= 59.140
  11:30:08 STEP  964 H= 241:00 +CPU= 60.074
  11:31:08 STEP  965 H= 241:15 +CPU= 60.494
  11:32:08 STEP  966 H= 241:30 +CPU= 59.468
  11:33:11 STEP  967 H= 241:45 +CPU= 63.039
  11:34:08 STEP  968 H= 242:00 +CPU= 57.462
  11:35:07 STEP  969 H= 242:15 +CPU= 58.459
  11:36:06 STEP  970 H= 242:30 +CPU= 59.817
  11:37:03 STEP  971 H= 242:45 +CPU= 56.200
  11:38:15 STEP  972 H= 243:00 +CPU= 72.169
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000972
  11:39:14 STEP  973 H= 243:15 +CPU= 59.477
  11:40:17 STEP  974 H= 243:30 +CPU= 62.289
  11:41:18 STEP  975 H= 243:45 +CPU= 61.171
  11:42:19 STEP  976 H= 244:00 +CPU= 61.642
  11:43:20 STEP  977 H= 244:15 +CPU= 60.747
  11:44:18 STEP  978 H= 244:30 +CPU= 57.882
  11:45:19 STEP  979 H= 244:45 +CPU= 60.595
  11:46:15 STEP  980 H= 245:00 +CPU= 55.918
  11:47:16 STEP  981 H= 245:15 +CPU= 61.784
  11:48:17 STEP  982 H= 245:30 +CPU= 61.031
  11:49:18 STEP  983 H= 245:45 +CPU= 60.287
  11:50:27 STEP  984 H= 246:00 +CPU= 69.152
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000984
  11:51:28 STEP  985 H= 246:15 +CPU= 61.127
  11:52:29 STEP  986 H= 246:30 +CPU= 60.779
  11:53:32 STEP  987 H= 246:45 +CPU= 62.709
  11:54:34 STEP  988 H= 247:00 +CPU= 62.653
  11:55:35 STEP  989 H= 247:15 +CPU= 60.341
  11:56:33 STEP  990 H= 247:30 +CPU= 58.093
  11:57:32 STEP  991 H= 247:45 +CPU= 59.208
  11:58:31 STEP  992 H= 248:00 +CPU= 59.556
  11:59:33 STEP  993 H= 248:15 +CPU= 61.665
  12:00:29 STEP  994 H= 248:30 +CPU= 55.990
  12:01:25 STEP  995 H= 248:45 +CPU= 55.853
  12:02:32 STEP  996 H= 249:00 +CPU= 67.384
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000996
  12:03:29 STEP  997 H= 249:15 +CPU= 56.877
  12:04:25 STEP  998 H= 249:30 +CPU= 56.008
  12:05:22 STEP  999 H= 249:45 +CPU= 56.316
  12:06:20 STEP 1000 H= 250:00 +CPU= 58.643
  12:07:11 STEP 1001 H= 250:15 +CPU= 50.461
  12:08:04 STEP 1002 H= 250:30 +CPU= 52.824
  12:08:57 STEP 1003 H= 250:45 +CPU= 53.021
  12:09:50 STEP 1004 H= 251:00 +CPU= 53.468
  12:10:49 STEP 1005 H= 251:15 +CPU= 58.834
  12:11:49 STEP 1006 H= 251:30 +CPU= 59.667
  12:12:44 STEP 1007 H= 251:45 +CPU= 55.648
  12:14:03 STEP 1008 H= 252:00 +CPU= 78.422
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001008
  12:14:59 STEP 1009 H= 252:15 +CPU= 56.747
  12:15:57 STEP 1010 H= 252:30 +CPU= 57.666
  12:16:49 STEP 1011 H= 252:45 +CPU= 52.181
  12:17:45 STEP 1012 H= 253:00 +CPU= 55.813
  12:18:40 STEP 1013 H= 253:15 +CPU= 54.617
  12:19:33 STEP 1014 H= 253:30 +CPU= 53.382
  12:20:30 STEP 1015 H= 253:45 +CPU= 57.214
  12:21:35 STEP 1016 H= 254:00 +CPU= 64.148
  12:22:35 STEP 1017 H= 254:15 +CPU= 60.443
  12:23:36 STEP 1018 H= 254:30 +CPU= 61.137
  12:24:35 STEP 1019 H= 254:45 +CPU= 58.486
  12:25:39 STEP 1020 H= 255:00 +CPU= 64.701
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001020
  12:26:38 STEP 1021 H= 255:15 +CPU= 58.767
  12:27:33 STEP 1022 H= 255:30 +CPU= 54.508
  12:28:32 STEP 1023 H= 255:45 +CPU= 59.281
  12:29:36 STEP 1024 H= 256:00 +CPU= 63.906
  12:30:36 STEP 1025 H= 256:15 +CPU= 60.459
  12:31:33 STEP 1026 H= 256:30 +CPU= 56.598
  12:32:26 STEP 1027 H= 256:45 +CPU= 53.379
  12:33:29 STEP 1028 H= 257:00 +CPU= 62.482
  12:34:23 STEP 1029 H= 257:15 +CPU= 54.491
  12:35:17 STEP 1030 H= 257:30 +CPU= 53.838
  12:36:11 STEP 1031 H= 257:45 +CPU= 53.588
  12:37:18 STEP 1032 H= 258:00 +CPU= 67.454
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001032
  12:38:11 STEP 1033 H= 258:15 +CPU= 52.464
  12:39:05 STEP 1034 H= 258:30 +CPU= 54.252
  12:40:00 STEP 1035 H= 258:45 +CPU= 55.104
  12:41:02 STEP 1036 H= 259:00 +CPU= 61.758
  12:42:05 STEP 1037 H= 259:15 +CPU= 63.308
  12:43:05 STEP 1038 H= 259:30 +CPU= 59.806
  12:43:59 STEP 1039 H= 259:45 +CPU= 54.340
  12:44:54 STEP 1040 H= 260:00 +CPU= 54.111
  12:45:48 STEP 1041 H= 260:15 +CPU= 54.532
  12:46:46 STEP 1042 H= 260:30 +CPU= 57.955
  12:47:39 STEP 1043 H= 260:45 +CPU= 53.252
  12:48:46 STEP 1044 H= 261:00 +CPU= 66.515
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001044
  12:49:40 STEP 1045 H= 261:15 +CPU= 54.447
  12:50:36 STEP 1046 H= 261:30 +CPU= 55.400
  12:51:28 STEP 1047 H= 261:45 +CPU= 52.467
  12:52:24 STEP 1048 H= 262:00 +CPU= 55.372
  12:53:17 STEP 1049 H= 262:15 +CPU= 53.172
  12:54:20 STEP 1050 H= 262:30 +CPU= 63.556
  12:55:21 STEP 1051 H= 262:45 +CPU= 60.955
  12:56:24 STEP 1052 H= 263:00 +CPU= 63.183
  12:57:22 STEP 1053 H= 263:15 +CPU= 57.538
  12:58:17 STEP 1054 H= 263:30 +CPU= 55.517
  12:59:14 STEP 1055 H= 263:45 +CPU= 56.175
  13:00:34 STEP 1056 H= 264:00 +CPU= 80.223
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a394_2016092300_15_991_12211003_0_r1080860046_10.zip
Uploading the intermediate file: upload_file_10.zip
  13:01:35 STEP 1057 H= 264:15 +CPU= 61.179
  13:02:35 STEP 1058 H= 264:30 +CPU= 60.040
  13:03:35 STEP 1059 H= 264:45 +CPU= 60.147
  13:04:32 STEP 1060 H= 265:00 +CPU= 56.390
  13:05:30 STEP 1061 H= 265:15 +CPU= 58.306
  13:06:25 STEP 1062 H= 265:30 +CPU= 55.165
  13:07:21 STEP 1063 H= 265:45 +CPU= 55.526
  13:08:15 STEP 1064 H= 266:00 +CPU= 54.243
  13:09:10 STEP 1065 H= 266:15 +CPU= 54.553
  13:10:03 STEP 1066 H= 266:30 +CPU= 53.944
  13:11:00 STEP 1067 H= 266:45 +CPU= 56.587
  13:12:06 STEP 1068 H= 267:00 +CPU= 65.611
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001068
  13:13:02 STEP 1069 H= 267:15 +CPU= 56.421
  13:14:00 STEP 1070 H= 267:30 +CPU= 57.537
  13:15:00 STEP 1071 H= 267:45 +CPU= 60.435
  13:15:56 STEP 1072 H= 268:00 +CPU= 56.347
  13:16:52 STEP 1073 H= 268:15 +CPU= 55.248
  13:17:47 STEP 1074 H= 268:30 +CPU= 55.564
  13:18:44 STEP 1075 H= 268:45 +CPU= 56.436
  13:19:42 STEP 1076 H= 269:00 +CPU= 58.365
  13:20:38 STEP 1077 H= 269:15 +CPU= 56.313
  13:21:34 STEP 1078 H= 269:30 +CPU= 55.369
  13:22:29 STEP 1079 H= 269:45 +CPU= 54.905
  13:23:34 STEP 1080 H= 270:00 +CPU= 65.410
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001080
  13:24:30 STEP 1081 H= 270:15 +CPU= 56.224
  13:25:29 STEP 1082 H= 270:30 +CPU= 58.780
  13:26:30 STEP 1083 H= 270:45 +CPU= 60.842
  13:27:29 STEP 1084 H= 271:00 +CPU= 58.583
  13:28:26 STEP 1085 H= 271:15 +CPU= 57.599
  13:29:26 STEP 1086 H= 271:30 +CPU= 59.472
  13:30:25 STEP 1087 H= 271:45 +CPU= 59.206
  13:31:22 STEP 1088 H= 272:00 +CPU= 57.341
  13:32:17 STEP 1089 H= 272:15 +CPU= 54.630
  13:33:13 STEP 1090 H= 272:30 +CPU= 56.361
  13:34:14 STEP 1091 H= 272:45 +CPU= 60.161
  13:35:25 STEP 1092 H= 273:00 +CPU= 71.457
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001092
  13:36:25 STEP 1093 H= 273:15 +CPU= 59.853
  13:37:23 STEP 1094 H= 273:30 +CPU= 58.378
  13:38:22 STEP 1095 H= 273:45 +CPU= 58.937
  13:39:15 STEP 1096 H= 274:00 +CPU= 52.897
  13:40:10 STEP 1097 H= 274:15 +CPU= 55.173
  13:41:13 STEP 1098 H= 274:30 +CPU= 62.828
  13:42:16 STEP 1099 H= 274:45 +CPU= 62.491
  13:43:18 STEP 1100 H= 275:00 +CPU= 62.714
  13:44:20 STEP 1101 H= 275:15 +CPU= 62.134
  13:45:25 STEP 1102 H= 275:30 +CPU= 64.699
  13:46:27 STEP 1103 H= 275:45 +CPU= 61.641
  13:47:54 STEP 1104 H= 276:00 +CPU= 87.252
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001104
  13:48:48 STEP 1105 H= 276:15 +CPU= 54.286
  13:49:46 STEP 1106 H= 276:30 +CPU= 57.326
  13:50:47 STEP 1107 H= 276:45 +CPU= 61.636
  13:51:46 STEP 1108 H= 277:00 +CPU= 58.329
  13:52:43 STEP 1109 H= 277:15 +CPU= 57.282
  13:53:41 STEP 1110 H= 277:30 +CPU= 57.986
  13:54:40 STEP 1111 H= 277:45 +CPU= 58.867
  13:55:41 STEP 1112 H= 278:00 +CPU= 60.632
  13:56:41 STEP 1113 H= 278:15 +CPU= 60.676
  13:57:43 STEP 1114 H= 278:30 +CPU= 61.982
  13:58:43 STEP 1115 H= 278:45 +CPU= 59.287
  13:59:51 STEP 1116 H= 279:00 +CPU= 68.788
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001116
  14:00:52 STEP 1117 H= 279:15 +CPU= 60.536
  14:01:53 STEP 1118 H= 279:30 +CPU= 60.943
  14:02:54 STEP 1119 H= 279:45 +CPU= 61.576
  14:03:57 STEP 1120 H= 280:00 +CPU= 62.829
  14:04:57 STEP 1121 H= 280:15 +CPU= 59.633
  14:05:57 STEP 1122 H= 280:30 +CPU= 59.976
  14:06:54 STEP 1123 H= 280:45 +CPU= 57.430
  14:07:48 STEP 1124 H= 281:00 +CPU= 54.147
  14:08:48 STEP 1125 H= 281:15 +CPU= 59.214
  14:09:48 STEP 1126 H= 281:30 +CPU= 60.461
  14:10:47 STEP 1127 H= 281:45 +CPU= 59.058
  14:11:51 STEP 1128 H= 282:00 +CPU= 63.703
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001128
  14:12:44 STEP 1129 H= 282:15 +CPU= 52.589
  14:13:38 STEP 1130 H= 282:30 +CPU= 54.048
  14:14:33 STEP 1131 H= 282:45 +CPU= 55.063
  14:15:30 STEP 1132 H= 283:00 +CPU= 57.674
  14:16:25 STEP 1133 H= 283:15 +CPU= 54.954
  14:17:25 STEP 1134 H= 283:30 +CPU= 59.984
  14:18:23 STEP 1135 H= 283:45 +CPU= 57.922
  14:19:19 STEP 1136 H= 284:00 +CPU= 55.874
  14:20:20 STEP 1137 H= 284:15 +CPU= 60.920
  14:21:20 STEP 1138 H= 284:30 +CPU= 59.677
  14:22:15 STEP 1139 H= 284:45 +CPU= 54.681
  14:23:17 STEP 1140 H= 285:00 +CPU= 62.500
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001140
  14:24:17 STEP 1141 H= 285:15 +CPU= 59.465
  14:25:13 STEP 1142 H= 285:30 +CPU= 56.948
  14:26:08 STEP 1143 H= 285:45 +CPU= 54.736
  14:27:11 STEP 1144 H= 286:00 +CPU= 62.505
  14:28:10 STEP 1145 H= 286:15 +CPU= 59.681
  14:29:09 STEP 1146 H= 286:30 +CPU= 58.586
  14:30:01 STEP 1147 H= 286:45 +CPU= 52.179
  14:31:02 STEP 1148 H= 287:00 +CPU= 60.306
  14:31:54 STEP 1149 H= 287:15 +CPU= 52.878
  14:32:49 STEP 1150 H= 287:30 +CPU= 54.745
  14:33:44 STEP 1151 H= 287:45 +CPU= 54.672
  14:34:53 STEP 1152 H= 288:00 +CPU= 69.515
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211003/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a394_2016092300_15_991_12211003_0_r1080860046_11.zip
Uploading the intermediate file: upload_file_11.zip
  14:35:46 STEP 1153 H= 288:15 +CPU= 52.180
  14:36:38 STEP 1154 H= 288:30 +CPU= 52.251
  14:37:34 STEP 1155 H= 288:45 +CPU= 55.723
  14:38:31 STEP 1156 H= 289:00 +CPU= 57.045
  14:39:29 STEP 1157 H= 289:15 +CPU= 58.101
  14:40:27 STEP 1158 H= 289:30 +CPU= 58.507
  14:41:26 STEP 1159 H= 289:45 +CPU= 58.154
  14:42:23 STEP 1160 H= 290:00 +CPU= 57.220
  14:43:17 STEP 1161 H= 290:15 +CPU= 54.253
  14:44:13 STEP 1162 H= 290:30 +CPU= 55.808
  14:45:07 STEP 1163 H= 290:45 +CPU= 53.862
  14:46:12 STEP 1164 H= 291:00 +CPU= 65.533
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001164
  14:47:08 STEP 1165 H= 291:15 +CPU= 55.960
  14:48:07 STEP 1166 H= 291:30 +CPU= 59.118
  14:49:07 STEP 1167 H= 291:45 +CPU= 59.716
  14:50:05 STEP 1168 H= 292:00 +CPU= 57.541
  14:50:55 STEP 1169 H= 292:15 +CPU= 50.415
  14:51:48 STEP 1170 H= 292:30 +CPU= 53.268
  14:52:42 STEP 1171 H= 292:45 +CPU= 53.749
  14:53:41 STEP 1172 H= 293:00 +CPU= 58.454
  14:54:35 STEP 1173 H= 293:15 +CPU= 54.139
  14:55:28 STEP 1174 H= 293:30 +CPU= 52.902
  14:56:25 STEP 1175 H= 293:45 +CPU= 57.361
  14:57:33 STEP 1176 H= 294:00 +CPU= 67.517
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+001176
  14:58:27 STEP 1177 H= 294:15 +CPU= 53.982
  14:59:18 STEP 1178 H= 294:30 +CPU= 51.788
  15:00:12 STEP 1179 H= 294:45 +CPU= 53.135
  15:01:01 STEP 1180 H= 295:00 +CPU= 49.610
  15:01:55 STEP 1181 H= 295:15 +CPU= 54.020
  15:02:49 STEP 1182 H= 295:30 +CPU= 53.547
  15:03:40 STEP 1183 H= 295:45 +CPU= 51.403
  15:04:33 STEP 1184 H= 296:00 +CPU= 52.565
  15:05:27 STEP 1185 H= 296:15 +CPU= 54.445
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.521] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 4362MB (heap), 5067MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.522] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.522] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f0c0 (old at (nil))
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.522] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc4a8f], nsigs = 1
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.522] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.522] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3843] 4362 MB (maxheap), 5067 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.522] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150600:1676469960:67962.523] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/22/oifs_43r3_model.exe' : sigcontextptr=0x7ffd0862e650
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 24) with plain hex-dump
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [0]: [0x838803] : 0x838803
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [1]: [0x838e09] : 0x838e09
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [2]: [0x1d8bb20] : 0x1d8bb20
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [3]: [0x1cc4a8f] : 0x1cc4a8f
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [4]: [0x12d283b] : 0x12d283b
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [5]: [0x12ba8d9] : 0x12ba8d9
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [6]: [0x12b0762] : 0x12b0762
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [7]: [0x1163fc3] : 0x1163fc3
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [8]: [0x10b682f] : 0x10b682f
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [9]: [0x10a915c] : 0x10a915c
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [10]: [0xe680c3] : 0xe680c3
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [11]: [0x15c1e62] : 0x15c1e62
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [12]: [0x15ac40e] : 0x15ac40e
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [13]: [0x6620b4] : 0x6620b4
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [14]: [0x4a0e7c] : 0x4a0e7c
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [15]: [0x49dc29] : 0x49dc29
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [16]: [0x49ce72] : 0x49ce72
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [17]: [0x49cd68] : 0x49cd68
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [18]: [0x4078ad] : 0x4078ad
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [19]: [0x40276f] : 0x40276f
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [20]: [0x40270d] : 0x40270d
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [21]: [0x1d8287a] : 0x1d8287a
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [22]: [0x1d840d7] : 0x1d840d7
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [23]: [0x4025e5] : 0x4025e5
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.523] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/22/oifs_43r3_model.exe' : sigcontextptr=0x7ffd0862e560
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 25) with plain hex-dump
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [0]: [0x838803] : 0x838803
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [1]: [0x838e09] : 0x838e09
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [2]: [0x820d70] : 0x820d70
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [3]: [0x1d8bb20] : 0x1d8bb20
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [4]: [0x1cc4a8f] : 0x1cc4a8f
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [5]: [0x12d283b] : 0x12d283b
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [6]: [0x12ba8d9] : 0x12ba8d9
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [7]: [0x12b0762] : 0x12b0762
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [8]: [0x1163fc3] : 0x1163fc3
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [9]: [0x10b682f] : 0x10b682f
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [10]: [0x10a915c] : 0x10a915c
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [11]: [0xe680c3] : 0xe680c3
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [12]: [0x15c1e62] : 0x15c1e62
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [13]: [0x15ac40e] : 0x15ac40e
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [14]: [0x6620b4] : 0x6620b4
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [15]: [0x4a0e7c] : 0x4a0e7c
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [16]: [0x49dc29] : 0x49dc29
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [17]: [0x49ce72] : 0x49ce72
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [18]: [0x49cd68] : 0x49cd68
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [19]: [0x4078ad] : 0x4078ad
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [20]: [0x40276f] : 0x40276f
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [21]: [0x40270d] : 0x40270d
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [22]: [0x1d8287a] : 0x1d8287a
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [23]: [0x1d840d7] : 0x1d840d7
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:340] [24]: [0x4025e5] : 0x4025e5
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.546] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:Dinger:1:1:153939:153939] [20230215:150601:1676469961:67963.546] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x401140 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  000000000081FCF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D8BB20  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC4A8F  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000012D283B  vdfmain_                  874  vdfmain.F90
oifs_43r3_model.e  00000000012BA8D9  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000012B0762  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000001163FC3  callpar_                  869  callpar.F90
oifs_43r3_model.e  00000000010B682F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  00000000010A915C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000E680C3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000015C1E62  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  00000000015AC40E  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  00000000006620B4  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004A0E7C  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  000000000049DC29  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  000000000049CE72  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049CD68  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  00000000004078AD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040276F  MAIN__                     96  master.F90
oifs_43r3_model.e  000000000040270D  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D8287A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D840D7  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000004025E5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

         AVE   0.187203847684983E-04 0.000000000000000E+00 0.290932832969392E-01
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.163686606708671E-04 0.000000000000000E+00 0.660972317641777E-02
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.382569854061359E-05 0.000000000000000E+00 0.169659843980495E-01
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.121154413827292E-05 0.000000000000000E+00 0.880824377434871E-02
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.170347378056560E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1176 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1176 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001176 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001176
 MPI-TASK:   1 -    309424746 BYTES IN      6 RECORDS TRANSFERRED IN    0.0012 SECONDS257854.0 Mbytes/s, TOTAL TIME=    0.0163( 7.4%)
 NSTEP =  1176 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001176 MODE=a
 IO-STREAM CLOSED - ./ICMSHh7zg+001176
 MPI-TASK:   1 -    707765250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0242 SECONDS 29246.5 Mbytes/s, TOTAL TIME=    0.3292( 7.4%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001176 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001176
 MPI-TASK:   1 -   4950795927 BYTES IN    125 RECORDS TRANSFERRED IN    0.0195 SECONDS253887.0 Mbytes/s, TOTAL TIME=    0.2910( 6.7%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001176 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001176
 MPI-TASK:   1 -    300851113 BYTES IN      2 RECORDS TRANSFERRED IN    0.0004 SECONDS752127.8 Mbytes/s, TOTAL TIME=    0.0081( 4.9%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001176 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001176
 MPI-TASK:   1 -    309424746 BYTES IN      4 RECORDS TRANSFERRED IN    0.0008 SECONDS386780.9 Mbytes/s, TOTAL TIME=    0.0111( 7.2%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1176 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1176 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001176 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001176
 MPI-TASK:   1 -    309424737 BYTES IN      7 RECORDS TRANSFERRED IN    0.0010 SECONDS309424.7 Mbytes/s, TOTAL TIME=    0.0178( 5.6%)
 14:57:33 STEP 1176 H= 294:00 +CPU= 67.517
 NSTEP =  1177 STEPO   0AAA00AAA
 14:58:27 STEP 1177 H= 294:15 +CPU= 53.982
 NSTEP =  1178 STEPO   0AAA00AAA
 14:59:18 STEP 1178 H= 294:30 +CPU= 51.788
 NSTEP =  1179 STEPO   0AAA00AAA
 15:00:12 STEP 1179 H= 294:45 +CPU= 53.135
ISTEP=    1180 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1180 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1180 STEPO   0AAA00AAA
 15:01:01 STEP 1180 H= 295:00 +CPU= 49.610
 NSTEP =  1181 STEPO   0AAA00AAA
 15:01:55 STEP 1181 H= 295:15 +CPU= 54.020
 NSTEP =  1182 STEPO   0AAA00AAA
 15:02:49 STEP 1182 H= 295:30 +CPU= 53.547
 NSTEP =  1183 STEPO   0AAA00AAA
 15:03:40 STEP 1183 H= 295:45 +CPU= 51.403
 NSTEP =  1184 STEPO   0AAA00AAA
 15:04:33 STEP 1184 H= 296:00 +CPU= 52.565
ISTEP=    1185 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1185 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1185 STEPO   0AAA00AAA
 15:05:27 STEP 1185 H= 296:15 +CPU= 54.445
 NSTEP =  1186 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 15:00:12 0AAA00AAA STEPO     1180    53.135   53.135   53.146   1126:35   1126:53 0.33815624078691E-04       2GB       0MB
 15:01:01 0AAA00AAA STEPO     1181    49.611   49.611   49.629   1127:25   1127:43 0.33815624078691E-04       2GB       0MB
 15:01:55 0AAA00AAA STEPO     1182    54.021   54.021   54.037   1128:19   1128:37 0.33815624078691E-04       2GB       0MB
 15:02:49 0AAA00AAA STEPO     1183    53.547   53.547   53.557   1129:12   1129:31 0.33815624078691E-04       2GB       0MB
 15:03:40 0AAA00AAA STEPO     1184    51.404   51.404   51.424   1130:04   1130:22 0.33815624078691E-04       2GB       0MB
 15:04:33 0AAA00AAA STEPO     1185    52.566   52.566   52.600   1130:56   1131:15 0.33815624078691E-04       2GB       0MB
 15:05:27 0AAA00AAA STEPO     1186    54.445   54.445   54.450   1131:51   1132:09 0.33815624078691E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1152",
 CTIME   = "00120000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1140, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/22/progress_file_12211003.xml

<running_values>
  <last_cpu_time>67944.070000</last_cpu_time>
  <upload_file_number>12</upload_file_number>
  <last_iter>1186</last_iter>
  <last_upload>1036800</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:51:38 1518460 22311240 oifs_43r3_bl_a394_2016092300_15_991_12211003_0 864,000 54,765 0.0634


©2024 climateprediction.net