climateprediction.net home page
Task 22289328

Task 22289328

Name oifs_43r3_ps_0261_2017050100_123_986_12202905_0
Workunit 12202905
Created 24 Dec 2022, 17:33:25 UTC
Sent 29 Dec 2022, 7:58:33 UTC
Report deadline 28 Jan 2023, 7:58:33 UTC
Received 3 Jan 2023, 18:56:16 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 151 (0x00000097) Unknown error code
Computer ID 1537334
Run time 13 hours 37 min 32 sec
CPU time 13 hours 31 min 29 sec
Validate state Invalid
Credit 6,894.72
Device peak FLOPS 3.01 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak disk usage 4,019.08 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 151 (0x97, -105)</message>
<stderr_txt>
P  989 H= 989:00 +CPU= 26.106
  10:14:28 STEP  990 H= 990:00 +CPU= 42.184
  10:14:55 STEP  991 H= 991:00 +CPU= 26.159
  10:15:21 STEP  992 H= 992:00 +CPU= 26.568
  10:16:03 STEP  993 H= 993:00 +CPU= 41.508
  10:16:30 STEP  994 H= 994:00 +CPU= 26.817
  10:16:56 STEP  995 H= 995:00 +CPU= 26.037
  10:17:42 STEP  996 H= 996:00 +CPU= 45.509
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+000996
  10:18:08 STEP  997 H= 997:00 +CPU= 26.291
  10:18:35 STEP  998 H= 998:00 +CPU= 26.778
  10:19:17 STEP  999 H= 999:00 +CPU= 41.417
  10:19:44 STEP 1000 H=1000:00 +CPU= 26.752
  10:20:10 STEP 1001 H=1001:00 +CPU= 26.139
  10:20:52 STEP 1002 H=1002:00 +CPU= 42.181
  10:21:19 STEP 1003 H=1003:00 +CPU= 26.057
  10:21:45 STEP 1004 H=1004:00 +CPU= 26.667
  10:22:27 STEP 1005 H=1005:00 +CPU= 41.374
  10:22:54 STEP 1006 H=1006:00 +CPU= 26.661
  10:23:21 STEP 1007 H=1007:00 +CPU= 26.930
  10:24:10 STEP 1008 H=1008:00 +CPU= 48.680
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_41.zip
Uploading the intermediate file: upload_file_41.zip
  10:24:36 STEP 1009 H=1009:00 +CPU= 26.126
  10:25:03 STEP 1010 H=1010:00 +CPU= 26.751
  10:25:45 STEP 1011 H=1011:00 +CPU= 41.211
  10:26:12 STEP 1012 H=1012:00 +CPU= 26.672
  10:26:38 STEP 1013 H=1013:00 +CPU= 25.979
  10:27:20 STEP 1014 H=1014:00 +CPU= 42.069
  10:27:46 STEP 1015 H=1015:00 +CPU= 26.102
  10:28:13 STEP 1016 H=1016:00 +CPU= 26.765
  10:28:54 STEP 1017 H=1017:00 +CPU= 41.134
  10:29:21 STEP 1018 H=1018:00 +CPU= 26.705
  10:29:48 STEP 1019 H=1019:00 +CPU= 26.088
  10:30:33 STEP 1020 H=1020:00 +CPU= 45.356
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001020
  10:30:59 STEP 1021 H=1021:00 +CPU= 25.987
  10:31:26 STEP 1022 H=1022:00 +CPU= 26.697
  10:32:07 STEP 1023 H=1023:00 +CPU= 41.197
  10:32:34 STEP 1024 H=1024:00 +CPU= 26.551
  10:33:00 STEP 1025 H=1025:00 +CPU= 26.115
  10:33:43 STEP 1026 H=1026:00 +CPU= 42.068
  10:34:09 STEP 1027 H=1027:00 +CPU= 25.916
  10:34:36 STEP 1028 H=1028:00 +CPU= 26.709
  10:35:17 STEP 1029 H=1029:00 +CPU= 41.318
  10:35:44 STEP 1030 H=1030:00 +CPU= 26.488
  10:36:11 STEP 1031 H=1031:00 +CPU= 26.748
  10:37:00 STEP 1032 H=1032:00 +CPU= 48.686
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_42.zip
Uploading the intermediate file: upload_file_42.zip
  10:37:26 STEP 1033 H=1033:00 +CPU= 26.115
  10:37:53 STEP 1034 H=1034:00 +CPU= 26.726
  10:38:35 STEP 1035 H=1035:00 +CPU= 41.296
  10:39:01 STEP 1036 H=1036:00 +CPU= 26.669
  10:39:27 STEP 1037 H=1037:00 +CPU= 25.947
  10:40:10 STEP 1038 H=1038:00 +CPU= 42.146
  10:40:36 STEP 1039 H=1039:00 +CPU= 26.129
  10:41:03 STEP 1040 H=1040:00 +CPU= 26.752
  10:41:44 STEP 1041 H=1041:00 +CPU= 41.289
  10:42:11 STEP 1042 H=1042:00 +CPU= 26.656
  10:42:37 STEP 1043 H=1043:00 +CPU= 25.949
  10:43:23 STEP 1044 H=1044:00 +CPU= 45.477
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001044
  10:43:49 STEP 1045 H=1045:00 +CPU= 26.039
  10:44:16 STEP 1046 H=1046:00 +CPU= 26.602
  10:44:57 STEP 1047 H=1047:00 +CPU= 41.293
  10:45:24 STEP 1048 H=1048:00 +CPU= 26.678
  10:45:50 STEP 1049 H=1049:00 +CPU= 25.922
  10:46:32 STEP 1050 H=1050:00 +CPU= 41.916
  10:46:58 STEP 1051 H=1051:00 +CPU= 25.950
  10:47:25 STEP 1052 H=1052:00 +CPU= 26.553
  10:48:06 STEP 1053 H=1053:00 +CPU= 41.006
  10:48:33 STEP 1054 H=1054:00 +CPU= 26.473
  10:49:00 STEP 1055 H=1055:00 +CPU= 26.820
  10:49:48 STEP 1056 H=1056:00 +CPU= 48.374
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_43.zip
Uploading the intermediate file: upload_file_43.zip
  10:50:15 STEP 1057 H=1057:00 +CPU= 25.993
  10:50:42 STEP 1058 H=1058:00 +CPU= 26.489
  10:51:23 STEP 1059 H=1059:00 +CPU= 40.909
  10:51:49 STEP 1060 H=1060:00 +CPU= 26.481
  10:52:15 STEP 1061 H=1061:00 +CPU= 25.927
  10:52:57 STEP 1062 H=1062:00 +CPU= 41.839
  10:53:24 STEP 1063 H=1063:00 +CPU= 25.945
  10:53:50 STEP 1064 H=1064:00 +CPU= 26.579
  10:54:32 STEP 1065 H=1065:00 +CPU= 41.283
  10:54:59 STEP 1066 H=1066:00 +CPU= 26.571
  10:55:25 STEP 1067 H=1067:00 +CPU= 26.063
  10:56:10 STEP 1068 H=1068:00 +CPU= 45.430
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001068
  10:56:37 STEP 1069 H=1069:00 +CPU= 26.017
  10:57:03 STEP 1070 H=1070:00 +CPU= 26.632
  10:57:45 STEP 1071 H=1071:00 +CPU= 41.305
  10:58:12 STEP 1072 H=1072:00 +CPU= 26.629
  10:58:38 STEP 1073 H=1073:00 +CPU= 25.938
  10:59:20 STEP 1074 H=1074:00 +CPU= 41.965
  10:59:46 STEP 1075 H=1075:00 +CPU= 25.993
  11:00:13 STEP 1076 H=1076:00 +CPU= 26.628
  11:00:54 STEP 1077 H=1077:00 +CPU= 41.187
  11:01:21 STEP 1078 H=1078:00 +CPU= 26.507
  11:01:48 STEP 1079 H=1079:00 +CPU= 26.735
  11:02:37 STEP 1080 H=1080:00 +CPU= 48.654
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_44.zip
Uploading the intermediate file: upload_file_44.zip
  11:03:03 STEP 1081 H=1081:00 +CPU= 25.830
  11:03:30 STEP 1082 H=1082:00 +CPU= 26.418
  11:04:11 STEP 1083 H=1083:00 +CPU= 41.044
  11:04:38 STEP 1084 H=1084:00 +CPU= 26.390
  11:05:04 STEP 1085 H=1085:00 +CPU= 25.886
  11:05:46 STEP 1086 H=1086:00 +CPU= 41.821
  11:06:12 STEP 1087 H=1087:00 +CPU= 25.853
  11:06:38 STEP 1088 H=1088:00 +CPU= 26.514
  11:07:20 STEP 1089 H=1089:00 +CPU= 41.231
  11:07:47 STEP 1090 H=1090:00 +CPU= 26.617
  11:08:13 STEP 1091 H=1091:00 +CPU= 25.873
  11:08:58 STEP 1092 H=1092:00 +CPU= 45.208
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001092
  11:09:24 STEP 1093 H=1093:00 +CPU= 25.863
  11:09:51 STEP 1094 H=1094:00 +CPU= 26.508
  11:10:32 STEP 1095 H=1095:00 +CPU= 41.149
  11:10:59 STEP 1096 H=1096:00 +CPU= 26.474
  11:11:25 STEP 1097 H=1097:00 +CPU= 25.788
  11:12:07 STEP 1098 H=1098:00 +CPU= 41.873
  11:12:33 STEP 1099 H=1099:00 +CPU= 25.898
  11:12:59 STEP 1100 H=1100:00 +CPU= 26.372
  11:13:40 STEP 1101 H=1101:00 +CPU= 40.914
  11:14:07 STEP 1102 H=1102:00 +CPU= 26.340
  11:14:33 STEP 1103 H=1103:00 +CPU= 26.394
  11:15:22 STEP 1104 H=1104:00 +CPU= 48.349
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_45.zip
Uploading the intermediate file: upload_file_45.zip
  11:15:48 STEP 1105 H=1105:00 +CPU= 25.918
  11:16:15 STEP 1106 H=1106:00 +CPU= 26.320
  11:16:56 STEP 1107 H=1107:00 +CPU= 40.900
  11:17:23 STEP 1108 H=1108:00 +CPU= 26.450
  11:17:49 STEP 1109 H=1109:00 +CPU= 25.737
  11:18:31 STEP 1110 H=1110:00 +CPU= 41.845
  11:18:57 STEP 1111 H=1111:00 +CPU= 25.813
  11:19:23 STEP 1112 H=1112:00 +CPU= 26.434
  11:20:05 STEP 1113 H=1113:00 +CPU= 41.196
  11:20:31 STEP 1114 H=1114:00 +CPU= 26.554
  11:20:57 STEP 1115 H=1115:00 +CPU= 25.780
  11:21:43 STEP 1116 H=1116:00 +CPU= 45.189
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001116
  11:22:09 STEP 1117 H=1117:00 +CPU= 25.881
  11:22:35 STEP 1118 H=1118:00 +CPU= 26.545
  11:23:17 STEP 1119 H=1119:00 +CPU= 41.115
  11:23:43 STEP 1120 H=1120:00 +CPU= 26.482
  11:24:09 STEP 1121 H=1121:00 +CPU= 25.781
  11:24:51 STEP 1122 H=1122:00 +CPU= 42.074
  11:25:18 STEP 1123 H=1123:00 +CPU= 25.976
  11:25:44 STEP 1124 H=1124:00 +CPU= 26.515
  11:26:27 STEP 1125 H=1125:00 +CPU= 42.122
  11:26:54 STEP 1126 H=1126:00 +CPU= 26.591
  11:27:20 STEP 1127 H=1127:00 +CPU= 26.700
  11:28:09 STEP 1128 H=1128:00 +CPU= 48.742
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_46.zip
Uploading the intermediate file: upload_file_46.zip
  11:28:36 STEP 1129 H=1129:00 +CPU= 26.112
Uploading trickle at timestep: 4060800
  11:29:03 STEP 1130 H=1130:00 +CPU= 26.556
  11:29:44 STEP 1131 H=1131:00 +CPU= 41.164
  11:30:11 STEP 1132 H=1132:00 +CPU= 26.585
  11:30:37 STEP 1133 H=1133:00 +CPU= 26.037
  11:31:20 STEP 1134 H=1134:00 +CPU= 42.240
  11:31:46 STEP 1135 H=1135:00 +CPU= 26.214
  11:32:13 STEP 1136 H=1136:00 +CPU= 26.549
  11:32:54 STEP 1137 H=1137:00 +CPU= 41.326
  11:33:21 STEP 1138 H=1138:00 +CPU= 26.567
  11:33:47 STEP 1139 H=1139:00 +CPU= 25.858
  11:34:33 STEP 1140 H=1140:00 +CPU= 45.370
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001140
  11:34:59 STEP 1141 H=1141:00 +CPU= 25.967
  11:35:26 STEP 1142 H=1142:00 +CPU= 26.602
  11:36:07 STEP 1143 H=1143:00 +CPU= 41.147
  11:36:34 STEP 1144 H=1144:00 +CPU= 26.553
  11:37:00 STEP 1145 H=1145:00 +CPU= 25.908
  11:37:42 STEP 1146 H=1146:00 +CPU= 42.014
  11:38:08 STEP 1147 H=1147:00 +CPU= 26.071
  11:38:35 STEP 1148 H=1148:00 +CPU= 26.598
  11:39:16 STEP 1149 H=1149:00 +CPU= 41.099
  11:39:43 STEP 1150 H=1150:00 +CPU= 26.544
  11:40:10 STEP 1151 H=1151:00 +CPU= 26.680
  11:40:59 STEP 1152 H=1152:00 +CPU= 48.760
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_47.zip
Uploading the intermediate file: upload_file_47.zip
  11:41:25 STEP 1153 H=1153:00 +CPU= 26.175
  11:41:52 STEP 1154 H=1154:00 +CPU= 26.527
  11:42:33 STEP 1155 H=1155:00 +CPU= 41.185
  11:43:00 STEP 1156 H=1156:00 +CPU= 26.656
  11:43:26 STEP 1157 H=1157:00 +CPU= 26.238
  11:44:09 STEP 1158 H=1158:00 +CPU= 42.099
  11:44:35 STEP 1159 H=1159:00 +CPU= 26.129
  11:45:02 STEP 1160 H=1160:00 +CPU= 26.769
  11:45:43 STEP 1161 H=1161:00 +CPU= 41.351
  11:46:10 STEP 1162 H=1162:00 +CPU= 26.774
  11:46:36 STEP 1163 H=1163:00 +CPU= 25.997
  11:47:22 STEP 1164 H=1164:00 +CPU= 45.376
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001164
  11:47:48 STEP 1165 H=1165:00 +CPU= 26.153
  11:48:15 STEP 1166 H=1166:00 +CPU= 26.710
  11:48:57 STEP 1167 H=1167:00 +CPU= 41.266
  11:49:23 STEP 1168 H=1168:00 +CPU= 26.605
  11:49:50 STEP 1169 H=1169:00 +CPU= 25.870
  11:50:32 STEP 1170 H=1170:00 +CPU= 41.862
  11:50:58 STEP 1171 H=1171:00 +CPU= 25.928
  11:51:24 STEP 1172 H=1172:00 +CPU= 26.542
  11:52:06 STEP 1173 H=1173:00 +CPU= 41.189
  11:52:32 STEP 1174 H=1174:00 +CPU= 26.549
  11:52:59 STEP 1175 H=1175:00 +CPU= 26.790
  11:53:48 STEP 1176 H=1176:00 +CPU= 48.643
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_48.zip
Uploading the intermediate file: upload_file_48.zip
  11:54:15 STEP 1177 H=1177:00 +CPU= 26.251
  11:54:42 STEP 1178 H=1178:00 +CPU= 26.660
  11:55:23 STEP 1179 H=1179:00 +CPU= 41.345
  11:55:50 STEP 1180 H=1180:00 +CPU= 26.733
  11:56:17 STEP 1181 H=1181:00 +CPU= 26.184
  11:56:59 STEP 1182 H=1182:00 +CPU= 41.999
  11:57:25 STEP 1183 H=1183:00 +CPU= 26.146
  11:57:52 STEP 1184 H=1184:00 +CPU= 26.746
  11:58:34 STEP 1185 H=1185:00 +CPU= 41.253
  11:59:00 STEP 1186 H=1186:00 +CPU= 26.573
  11:59:26 STEP 1187 H=1187:00 +CPU= 25.993
  12:00:12 STEP 1188 H=1188:00 +CPU= 45.476
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001188
  12:00:38 STEP 1189 H=1189:00 +CPU= 26.265
  12:01:06 STEP 1190 H=1190:00 +CPU= 26.868
  12:01:47 STEP 1191 H=1191:00 +CPU= 41.205
  12:02:14 STEP 1192 H=1192:00 +CPU= 26.552
  12:02:40 STEP 1193 H=1193:00 +CPU= 26.093
  12:03:22 STEP 1194 H=1194:00 +CPU= 42.122
  12:03:49 STEP 1195 H=1195:00 +CPU= 26.033
  12:04:15 STEP 1196 H=1196:00 +CPU= 26.756
  12:04:57 STEP 1197 H=1197:00 +CPU= 41.386
  12:05:24 STEP 1198 H=1198:00 +CPU= 26.873
  12:05:51 STEP 1199 H=1199:00 +CPU= 26.844
  12:06:40 STEP 1200 H=1200:00 +CPU= 48.878
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_49.zip
Uploading the intermediate file: upload_file_49.zip
  12:07:07 STEP 1201 H=1201:00 +CPU= 26.355
  12:07:34 STEP 1202 H=1202:00 +CPU= 26.811
  12:08:16 STEP 1203 H=1203:00 +CPU= 41.347
  12:08:42 STEP 1204 H=1204:00 +CPU= 26.731
  12:09:09 STEP 1205 H=1205:00 +CPU= 26.336
  12:09:51 STEP 1206 H=1206:00 +CPU= 42.136
  12:10:18 STEP 1207 H=1207:00 +CPU= 26.197
  12:10:45 STEP 1208 H=1208:00 +CPU= 26.746
  12:11:26 STEP 1209 H=1209:00 +CPU= 41.210
  12:11:53 STEP 1210 H=1210:00 +CPU= 26.622
  12:12:19 STEP 1211 H=1211:00 +CPU= 26.000
  12:13:05 STEP 1212 H=1212:00 +CPU= 45.758
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001212
  12:13:31 STEP 1213 H=1213:00 +CPU= 26.101
  12:13:58 STEP 1214 H=1214:00 +CPU= 26.887
  12:14:40 STEP 1215 H=1215:00 +CPU= 41.167
  12:15:06 STEP 1216 H=1216:00 +CPU= 26.666
  12:15:33 STEP 1217 H=1217:00 +CPU= 26.196
  12:16:15 STEP 1218 H=1218:00 +CPU= 42.243
  12:16:42 STEP 1219 H=1219:00 +CPU= 26.158
  12:17:08 STEP 1220 H=1220:00 +CPU= 26.646
  12:17:50 STEP 1221 H=1221:00 +CPU= 41.322
  12:18:17 STEP 1222 H=1222:00 +CPU= 26.752
  12:18:44 STEP 1223 H=1223:00 +CPU= 26.774
  12:19:33 STEP 1224 H=1224:00 +CPU= 48.487
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_50.zip
Uploading the intermediate file: upload_file_50.zip
  12:19:59 STEP 1225 H=1225:00 +CPU= 25.992
  12:20:26 STEP 1226 H=1226:00 +CPU= 26.608
  12:21:07 STEP 1227 H=1227:00 +CPU= 41.221
  12:21:34 STEP 1228 H=1228:00 +CPU= 26.604
  12:22:00 STEP 1229 H=1229:00 +CPU= 25.985
  12:22:42 STEP 1230 H=1230:00 +CPU= 41.861
  12:23:08 STEP 1231 H=1231:00 +CPU= 26.128
  12:23:35 STEP 1232 H=1232:00 +CPU= 26.724
  12:24:17 STEP 1233 H=1233:00 +CPU= 41.293
  12:24:44 STEP 1234 H=1234:00 +CPU= 26.705
  12:25:10 STEP 1235 H=1235:00 +CPU= 26.054
  12:25:55 STEP 1236 H=1236:00 +CPU= 45.317
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001236
  12:26:22 STEP 1237 H=1237:00 +CPU= 26.059
  12:26:49 STEP 1238 H=1238:00 +CPU= 26.740
  12:27:30 STEP 1239 H=1239:00 +CPU= 41.119
  12:27:57 STEP 1240 H=1240:00 +CPU= 26.649
  12:28:23 STEP 1241 H=1241:00 +CPU= 25.905
  12:29:05 STEP 1242 H=1242:00 +CPU= 41.928
  12:29:31 STEP 1243 H=1243:00 +CPU= 25.813
  12:29:57 STEP 1244 H=1244:00 +CPU= 26.479
  12:30:39 STEP 1245 H=1245:00 +CPU= 41.102
  12:31:05 STEP 1246 H=1246:00 +CPU= 26.569
  12:31:32 STEP 1247 H=1247:00 +CPU= 26.811
  12:32:21 STEP 1248 H=1248:00 +CPU= 48.502
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_51.zip
Uploading the intermediate file: upload_file_51.zip
  12:32:48 STEP 1249 H=1249:00 +CPU= 26.007
  12:33:15 STEP 1250 H=1250:00 +CPU= 26.598
  12:33:56 STEP 1251 H=1251:00 +CPU= 41.148
  12:34:23 STEP 1252 H=1252:00 +CPU= 26.607
  12:34:49 STEP 1253 H=1253:00 +CPU= 25.915
  12:35:31 STEP 1254 H=1254:00 +CPU= 41.866
  12:35:57 STEP 1255 H=1255:00 +CPU= 26.104
  12:36:24 STEP 1256 H=1256:00 +CPU= 26.618
  12:37:05 STEP 1257 H=1257:00 +CPU= 41.246
  12:37:32 STEP 1258 H=1258:00 +CPU= 26.723
  12:37:58 STEP 1259 H=1259:00 +CPU= 25.949
  12:38:44 STEP 1260 H=1260:00 +CPU= 45.501
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001260
  12:39:10 STEP 1261 H=1261:00 +CPU= 26.038
  12:39:37 STEP 1262 H=1262:00 +CPU= 26.647
  12:40:18 STEP 1263 H=1263:00 +CPU= 41.207
  12:40:45 STEP 1264 H=1264:00 +CPU= 26.565
  12:41:11 STEP 1265 H=1265:00 +CPU= 26.122
  12:41:53 STEP 1266 H=1266:00 +CPU= 41.805
  12:42:20 STEP 1267 H=1267:00 +CPU= 25.952
  12:42:46 STEP 1268 H=1268:00 +CPU= 26.552
  12:43:28 STEP 1269 H=1269:00 +CPU= 41.099
  12:43:54 STEP 1270 H=1270:00 +CPU= 26.625
  12:44:21 STEP 1271 H=1271:00 +CPU= 26.725
  12:45:10 STEP 1272 H=1272:00 +CPU= 48.559
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_52.zip
Uploading the intermediate file: upload_file_52.zip
  12:45:36 STEP 1273 H=1273:00 +CPU= 25.894
  12:46:03 STEP 1274 H=1274:00 +CPU= 26.578
  12:46:44 STEP 1275 H=1275:00 +CPU= 41.119
  12:47:11 STEP 1276 H=1276:00 +CPU= 26.530
  12:47:37 STEP 1277 H=1277:00 +CPU= 25.928
  12:48:19 STEP 1278 H=1278:00 +CPU= 42.043
  12:48:46 STEP 1279 H=1279:00 +CPU= 26.168
  12:49:13 STEP 1280 H=1280:00 +CPU= 26.584
  12:49:54 STEP 1281 H=1281:00 +CPU= 41.389
  12:50:21 STEP 1282 H=1282:00 +CPU= 26.691
  12:50:47 STEP 1283 H=1283:00 +CPU= 26.008
  12:51:33 STEP 1284 H=1284:00 +CPU= 45.261
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001284
  12:51:59 STEP 1285 H=1285:00 +CPU= 26.015
  12:52:26 STEP 1286 H=1286:00 +CPU= 26.594
  12:53:07 STEP 1287 H=1287:00 +CPU= 41.208
  12:53:34 STEP 1288 H=1288:00 +CPU= 26.648
  12:54:00 STEP 1289 H=1289:00 +CPU= 25.889
  12:54:42 STEP 1290 H=1290:00 +CPU= 41.865
  12:55:08 STEP 1291 H=1291:00 +CPU= 25.834
  12:55:34 STEP 1292 H=1292:00 +CPU= 26.417
  12:56:16 STEP 1293 H=1293:00 +CPU= 41.123
  12:56:42 STEP 1294 H=1294:00 +CPU= 26.512
  12:57:09 STEP 1295 H=1295:00 +CPU= 26.702
  12:57:58 STEP 1296 H=1296:00 +CPU= 48.604
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_53.zip
Uploading the intermediate file: upload_file_53.zip
  12:58:25 STEP 1297 H=1297:00 +CPU= 26.155
  12:58:51 STEP 1298 H=1298:00 +CPU= 26.578
  12:59:33 STEP 1299 H=1299:00 +CPU= 41.218
  13:00:00 STEP 1300 H=1300:00 +CPU= 26.641
  13:00:26 STEP 1301 H=1301:00 +CPU= 25.937
  13:01:08 STEP 1302 H=1302:00 +CPU= 41.838
  13:01:34 STEP 1303 H=1303:00 +CPU= 25.992
  13:02:01 STEP 1304 H=1304:00 +CPU= 26.763
  13:02:42 STEP 1305 H=1305:00 +CPU= 41.312
  13:03:09 STEP 1306 H=1306:00 +CPU= 26.596
  13:03:35 STEP 1307 H=1307:00 +CPU= 25.926
  13:04:21 STEP 1308 H=1308:00 +CPU= 45.390
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001308
  13:04:47 STEP 1309 H=1309:00 +CPU= 25.976
  13:05:14 STEP 1310 H=1310:00 +CPU= 26.658
  13:05:55 STEP 1311 H=1311:00 +CPU= 41.334
  13:06:22 STEP 1312 H=1312:00 +CPU= 26.598
  13:06:48 STEP 1313 H=1313:00 +CPU= 25.896
  13:07:30 STEP 1314 H=1314:00 +CPU= 41.801
  13:07:56 STEP 1315 H=1315:00 +CPU= 25.937
  13:08:23 STEP 1316 H=1316:00 +CPU= 26.547
  13:09:04 STEP 1317 H=1317:00 +CPU= 41.138
  13:09:31 STEP 1318 H=1318:00 +CPU= 26.403
  13:09:57 STEP 1319 H=1319:00 +CPU= 26.526
  13:10:46 STEP 1320 H=1320:00 +CPU= 48.509
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_54.zip
Uploading the intermediate file: upload_file_54.zip
  13:11:13 STEP 1321 H=1321:00 +CPU= 25.980
  13:11:39 STEP 1322 H=1322:00 +CPU= 26.423
  13:12:21 STEP 1323 H=1323:00 +CPU= 41.191
  13:12:47 STEP 1324 H=1324:00 +CPU= 26.543
  13:13:13 STEP 1325 H=1325:00 +CPU= 25.900
  13:13:56 STEP 1326 H=1326:00 +CPU= 42.061
  13:14:22 STEP 1327 H=1327:00 +CPU= 25.903
  13:14:48 STEP 1328 H=1328:00 +CPU= 26.526
  13:15:30 STEP 1329 H=1329:00 +CPU= 41.127
  13:15:56 STEP 1330 H=1330:00 +CPU= 26.541
  13:16:22 STEP 1331 H=1331:00 +CPU= 25.861
  13:17:08 STEP 1332 H=1332:00 +CPU= 45.232
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001332
  13:17:34 STEP 1333 H=1333:00 +CPU= 25.841
  13:18:00 STEP 1334 H=1334:00 +CPU= 26.496
  13:18:42 STEP 1335 H=1335:00 +CPU= 41.178
  13:19:08 STEP 1336 H=1336:00 +CPU= 26.542
  13:19:34 STEP 1337 H=1337:00 +CPU= 25.859
  13:20:17 STEP 1338 H=1338:00 +CPU= 41.884
  13:20:43 STEP 1339 H=1339:00 +CPU= 26.049
  13:21:09 STEP 1340 H=1340:00 +CPU= 26.604
  13:21:51 STEP 1341 H=1341:00 +CPU= 41.205
  13:22:18 STEP 1342 H=1342:00 +CPU= 26.695
  13:22:45 STEP 1343 H=1343:00 +CPU= 27.069
  13:23:34 STEP 1344 H=1344:00 +CPU= 48.735
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_55.zip
Uploading the intermediate file: upload_file_55.zip
  13:24:00 STEP 1345 H=1345:00 +CPU= 26.211
  13:24:27 STEP 1346 H=1346:00 +CPU= 26.658
  13:25:09 STEP 1347 H=1347:00 +CPU= 41.318
  13:25:36 STEP 1348 H=1348:00 +CPU= 26.604
  13:26:02 STEP 1349 H=1349:00 +CPU= 26.094
  13:26:45 STEP 1350 H=1350:00 +CPU= 43.014
  13:27:11 STEP 1351 H=1351:00 +CPU= 26.191
  13:27:38 STEP 1352 H=1352:00 +CPU= 26.723
  13:28:20 STEP 1353 H=1353:00 +CPU= 41.374
  13:28:47 STEP 1354 H=1354:00 +CPU= 26.746
  13:29:13 STEP 1355 H=1355:00 +CPU= 26.136
  13:29:59 STEP 1356 H=1356:00 +CPU= 45.549
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001356
  13:30:25 STEP 1357 H=1357:00 +CPU= 26.255
  13:30:52 STEP 1358 H=1358:00 +CPU= 26.706
  13:31:34 STEP 1359 H=1359:00 +CPU= 41.394
  13:32:01 STEP 1360 H=1360:00 +CPU= 26.714
  13:32:27 STEP 1361 H=1361:00 +CPU= 25.965
  13:33:09 STEP 1362 H=1362:00 +CPU= 42.212
  13:33:35 STEP 1363 H=1363:00 +CPU= 26.116
  13:34:02 STEP 1364 H=1364:00 +CPU= 26.709
  13:34:44 STEP 1365 H=1365:00 +CPU= 41.326
  13:35:10 STEP 1366 H=1366:00 +CPU= 26.644
  13:35:38 STEP 1367 H=1367:00 +CPU= 26.923
  13:36:27 STEP 1368 H=1368:00 +CPU= 48.931
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_56.zip
Uploading the intermediate file: upload_file_56.zip
Uploading trickle at timestep: 4924800
  13:36:53 STEP 1369 H=1369:00 +CPU= 26.242
  13:37:20 STEP 1370 H=1370:00 +CPU= 26.723
  13:38:02 STEP 1371 H=1371:00 +CPU= 41.435
  13:38:29 STEP 1372 H=1372:00 +CPU= 26.782
  13:38:55 STEP 1373 H=1373:00 +CPU= 26.087
  13:39:37 STEP 1374 H=1374:00 +CPU= 42.109
  13:40:03 STEP 1375 H=1375:00 +CPU= 26.147
  13:40:30 STEP 1376 H=1376:00 +CPU= 26.743
  13:41:12 STEP 1377 H=1377:00 +CPU= 41.349
  13:41:39 STEP 1378 H=1378:00 +CPU= 26.814
  13:42:05 STEP 1379 H=1379:00 +CPU= 26.128
  13:42:51 STEP 1380 H=1380:00 +CPU= 45.564
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001380
  13:43:17 STEP 1381 H=1381:00 +CPU= 26.062
  13:43:44 STEP 1382 H=1382:00 +CPU= 26.703
  13:44:25 STEP 1383 H=1383:00 +CPU= 41.252
  13:44:52 STEP 1384 H=1384:00 +CPU= 26.616
  13:45:18 STEP 1385 H=1385:00 +CPU= 26.129
  13:46:01 STEP 1386 H=1386:00 +CPU= 42.111
  13:46:27 STEP 1387 H=1387:00 +CPU= 25.865
  13:46:54 STEP 1388 H=1388:00 +CPU= 26.697
  13:47:35 STEP 1389 H=1389:00 +CPU= 41.383
  13:48:02 STEP 1390 H=1390:00 +CPU= 26.813
  13:48:29 STEP 1391 H=1391:00 +CPU= 26.908
  13:49:18 STEP 1392 H=1392:00 +CPU= 48.809
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_57.zip
Uploading the intermediate file: upload_file_57.zip
  13:49:45 STEP 1393 H=1393:00 +CPU= 26.296
  13:50:12 STEP 1394 H=1394:00 +CPU= 26.653
  13:50:54 STEP 1395 H=1395:00 +CPU= 41.377
  13:51:20 STEP 1396 H=1396:00 +CPU= 26.751
  13:51:47 STEP 1397 H=1397:00 +CPU= 26.071
  13:52:29 STEP 1398 H=1398:00 +CPU= 41.891
  13:52:55 STEP 1399 H=1399:00 +CPU= 26.043
  13:53:22 STEP 1400 H=1400:00 +CPU= 26.700
  13:54:03 STEP 1401 H=1401:00 +CPU= 41.309
  13:54:30 STEP 1402 H=1402:00 +CPU= 26.795
  13:54:57 STEP 1403 H=1403:00 +CPU= 26.266
  13:55:42 STEP 1404 H=1404:00 +CPU= 45.492
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001404
  13:56:08 STEP 1405 H=1405:00 +CPU= 25.973
  13:56:35 STEP 1406 H=1406:00 +CPU= 26.702
  13:57:17 STEP 1407 H=1407:00 +CPU= 41.342
  13:57:44 STEP 1408 H=1408:00 +CPU= 26.566
  13:58:10 STEP 1409 H=1409:00 +CPU= 25.964
  13:58:52 STEP 1410 H=1410:00 +CPU= 41.832
  13:59:18 STEP 1411 H=1411:00 +CPU= 25.994
  13:59:45 STEP 1412 H=1412:00 +CPU= 26.610
  14:00:26 STEP 1413 H=1413:00 +CPU= 41.164
  14:00:53 STEP 1414 H=1414:00 +CPU= 26.801
  14:01:20 STEP 1415 H=1415:00 +CPU= 26.842
  14:02:09 STEP 1416 H=1416:00 +CPU= 49.200
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_58.zip
Uploading the intermediate file: upload_file_58.zip
  14:02:36 STEP 1417 H=1417:00 +CPU= 26.270
  14:03:03 STEP 1418 H=1418:00 +CPU= 26.837
  14:03:45 STEP 1419 H=1419:00 +CPU= 41.508
  14:04:17 STEP 1420 H=1420:00 +CPU= 32.343
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.463] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2549MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2597 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        WVCOUPLE 
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         WAVEMDL 
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          WAMODEL 
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           >OMP-WAMODEL 2            (1431) 
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            IMPLSCH 
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             SINPUT 
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140444:1672405484:17352.464] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              SINPUT_JAN 
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140445:1672405485:17353.524] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:egon-VirtualBox:1:1:2854:2854] [20221230:140445:1672405485:17353.524] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC282A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001A81DC2  sinput_jan_               252  sinput_jan.F
oifs_43r3_model.e  0000000001A7B775  sinput_                    68  sinput.F90
oifs_43r3_model.e  0000000001A30FE9  implsch_                  254  implsch.F
oifs_43r3_model.e  000000000168CBD7  wamodel_                 1008  wamodel.F
oifs_43r3_model.e  0000000000872295  wavemdl_                  584  wavemdl.F
oifs_43r3_model.e  00000000007E0165  wvcouple_                 260  wvcouple.F90
oifs_43r3_model.e  000000000055E4AD  cnt4_                    1283  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0261
(argv4) batchid: 986
(argv5) wuid: 12202905
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/14
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/14/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/14/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7432a49056c0b71e52c7cf12fe796c6e to: /var/lib/boinc-client/slots/14/oifs_43r3_ps_0261_2017050100_123_986_12202905.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/14/oifs_43r3_ps_0261_2017050100_123_986_12202905.zip
ic_ancil_file: ic_ancil_12202905
ifsdata_file: ifsdata_12202905
climate_data_file: clim_data_12202905
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6368220425fcca61853a7790c6c3a415 to: /var/lib/boinc-client/slots/14/ic_ancil_12202905.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/14/ic_ancil_12202905.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c01089f827cfdafa6f2382b2d6e6535d to: /var/lib/boinc-client/slots/14/ifsdata/ifsdata_12202905.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/14/ifsdata/ifsdata_12202905.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/14/159l_2/clim_data_12202905.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/14/159l_2/clim_data_12202905.zip
Checking for progress XML file: /var/lib/boinc-client/slots/14/progress_file_12202905.xml
Opened progress file ok : /var/lib/boinc-client/slots/14/progress_file_12202905.xml
-- Model is restarting --
Adjusting last_iter, 1421, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/14/progress_file_12202905.xml
last_cpu_time: 46844
upload_file_number: 59
last_iter: 1415
last_upload: 5097600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828
The child process has been launched with process id: 3062
Executing the command: /var/lib/boinc-client/slots/14/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:3062:3062] [20230103:174639:1672764399:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:47:39 STEP 1416 H=1416:00 +CPU= 52.855
  17:48:07 STEP 1417 H=1417:00 +CPU= 27.589
  17:48:36 STEP 1418 H=1418:00 +CPU= 28.201
  17:49:20 STEP 1419 H=1419:00 +CPU= 44.030
  17:49:49 STEP 1420 H=1420:00 +CPU= 28.563
  17:50:17 STEP 1421 H=1421:00 +CPU= 27.877
  17:51:02 STEP 1422 H=1422:00 +CPU= 45.041
  17:51:30 STEP 1423 H=1423:00 +CPU= 27.729
  17:51:59 STEP 1424 H=1424:00 +CPU= 28.832
  17:52:44 STEP 1425 H=1425:00 +CPU= 44.487
  17:53:13 STEP 1426 H=1426:00 +CPU= 28.878
  17:53:41 STEP 1427 H=1427:00 +CPU= 28.027
  17:54:30 STEP 1428 H=1428:00 +CPU= 49.192
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001428
  17:54:59 STEP 1429 H=1429:00 +CPU= 28.154
  17:55:28 STEP 1430 H=1430:00 +CPU= 28.951
  17:56:13 STEP 1431 H=1431:00 +CPU= 44.585
  17:56:42 STEP 1432 H=1432:00 +CPU= 29.227
  17:57:12 STEP 1433 H=1433:00 +CPU= 29.580
  17:57:58 STEP 1434 H=1434:00 +CPU= 45.946
  17:58:27 STEP 1435 H=1435:00 +CPU= 28.409
  17:58:57 STEP 1436 H=1436:00 +CPU= 29.964
  17:59:44 STEP 1437 H=1437:00 +CPU= 46.498
  18:00:15 STEP 1438 H=1438:00 +CPU= 30.070
  18:00:45 STEP 1439 H=1439:00 +CPU= 30.363
  18:01:48 STEP 1440 H=1440:00 +CPU= 61.105
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0261_2017050100_123_986_12202905_0_r476889828_59.zip
Uploading the intermediate file: upload_file_59.zip
  18:02:24 STEP 1441 H=1441:00 +CPU= 35.933
  18:03:02 STEP 1442 H=1442:00 +CPU= 36.888
  18:03:57 STEP 1443 H=1443:00 +CPU= 54.773
  18:04:33 STEP 1444 H=1444:00 +CPU= 35.719
  18:05:10 STEP 1445 H=1445:00 +CPU= 36.392
  18:06:07 STEP 1446 H=1446:00 +CPU= 56.064
  18:06:42 STEP 1447 H=1447:00 +CPU= 34.896
  18:07:17 STEP 1448 H=1448:00 +CPU= 34.422
  18:08:13 STEP 1449 H=1449:00 +CPU= 55.799
  18:08:48 STEP 1450 H=1450:00 +CPU= 34.879
  18:09:25 STEP 1451 H=1451:00 +CPU= 35.891
  18:10:26 STEP 1452 H=1452:00 +CPU= 60.312
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAhq0f+001452
  18:11:00 STEP 1453 H=1453:00 +CPU= 34.256
  18:11:37 STEP 1454 H=1454:00 +CPU= 36.706
  18:12:33 STEP 1455 H=1455:00 +CPU= 54.969
  18:13:08 STEP 1456 H=1456:00 +CPU= 34.954
  18:13:43 STEP 1457 H=1457:00 +CPU= 34.119
  18:14:38 STEP 1458 H=1458:00 +CPU= 54.164
  18:15:12 STEP 1459 H=1459:00 +CPU= 33.385
  18:15:46 STEP 1460 H=1460:00 +CPU= 34.160
  18:16:39 STEP 1461 H=1461:00 +CPU= 52.348
  18:17:14 STEP 1462 H=1462:00 +CPU= 33.970
  18:17:48 STEP 1463 H=1463:00 +CPU= 34.536
ECCODES ERROR   :  IO ERROR: No space left on device (No space left on device)
 GRIB_WRITE_BYTES_INT        50001  FAILED         -11
 GRIB_API ERROR MSG: Input output problem
MPL_ABORT: CALLED FROM PROCESSOR      1 THRD     1
 MPL_ABORT:(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0261
(argv4) batchid: 986
(argv5) wuid: 12202905
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/14
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12202905
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/14/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/14/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7432a49056c0b71e52c7cf12fe796c6e to: /var/lib/boinc-client/slots/14/oifs_43r3_ps_0261_2017050100_123_986_12202905.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/14/oifs_43r3_ps_0261_2017050100_123_986_12202905.zip
ic_ancil_file: ic_ancil_12202905
ifsdata_file: ifsdata_12202905
climate_data_file: clim_data_12202905
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6368220425fcca61853a7790c6c3a415 to: /var/lib/boinc-client/slots/14/ic_ancil_12202905.zip
..Copying the IC ancils to the working directory failed

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:07:05 1537334 22289328 oifs_43r3_ps_0261_2017050100_123_986_12202905_0 4,924,800 45,166 0.0092
09 May 2023 16:02:50 1537334 22289328 oifs_43r3_ps_0261_2017050100_123_986_12202905_0 4,060,800 37,519 0.0092
09 May 2023 16:01:44 1537334 22289328 oifs_43r3_ps_0261_2017050100_123_986_12202905_0 2,592,000 23,414 0.0090
09 May 2023 16:01:44 1537334 22289328 oifs_43r3_ps_0261_2017050100_123_986_12202905_0 2,592,000 23,414 0.0090
09 May 2023 16:01:37 1537334 22289328 oifs_43r3_ps_0261_2017050100_123_986_12202905_0 1,728,000 15,435 0.0089
09 May 2023 16:01:29 1537334 22289328 oifs_43r3_ps_0261_2017050100_123_986_12202905_0 864,000 7,775 0.0090


©2024 climateprediction.net