climateprediction.net home page
Task 22299161

Task 22299161

Name oifs_43r3_ps_0243_2003050100_123_972_12188887_1
Workunit 12188887
Created 17 Jan 2023, 21:31:57 UTC
Sent 19 Jan 2023, 19:44:31 UTC
Report deadline 20 Mar 2023, 19:44:31 UTC
Received 20 Jan 2023, 19:03:58 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1524158
Run time 11 hours 42 min 19 sec
CPU time 11 hours 39 min 12 sec
Validate state Invalid
Credit 7,257.60
Device peak FLOPS 4.65 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.05 MB
Peak swap size 4,974.13 MB
Peak disk usage 2,805.97 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
: /var/lib/boinc-client/slots/2/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000996
  06:49:12 STEP  997 H= 997:00 +CPU= 23.447
  06:49:36 STEP  998 H= 998:00 +CPU= 24.248
  06:50:13 STEP  999 H= 999:00 +CPU= 36.840
  06:50:37 STEP 1000 H=1000:00 +CPU= 24.107
  06:51:01 STEP 1001 H=1001:00 +CPU= 23.338
  06:51:38 STEP 1002 H=1002:00 +CPU= 37.583
  06:52:02 STEP 1003 H=1003:00 +CPU= 23.366
  06:52:26 STEP 1004 H=1004:00 +CPU= 24.159
  06:53:03 STEP 1005 H=1005:00 +CPU= 36.808
  06:53:27 STEP 1006 H=1006:00 +CPU= 24.058
  06:53:51 STEP 1007 H=1007:00 +CPU= 24.061
  06:54:34 STEP 1008 H=1008:00 +CPU= 42.810
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_41.zip
Uploading the intermediate file: upload_file_41.zip
  06:54:58 STEP 1009 H=1009:00 +CPU= 23.422
  06:55:22 STEP 1010 H=1010:00 +CPU= 24.367
  06:55:59 STEP 1011 H=1011:00 +CPU= 36.746
  06:56:23 STEP 1012 H=1012:00 +CPU= 24.126
  06:56:46 STEP 1013 H=1013:00 +CPU= 23.411
  06:57:24 STEP 1014 H=1014:00 +CPU= 37.606
  06:57:48 STEP 1015 H=1015:00 +CPU= 23.444
  06:58:12 STEP 1016 H=1016:00 +CPU= 24.296
  06:58:49 STEP 1017 H=1017:00 +CPU= 36.905
  06:59:13 STEP 1018 H=1018:00 +CPU= 24.264
  06:59:37 STEP 1019 H=1019:00 +CPU= 23.537
  07:00:18 STEP 1020 H=1020:00 +CPU= 40.675
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001020
  07:00:41 STEP 1021 H=1021:00 +CPU= 23.460
  07:01:05 STEP 1022 H=1022:00 +CPU= 24.193
  07:01:42 STEP 1023 H=1023:00 +CPU= 36.837
  07:02:06 STEP 1024 H=1024:00 +CPU= 24.057
  07:02:30 STEP 1025 H=1025:00 +CPU= 23.343
  07:03:08 STEP 1026 H=1026:00 +CPU= 37.582
  07:03:31 STEP 1027 H=1027:00 +CPU= 23.360
  07:03:55 STEP 1028 H=1028:00 +CPU= 24.130
  07:04:32 STEP 1029 H=1029:00 +CPU= 36.848
  07:04:56 STEP 1030 H=1030:00 +CPU= 24.079
  07:05:20 STEP 1031 H=1031:00 +CPU= 24.209
  07:06:03 STEP 1032 H=1032:00 +CPU= 42.860
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_42.zip
Uploading the intermediate file: upload_file_42.zip
  07:06:27 STEP 1033 H=1033:00 +CPU= 23.407
  07:06:51 STEP 1034 H=1034:00 +CPU= 24.343
  07:07:28 STEP 1035 H=1035:00 +CPU= 36.758
  07:07:52 STEP 1036 H=1036:00 +CPU= 24.075
  07:08:16 STEP 1037 H=1037:00 +CPU= 23.377
  07:08:53 STEP 1038 H=1038:00 +CPU= 37.544
  07:09:17 STEP 1039 H=1039:00 +CPU= 23.406
  07:09:41 STEP 1040 H=1040:00 +CPU= 24.237
  07:10:18 STEP 1041 H=1041:00 +CPU= 36.878
  07:10:42 STEP 1042 H=1042:00 +CPU= 24.177
  07:11:06 STEP 1043 H=1043:00 +CPU= 23.451
  07:11:47 STEP 1044 H=1044:00 +CPU= 40.666
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001044
  07:12:10 STEP 1045 H=1045:00 +CPU= 23.494
  07:12:34 STEP 1046 H=1046:00 +CPU= 24.201
  07:13:11 STEP 1047 H=1047:00 +CPU= 36.905
  07:13:36 STEP 1048 H=1048:00 +CPU= 24.129
  07:13:59 STEP 1049 H=1049:00 +CPU= 23.423
  07:14:37 STEP 1050 H=1050:00 +CPU= 37.710
  07:15:00 STEP 1051 H=1051:00 +CPU= 23.430
  07:15:25 STEP 1052 H=1052:00 +CPU= 24.234
  07:16:02 STEP 1053 H=1053:00 +CPU= 36.889
  07:16:26 STEP 1054 H=1054:00 +CPU= 24.092
  07:16:50 STEP 1055 H=1055:00 +CPU= 24.221
  07:17:33 STEP 1056 H=1056:00 +CPU= 42.845
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_43.zip
Uploading the intermediate file: upload_file_43.zip
  07:17:57 STEP 1057 H=1057:00 +CPU= 23.433
  07:18:21 STEP 1058 H=1058:00 +CPU= 24.267
  07:18:58 STEP 1059 H=1059:00 +CPU= 36.795
  07:19:22 STEP 1060 H=1060:00 +CPU= 24.065
  07:19:45 STEP 1061 H=1061:00 +CPU= 23.328
  07:20:23 STEP 1062 H=1062:00 +CPU= 37.598
  07:20:46 STEP 1063 H=1063:00 +CPU= 23.401
  07:21:11 STEP 1064 H=1064:00 +CPU= 24.117
  07:21:47 STEP 1065 H=1065:00 +CPU= 36.784
  07:22:12 STEP 1066 H=1066:00 +CPU= 24.115
  07:22:35 STEP 1067 H=1067:00 +CPU= 23.366
  07:23:16 STEP 1068 H=1068:00 +CPU= 40.796
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001068
  07:23:40 STEP 1069 H=1069:00 +CPU= 23.594
  07:24:04 STEP 1070 H=1070:00 +CPU= 24.373
  07:24:41 STEP 1071 H=1071:00 +CPU= 36.977
  07:25:05 STEP 1072 H=1072:00 +CPU= 24.214
  07:25:29 STEP 1073 H=1073:00 +CPU= 23.473
  07:26:07 STEP 1074 H=1074:00 +CPU= 37.726
  07:26:30 STEP 1075 H=1075:00 +CPU= 23.555
  07:26:55 STEP 1076 H=1076:00 +CPU= 24.333
  07:27:32 STEP 1077 H=1077:00 +CPU= 36.971
  07:27:56 STEP 1078 H=1078:00 +CPU= 24.251
  07:28:21 STEP 1079 H=1079:00 +CPU= 24.369
  07:29:04 STEP 1080 H=1080:00 +CPU= 42.888
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_44.zip
Uploading the intermediate file: upload_file_44.zip
  07:29:27 STEP 1081 H=1081:00 +CPU= 23.417
  07:29:51 STEP 1082 H=1082:00 +CPU= 24.180
  07:30:28 STEP 1083 H=1083:00 +CPU= 36.827
  07:30:52 STEP 1084 H=1084:00 +CPU= 24.049
  07:31:16 STEP 1085 H=1085:00 +CPU= 23.315
  07:31:53 STEP 1086 H=1086:00 +CPU= 37.610
  07:32:17 STEP 1087 H=1087:00 +CPU= 23.414
  07:32:41 STEP 1088 H=1088:00 +CPU= 24.065
  07:33:18 STEP 1089 H=1089:00 +CPU= 36.810
  07:33:42 STEP 1090 H=1090:00 +CPU= 24.110
  07:34:05 STEP 1091 H=1091:00 +CPU= 23.263
  07:34:46 STEP 1092 H=1092:00 +CPU= 40.825
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001092
  07:35:10 STEP 1093 H=1093:00 +CPU= 23.682
  07:35:35 STEP 1094 H=1094:00 +CPU= 24.378
  07:36:12 STEP 1095 H=1095:00 +CPU= 37.020
  07:36:36 STEP 1096 H=1096:00 +CPU= 24.244
  07:37:00 STEP 1097 H=1097:00 +CPU= 23.522
  07:37:37 STEP 1098 H=1098:00 +CPU= 37.785
  07:38:01 STEP 1099 H=1099:00 +CPU= 23.575
  07:38:25 STEP 1100 H=1100:00 +CPU= 24.246
  07:39:02 STEP 1101 H=1101:00 +CPU= 36.974
  07:39:27 STEP 1102 H=1102:00 +CPU= 24.156
  07:39:51 STEP 1103 H=1103:00 +CPU= 24.265
  07:40:34 STEP 1104 H=1104:00 +CPU= 43.063
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_45.zip
Uploading the intermediate file: upload_file_45.zip
  07:40:58 STEP 1105 H=1105:00 +CPU= 23.563
  07:41:22 STEP 1106 H=1106:00 +CPU= 24.272
  07:41:59 STEP 1107 H=1107:00 +CPU= 36.926
  07:42:23 STEP 1108 H=1108:00 +CPU= 24.168
  07:42:47 STEP 1109 H=1109:00 +CPU= 23.419
  07:43:25 STEP 1110 H=1110:00 +CPU= 37.699
  07:43:48 STEP 1111 H=1111:00 +CPU= 23.518
  07:44:12 STEP 1112 H=1112:00 +CPU= 24.158
  07:44:49 STEP 1113 H=1113:00 +CPU= 36.909
  07:45:14 STEP 1114 H=1114:00 +CPU= 24.169
  07:45:37 STEP 1115 H=1115:00 +CPU= 23.366
  07:46:18 STEP 1116 H=1116:00 +CPU= 40.778
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001116
  07:46:42 STEP 1117 H=1117:00 +CPU= 23.634
  07:47:06 STEP 1118 H=1118:00 +CPU= 24.223
  07:47:43 STEP 1119 H=1119:00 +CPU= 37.118
  07:48:07 STEP 1120 H=1120:00 +CPU= 24.192
  07:48:31 STEP 1121 H=1121:00 +CPU= 23.387
  07:49:09 STEP 1122 H=1122:00 +CPU= 37.823
  07:49:32 STEP 1123 H=1123:00 +CPU= 23.566
  07:49:57 STEP 1124 H=1124:00 +CPU= 24.182
  07:50:34 STEP 1125 H=1125:00 +CPU= 36.965
  07:50:58 STEP 1126 H=1126:00 +CPU= 24.069
  07:51:22 STEP 1127 H=1127:00 +CPU= 24.050
  07:52:05 STEP 1128 H=1128:00 +CPU= 43.137
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_46.zip
Uploading the intermediate file: upload_file_46.zip
  07:52:29 STEP 1129 H=1129:00 +CPU= 23.662
  07:52:53 STEP 1130 H=1130:00 +CPU= 24.326
  07:53:30 STEP 1131 H=1131:00 +CPU= 37.122
  07:53:55 STEP 1132 H=1132:00 +CPU= 24.247
  07:54:18 STEP 1133 H=1133:00 +CPU= 23.415
  07:54:56 STEP 1134 H=1134:00 +CPU= 37.899
  07:55:20 STEP 1135 H=1135:00 +CPU= 23.555
  07:55:44 STEP 1136 H=1136:00 +CPU= 24.011
  07:56:21 STEP 1137 H=1137:00 +CPU= 37.035
  07:56:45 STEP 1138 H=1138:00 +CPU= 24.296
  07:57:09 STEP 1139 H=1139:00 +CPU= 23.464
  07:57:50 STEP 1140 H=1140:00 +CPU= 41.053
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001140
  07:58:14 STEP 1141 H=1141:00 +CPU= 23.489
  07:58:38 STEP 1142 H=1142:00 +CPU= 23.989
  07:59:15 STEP 1143 H=1143:00 +CPU= 37.096
  07:59:39 STEP 1144 H=1144:00 +CPU= 24.157
  08:00:02 STEP 1145 H=1145:00 +CPU= 23.199
  08:00:40 STEP 1146 H=1146:00 +CPU= 37.843
  08:01:04 STEP 1147 H=1147:00 +CPU= 23.530
  08:01:28 STEP 1148 H=1148:00 +CPU= 23.905
  08:02:05 STEP 1149 H=1149:00 +CPU= 36.992
  08:02:29 STEP 1150 H=1150:00 +CPU= 24.097
  08:02:53 STEP 1151 H=1151:00 +CPU= 23.935
  08:03:36 STEP 1152 H=1152:00 +CPU= 43.175
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_47.zip
Uploading the intermediate file: upload_file_47.zip
  08:04:00 STEP 1153 H=1153:00 +CPU= 23.761
  08:04:25 STEP 1154 H=1154:00 +CPU= 24.296
  08:05:02 STEP 1155 H=1155:00 +CPU= 37.176
  08:05:26 STEP 1156 H=1156:00 +CPU= 24.387
  08:05:50 STEP 1157 H=1157:00 +CPU= 23.414
  08:06:28 STEP 1158 H=1158:00 +CPU= 37.943
  08:06:52 STEP 1159 H=1159:00 +CPU= 23.766
  08:07:16 STEP 1160 H=1160:00 +CPU= 24.045
  08:07:53 STEP 1161 H=1161:00 +CPU= 37.136
  08:08:17 STEP 1162 H=1162:00 +CPU= 24.159
  08:08:41 STEP 1163 H=1163:00 +CPU= 23.239
  08:09:22 STEP 1164 H=1164:00 +CPU= 41.036
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001164
  08:09:45 STEP 1165 H=1165:00 +CPU= 23.507
  08:10:09 STEP 1166 H=1166:00 +CPU= 23.998
  08:10:46 STEP 1167 H=1167:00 +CPU= 37.052
  08:11:11 STEP 1168 H=1168:00 +CPU= 24.119
  08:11:34 STEP 1169 H=1169:00 +CPU= 23.183
  08:12:12 STEP 1170 H=1170:00 +CPU= 37.812
  08:12:35 STEP 1171 H=1171:00 +CPU= 23.347
  08:12:59 STEP 1172 H=1172:00 +CPU= 24.034
  08:13:36 STEP 1173 H=1173:00 +CPU= 37.081
  08:14:01 STEP 1174 H=1174:00 +CPU= 24.180
  08:14:25 STEP 1175 H=1175:00 +CPU= 24.009
  08:15:08 STEP 1176 H=1176:00 +CPU= 43.061
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_48.zip
Uploading the intermediate file: upload_file_48.zip
  08:15:31 STEP 1177 H=1177:00 +CPU= 23.388
  08:15:55 STEP 1178 H=1178:00 +CPU= 23.945
  08:16:33 STEP 1179 H=1179:00 +CPU= 37.072
  08:16:57 STEP 1180 H=1180:00 +CPU= 24.026
  08:17:20 STEP 1181 H=1181:00 +CPU= 23.122
  08:17:58 STEP 1182 H=1182:00 +CPU= 37.886
  08:18:21 STEP 1183 H=1183:00 +CPU= 23.359
  08:18:45 STEP 1184 H=1184:00 +CPU= 24.045
  08:19:22 STEP 1185 H=1185:00 +CPU= 37.095
  08:19:47 STEP 1186 H=1186:00 +CPU= 24.373
  08:20:10 STEP 1187 H=1187:00 +CPU= 23.396
  08:20:52 STEP 1188 H=1188:00 +CPU= 41.159
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001188
  08:21:15 STEP 1189 H=1189:00 +CPU= 23.546
  08:21:39 STEP 1190 H=1190:00 +CPU= 23.945
  08:22:17 STEP 1191 H=1191:00 +CPU= 37.232
  08:22:41 STEP 1192 H=1192:00 +CPU= 24.196
  08:23:04 STEP 1193 H=1193:00 +CPU= 23.103
  08:23:42 STEP 1194 H=1194:00 +CPU= 37.965
  08:24:06 STEP 1195 H=1195:00 +CPU= 23.418
  08:24:29 STEP 1196 H=1196:00 +CPU= 23.924
  08:25:07 STEP 1197 H=1197:00 +CPU= 37.060
  08:25:31 STEP 1198 H=1198:00 +CPU= 24.102
  08:25:55 STEP 1199 H=1199:00 +CPU= 23.801
  08:26:38 STEP 1200 H=1200:00 +CPU= 43.146
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_49.zip
Uploading the intermediate file: upload_file_49.zip
  08:27:01 STEP 1201 H=1201:00 +CPU= 23.528
Uploading trickle at timestep: 4320000
  08:27:26 STEP 1202 H=1202:00 +CPU= 23.972
  08:28:03 STEP 1203 H=1203:00 +CPU= 37.139
  08:28:27 STEP 1204 H=1204:00 +CPU= 24.184
  08:28:50 STEP 1205 H=1205:00 +CPU= 23.092
  08:29:28 STEP 1206 H=1206:00 +CPU= 37.926
  08:29:52 STEP 1207 H=1207:00 +CPU= 23.556
  08:30:16 STEP 1208 H=1208:00 +CPU= 23.901
  08:30:53 STEP 1209 H=1209:00 +CPU= 37.086
  08:31:17 STEP 1210 H=1210:00 +CPU= 24.123
  08:31:40 STEP 1211 H=1211:00 +CPU= 23.183
  08:32:21 STEP 1212 H=1212:00 +CPU= 41.039
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001212
  08:32:45 STEP 1213 H=1213:00 +CPU= 23.560
  08:33:09 STEP 1214 H=1214:00 +CPU= 23.834
  08:33:46 STEP 1215 H=1215:00 +CPU= 37.143
  08:34:10 STEP 1216 H=1216:00 +CPU= 24.131
  08:34:33 STEP 1217 H=1217:00 +CPU= 22.990
  08:35:11 STEP 1218 H=1218:00 +CPU= 37.857
  08:35:35 STEP 1219 H=1219:00 +CPU= 23.405
  08:35:59 STEP 1220 H=1220:00 +CPU= 23.861
  08:36:36 STEP 1221 H=1221:00 +CPU= 37.036
  08:37:00 STEP 1222 H=1222:00 +CPU= 24.123
  08:37:24 STEP 1223 H=1223:00 +CPU= 23.804
  08:38:07 STEP 1224 H=1224:00 +CPU= 43.246
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_50.zip
Uploading the intermediate file: upload_file_50.zip
  08:38:31 STEP 1225 H=1225:00 +CPU= 23.595
  08:38:55 STEP 1226 H=1226:00 +CPU= 23.998
  08:39:32 STEP 1227 H=1227:00 +CPU= 37.159
  08:39:57 STEP 1228 H=1228:00 +CPU= 24.230
  08:40:20 STEP 1229 H=1229:00 +CPU= 23.122
  08:40:58 STEP 1230 H=1230:00 +CPU= 37.916
  08:41:21 STEP 1231 H=1231:00 +CPU= 23.562
  08:41:45 STEP 1232 H=1232:00 +CPU= 23.848
  08:42:22 STEP 1233 H=1233:00 +CPU= 37.038
  08:42:47 STEP 1234 H=1234:00 +CPU= 24.077
  08:43:10 STEP 1235 H=1235:00 +CPU= 23.069
  08:43:51 STEP 1236 H=1236:00 +CPU= 41.038
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001236
  08:44:14 STEP 1237 H=1237:00 +CPU= 23.529
  08:44:38 STEP 1238 H=1238:00 +CPU= 23.884
  08:45:16 STEP 1239 H=1239:00 +CPU= 37.229
  08:45:40 STEP 1240 H=1240:00 +CPU= 24.114
  08:46:03 STEP 1241 H=1241:00 +CPU= 23.035
  08:46:41 STEP 1242 H=1242:00 +CPU= 37.924
  08:47:04 STEP 1243 H=1243:00 +CPU= 23.380
  08:47:28 STEP 1244 H=1244:00 +CPU= 23.923
  08:48:06 STEP 1245 H=1245:00 +CPU= 37.104
  08:48:30 STEP 1246 H=1246:00 +CPU= 24.151
  08:48:54 STEP 1247 H=1247:00 +CPU= 23.898
  08:49:37 STEP 1248 H=1248:00 +CPU= 43.265
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_51.zip
Uploading the intermediate file: upload_file_51.zip
  08:50:01 STEP 1249 H=1249:00 +CPU= 23.460
  08:50:25 STEP 1250 H=1250:00 +CPU= 24.059
  08:51:02 STEP 1251 H=1251:00 +CPU= 37.181
  08:51:26 STEP 1252 H=1252:00 +CPU= 24.066
  08:51:49 STEP 1253 H=1253:00 +CPU= 23.146
  08:52:28 STEP 1254 H=1254:00 +CPU= 37.971
  08:52:51 STEP 1255 H=1255:00 +CPU= 23.625
  08:53:15 STEP 1256 H=1256:00 +CPU= 23.820
  08:53:52 STEP 1257 H=1257:00 +CPU= 37.188
  08:54:17 STEP 1258 H=1258:00 +CPU= 24.288
  08:54:40 STEP 1259 H=1259:00 +CPU= 23.042
  08:55:21 STEP 1260 H=1260:00 +CPU= 41.075
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001260
  08:55:44 STEP 1261 H=1261:00 +CPU= 23.439
  08:56:08 STEP 1262 H=1262:00 +CPU= 23.873
  08:56:46 STEP 1263 H=1263:00 +CPU= 37.293
  08:57:10 STEP 1264 H=1264:00 +CPU= 24.047
  08:57:33 STEP 1265 H=1265:00 +CPU= 23.085
  08:58:11 STEP 1266 H=1266:00 +CPU= 37.994
  08:58:35 STEP 1267 H=1267:00 +CPU= 23.516
  08:58:58 STEP 1268 H=1268:00 +CPU= 23.658
  08:59:36 STEP 1269 H=1269:00 +CPU= 37.211
  09:00:00 STEP 1270 H=1270:00 +CPU= 24.014
  09:00:24 STEP 1271 H=1271:00 +CPU= 23.733
  09:01:07 STEP 1272 H=1272:00 +CPU= 43.369
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_52.zip
Uploading the intermediate file: upload_file_52.zip
  09:01:31 STEP 1273 H=1273:00 +CPU= 23.446
  09:01:55 STEP 1274 H=1274:00 +CPU= 24.025
  09:02:32 STEP 1275 H=1275:00 +CPU= 37.251
  09:02:56 STEP 1276 H=1276:00 +CPU= 24.101
  09:03:19 STEP 1277 H=1277:00 +CPU= 23.173
  09:03:57 STEP 1278 H=1278:00 +CPU= 37.990
  09:04:21 STEP 1279 H=1279:00 +CPU= 23.562
  09:04:45 STEP 1280 H=1280:00 +CPU= 23.809
  09:05:22 STEP 1281 H=1281:00 +CPU= 37.166
  09:05:46 STEP 1282 H=1282:00 +CPU= 24.148
  09:06:09 STEP 1283 H=1283:00 +CPU= 23.054
  09:06:51 STEP 1284 H=1284:00 +CPU= 41.073
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001284
  09:07:14 STEP 1285 H=1285:00 +CPU= 23.453
  09:07:38 STEP 1286 H=1286:00 +CPU= 24.010
  09:08:16 STEP 1287 H=1287:00 +CPU= 37.205
  09:08:40 STEP 1288 H=1288:00 +CPU= 24.057
  09:09:03 STEP 1289 H=1289:00 +CPU= 23.171
  09:09:41 STEP 1290 H=1290:00 +CPU= 37.845
  09:10:04 STEP 1291 H=1291:00 +CPU= 23.524
  09:10:28 STEP 1292 H=1292:00 +CPU= 23.675
  09:11:05 STEP 1293 H=1293:00 +CPU= 37.171
  09:11:30 STEP 1294 H=1294:00 +CPU= 24.112
  09:11:53 STEP 1295 H=1295:00 +CPU= 23.678
  09:12:37 STEP 1296 H=1296:00 +CPU= 43.326
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_53.zip
Uploading the intermediate file: upload_file_53.zip
  09:13:00 STEP 1297 H=1297:00 +CPU= 23.443
  09:13:25 STEP 1298 H=1298:00 +CPU= 24.127
  09:14:02 STEP 1299 H=1299:00 +CPU= 37.034
  09:14:26 STEP 1300 H=1300:00 +CPU= 24.274
  09:14:49 STEP 1301 H=1301:00 +CPU= 23.322
  09:15:27 STEP 1302 H=1302:00 +CPU= 37.822
  09:15:51 STEP 1303 H=1303:00 +CPU= 23.479
  09:16:15 STEP 1304 H=1304:00 +CPU= 23.862
  09:16:52 STEP 1305 H=1305:00 +CPU= 37.289
  09:17:16 STEP 1306 H=1306:00 +CPU= 24.110
  09:17:40 STEP 1307 H=1307:00 +CPU= 23.115
  09:18:21 STEP 1308 H=1308:00 +CPU= 41.130
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001308
  09:18:44 STEP 1309 H=1309:00 +CPU= 23.410
  09:19:08 STEP 1310 H=1310:00 +CPU= 24.079
  09:19:46 STEP 1311 H=1311:00 +CPU= 37.126
  09:20:10 STEP 1312 H=1312:00 +CPU= 24.019
  09:20:33 STEP 1313 H=1313:00 +CPU= 23.279
  09:21:11 STEP 1314 H=1314:00 +CPU= 37.856
  09:21:35 STEP 1315 H=1315:00 +CPU= 23.457
  09:21:58 STEP 1316 H=1316:00 +CPU= 23.700
  09:22:36 STEP 1317 H=1317:00 +CPU= 37.226
  09:23:00 STEP 1318 H=1318:00 +CPU= 23.957
  09:23:23 STEP 1319 H=1319:00 +CPU= 23.639
  09:24:07 STEP 1320 H=1320:00 +CPU= 43.391
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_54.zip
Uploading the intermediate file: upload_file_54.zip
  09:24:30 STEP 1321 H=1321:00 +CPU= 23.476
  09:24:55 STEP 1322 H=1322:00 +CPU= 24.309
  09:25:32 STEP 1323 H=1323:00 +CPU= 37.210
  09:25:56 STEP 1324 H=1324:00 +CPU= 24.089
  09:26:20 STEP 1325 H=1325:00 +CPU= 23.511
  09:26:58 STEP 1326 H=1326:00 +CPU= 37.937
  09:27:21 STEP 1327 H=1327:00 +CPU= 23.512
  09:27:45 STEP 1328 H=1328:00 +CPU= 24.005
  09:28:23 STEP 1329 H=1329:00 +CPU= 37.058
  09:28:47 STEP 1330 H=1330:00 +CPU= 24.107
  09:29:10 STEP 1331 H=1331:00 +CPU= 23.285
  09:29:51 STEP 1332 H=1332:00 +CPU= 41.073
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001332
  09:30:15 STEP 1333 H=1333:00 +CPU= 23.432
  09:30:39 STEP 1334 H=1334:00 +CPU= 24.155
  09:31:16 STEP 1335 H=1335:00 +CPU= 36.976
  09:31:40 STEP 1336 H=1336:00 +CPU= 24.050
  09:32:04 STEP 1337 H=1337:00 +CPU= 23.314
  09:32:41 STEP 1338 H=1338:00 +CPU= 37.681
  09:33:05 STEP 1339 H=1339:00 +CPU= 23.458
  09:33:29 STEP 1340 H=1340:00 +CPU= 23.791
  09:34:06 STEP 1341 H=1341:00 +CPU= 37.148
  09:34:30 STEP 1342 H=1342:00 +CPU= 24.061
  09:34:54 STEP 1343 H=1343:00 +CPU= 23.803
  09:35:37 STEP 1344 H=1344:00 +CPU= 43.020
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_55.zip
Uploading the intermediate file: upload_file_55.zip
  09:36:01 STEP 1345 H=1345:00 +CPU= 23.421
  09:36:25 STEP 1346 H=1346:00 +CPU= 24.211
  09:37:02 STEP 1347 H=1347:00 +CPU= 36.874
  09:37:26 STEP 1348 H=1348:00 +CPU= 24.033
  09:37:49 STEP 1349 H=1349:00 +CPU= 23.365
  09:38:27 STEP 1350 H=1350:00 +CPU= 37.624
  09:38:51 STEP 1351 H=1351:00 +CPU= 23.488
  09:39:15 STEP 1352 H=1352:00 +CPU= 24.102
  09:39:52 STEP 1353 H=1353:00 +CPU= 37.190
  09:40:16 STEP 1354 H=1354:00 +CPU= 24.103
  09:40:40 STEP 1355 H=1355:00 +CPU= 23.431
  09:41:21 STEP 1356 H=1356:00 +CPU= 41.068
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001356
  09:41:44 STEP 1357 H=1357:00 +CPU= 23.360
  09:42:09 STEP 1358 H=1358:00 +CPU= 24.228
  09:42:46 STEP 1359 H=1359:00 +CPU= 36.899
  09:43:10 STEP 1360 H=1360:00 +CPU= 23.941
  09:43:33 STEP 1361 H=1361:00 +CPU= 23.414
  09:44:11 STEP 1362 H=1362:00 +CPU= 37.621
  09:44:34 STEP 1363 H=1363:00 +CPU= 23.454
  09:44:58 STEP 1364 H=1364:00 +CPU= 23.888
  09:45:35 STEP 1365 H=1365:00 +CPU= 36.985
  09:45:59 STEP 1366 H=1366:00 +CPU= 23.958
  09:46:23 STEP 1367 H=1367:00 +CPU= 23.844
  09:47:06 STEP 1368 H=1368:00 +CPU= 43.174
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_56.zip
Uploading the intermediate file: upload_file_56.zip
  09:47:30 STEP 1369 H=1369:00 +CPU= 23.470
  09:47:54 STEP 1370 H=1370:00 +CPU= 24.410
  09:48:32 STEP 1371 H=1371:00 +CPU= 36.958
  09:48:56 STEP 1372 H=1372:00 +CPU= 24.093
  09:49:19 STEP 1373 H=1373:00 +CPU= 23.599
  09:49:57 STEP 1374 H=1374:00 +CPU= 37.738
  09:50:21 STEP 1375 H=1375:00 +CPU= 23.532
  09:50:45 STEP 1376 H=1376:00 +CPU= 24.038
  09:51:22 STEP 1377 H=1377:00 +CPU= 36.937
  09:51:46 STEP 1378 H=1378:00 +CPU= 24.095
  09:52:09 STEP 1379 H=1379:00 +CPU= 23.355
  09:52:51 STEP 1380 H=1380:00 +CPU= 40.958
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001380
  09:53:14 STEP 1381 H=1381:00 +CPU= 23.367
  09:53:38 STEP 1382 H=1382:00 +CPU= 24.189
  09:54:15 STEP 1383 H=1383:00 +CPU= 37.065
  09:54:39 STEP 1384 H=1384:00 +CPU= 24.013
  09:55:03 STEP 1385 H=1385:00 +CPU= 23.396
  09:55:41 STEP 1386 H=1386:00 +CPU= 37.876
  09:56:04 STEP 1387 H=1387:00 +CPU= 23.311
  09:56:28 STEP 1388 H=1388:00 +CPU= 23.983
  09:57:05 STEP 1389 H=1389:00 +CPU= 37.160
  09:57:30 STEP 1390 H=1390:00 +CPU= 23.999
  09:57:54 STEP 1391 H=1391:00 +CPU= 23.928
  09:58:37 STEP 1392 H=1392:00 +CPU= 43.305
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_57.zip
Uploading the intermediate file: upload_file_57.zip
  09:59:00 STEP 1393 H=1393:00 +CPU= 23.209
  09:59:25 STEP 1394 H=1394:00 +CPU= 24.317
  10:00:02 STEP 1395 H=1395:00 +CPU= 36.965
  10:00:26 STEP 1396 H=1396:00 +CPU= 23.921
  10:00:49 STEP 1397 H=1397:00 +CPU= 23.466
  10:01:27 STEP 1398 H=1398:00 +CPU= 37.849
  10:01:51 STEP 1399 H=1399:00 +CPU= 23.367
  10:02:15 STEP 1400 H=1400:00 +CPU= 24.275
  10:02:52 STEP 1401 H=1401:00 +CPU= 37.005
  10:03:16 STEP 1402 H=1402:00 +CPU= 24.143
  10:03:40 STEP 1403 H=1403:00 +CPU= 23.478
  10:04:21 STEP 1404 H=1404:00 +CPU= 40.947
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001404
  10:04:44 STEP 1405 H=1405:00 +CPU= 23.172
  10:05:08 STEP 1406 H=1406:00 +CPU= 24.153
  10:05:45 STEP 1407 H=1407:00 +CPU= 37.027
  10:06:09 STEP 1408 H=1408:00 +CPU= 23.794
  10:06:33 STEP 1409 H=1409:00 +CPU= 23.356
  10:07:10 STEP 1410 H=1410:00 +CPU= 37.772
  10:07:34 STEP 1411 H=1411:00 +CPU= 23.272
  10:07:58 STEP 1412 H=1412:00 +CPU= 23.950
  10:08:35 STEP 1413 H=1413:00 +CPU= 36.938
  10:08:59 STEP 1414 H=1414:00 +CPU= 23.932
  10:09:23 STEP 1415 H=1415:00 +CPU= 24.101
  10:10:06 STEP 1416 H=1416:00 +CPU= 43.130
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_58.zip
Uploading the intermediate file: upload_file_58.zip
  10:10:30 STEP 1417 H=1417:00 +CPU= 23.337
  10:10:54 STEP 1418 H=1418:00 +CPU= 24.304
  10:11:31 STEP 1419 H=1419:00 +CPU= 37.034
  10:11:55 STEP 1420 H=1420:00 +CPU= 23.931
  10:12:19 STEP 1421 H=1421:00 +CPU= 23.454
  10:12:56 STEP 1422 H=1422:00 +CPU= 37.784
  10:13:20 STEP 1423 H=1423:00 +CPU= 23.318
  10:13:44 STEP 1424 H=1424:00 +CPU= 24.174
  10:14:21 STEP 1425 H=1425:00 +CPU= 37.038
  10:14:45 STEP 1426 H=1426:00 +CPU= 24.033
  10:15:09 STEP 1427 H=1427:00 +CPU= 23.426
  10:15:50 STEP 1428 H=1428:00 +CPU= 40.926
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001428
  10:16:13 STEP 1429 H=1429:00 +CPU= 23.381
  10:16:38 STEP 1430 H=1430:00 +CPU= 24.254
  10:17:15 STEP 1431 H=1431:00 +CPU= 37.197
  10:17:39 STEP 1432 H=1432:00 +CPU= 23.947
  10:18:02 STEP 1433 H=1433:00 +CPU= 23.420
  10:18:40 STEP 1434 H=1434:00 +CPU= 37.751
  10:19:04 STEP 1435 H=1435:00 +CPU= 23.302
  10:19:28 STEP 1436 H=1436:00 +CPU= 24.083
  10:20:05 STEP 1437 H=1437:00 +CPU= 36.982
  10:20:29 STEP 1438 H=1438:00 +CPU= 23.998
  10:20:53 STEP 1439 H=1439:00 +CPU= 24.029
  10:21:36 STEP 1440 H=1440:00 +CPU= 43.044
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463_59.zip
Uploading the intermediate file: upload_file_59.zip
  10:21:59 STEP 1441 H=1441:00 +CPU= 23.199
Uploading trickle at timestep: 5184000
  10:22:24 STEP 1442 H=1442:00 +CPU= 24.342
  10:23:01 STEP 1443 H=1443:00 +CPU= 36.945
  10:23:25 STEP 1444 H=1444:00 +CPU= 23.921
  10:23:48 STEP 1445 H=1445:00 +CPU= 23.495
  10:24:26 STEP 1446 H=1446:00 +CPU= 37.745
  10:24:50 STEP 1447 H=1447:00 +CPU= 23.303
  10:25:14 STEP 1448 H=1448:00 +CPU= 24.158
  10:25:51 STEP 1449 H=1449:00 +CPU= 37.031
  10:26:15 STEP 1450 H=1450:00 +CPU= 23.968
  10:26:38 STEP 1451 H=1451:00 +CPU= 23.443
  10:27:19 STEP 1452 H=1452:00 +CPU= 40.982
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001452
  10:27:43 STEP 1453 H=1453:00 +CPU= 23.203
  10:28:07 STEP 1454 H=1454:00 +CPU= 24.221
  10:28:44 STEP 1455 H=1455:00 +CPU= 37.101
  10:29:08 STEP 1456 H=1456:00 +CPU= 23.840
  10:29:31 STEP 1457 H=1457:00 +CPU= 23.393
  10:30:09 STEP 1458 H=1458:00 +CPU= 37.786
  10:30:33 STEP 1459 H=1459:00 +CPU= 23.296
  10:30:57 STEP 1460 H=1460:00 +CPU= 24.047
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0243
(argv4) batchid: 972
(argv5) wuid: 12188887
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188887
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d3db259fbe185387de03b185be086bcb to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0243_2003050100_123_972_12188887.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0243_2003050100_123_972_12188887.zip
ic_ancil_file: ic_ancil_12188887
ifsdata_file: ifsdata_12188887
climate_data_file: clim_data_12188887
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_dbaeeda73f894c583b4c23c20320611f to: /var/lib/boinc-client/slots/2/ic_ancil_12188887.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12188887.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e71a50f33b2080f00bba03d2351bb0a1 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12188887.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12188887.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12188887.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12188887.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12188887.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12188887.xml
-- Model is restarting --
Adjusting last_iter, 1461, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12188887.xml
last_cpu_time: 41915
upload_file_number: 60
last_iter: 1439
last_upload: 5184000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0243_2003050100_123_972_12188887_1_r1592974463
The child process has been launched with process id: 17272
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133659:1674239819:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
 ABORT!    1 !! *** WAVE MODEL HAS ABORTED 
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.111] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3520 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.111] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.111] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.111] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.111] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.111] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.111] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.111] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        WVCOUPLE 
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.111] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         WAVEMDL 
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.111] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          INITMDL 
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.111] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           USERIN 
 SDL_TRACEBACK: Calling INTEL_TRBK, THRD =            1
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/2/oifs_43r3_model.exe' : sigcontextptr=0x7ffc31966320
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with addr2line-cmd
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/2/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x81d89e 0x15f8aca 0x15d51c8 0x166b07f 0x15d7ff0 0x8711e0 0x7e0165 0x55e4ad 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x81d89e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x15f8aca] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x15d51c8] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x166b07f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x15d7ff0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x8711e0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x7e0165] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x55e4ad] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x5471c9] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x546412] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x545e28] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x40708d] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40220f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4021a2] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x1dca390] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40206e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 SDL_TRACEBACK: Done INTEL_TRBK, THRD =            1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.195] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3520MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.195] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.195] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.195] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x7f00004378], nsigs = 1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133736:1674239856:37.195] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133737:1674239857:38.197] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1
[EC_DRHOOK:digdug:1:1:17272:17272] [20230120:133737:1674239857:38.197] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC903B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008C2880  sdl_mod_mp_sdl_sr         100  sdl_mod.F90
oifs_43r3_model.e  000000000081D8AF  abor1_                     41  abor1.F90
oifs_43r3_model.e  00000000015F8ACA  mpabort_                   50  mpabort.F
oifs_43r3_model.e  00000000015D51C8  abort1_                    44  abort1.F
oifs_43r3_model.e  000000000166B07F  userin_                   553  userin.F
oifs_43r3_model.e  00000000015D7FF0  initmdl_                  452  initmdl.F
oifs_43r3_model.e  00000000008711E0  wavemdl_                  393  wavemdl.F
oifs_43r3_model.e  00000000007E0165  wvcouple_                 260  wvcouple.F90
oifs_43r3_model.e  000000000055E4AD  cnt4_                    1283  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:31:10 1524158 22299161 oifs_43r3_ps_0243_2003050100_123_972_12188887_1 5,184,000 41,344 0.0080
09 May 2023 16:30:59 1524158 22299161 oifs_43r3_ps_0243_2003050100_123_972_12188887_1 4,320,000 34,470 0.0080
09 May 2023 16:30:47 1524158 22299161 oifs_43r3_ps_0243_2003050100_123_972_12188887_1 3,456,000 27,576 0.0080
09 May 2023 16:30:35 1524158 22299161 oifs_43r3_ps_0243_2003050100_123_972_12188887_1 2,592,000 20,696 0.0080
09 May 2023 16:30:23 1524158 22299161 oifs_43r3_ps_0243_2003050100_123_972_12188887_1 1,728,000 13,810 0.0080
09 May 2023 16:30:11 1524158 22299161 oifs_43r3_ps_0243_2003050100_123_972_12188887_1 864,000 6,924 0.0080


©2024 climateprediction.net