climateprediction.net home page
Task 22298427

Task 22298427

Name oifs_43r3_ps_0377_2013050100_123_982_12199021_2
Workunit 12199021
Created 16 Jan 2023, 5:42:26 UTC
Sent 16 Jan 2023, 6:17:59 UTC
Report deadline 15 Feb 2023, 6:17:59 UTC
Received 17 Jan 2023, 1:33:38 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1538037
Run time 11 hours 53 min 56 sec
CPU time 11 hours 53 min 56 sec
Validate state Invalid
Credit 6,531.84
Device peak FLOPS 4.76 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,764.42 MB
Peak swap size 4,978.07 MB
Peak disk usage 2,266.93 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
EP  998 H= 998:00 +CPU= 23.104
  05:58:26 STEP  999 H= 999:00 +CPU= 35.272
  05:58:49 STEP 1000 H=1000:00 +CPU= 23.052
  05:59:11 STEP 1001 H=1001:00 +CPU= 22.451
  05:59:47 STEP 1002 H=1002:00 +CPU= 36.191
  06:00:10 STEP 1003 H=1003:00 +CPU= 22.317
  06:00:33 STEP 1004 H=1004:00 +CPU= 23.234
  06:01:08 STEP 1005 H=1005:00 +CPU= 35.337
  06:01:31 STEP 1006 H=1006:00 +CPU= 22.971
  06:01:54 STEP 1007 H=1007:00 +CPU= 23.081
  06:02:38 STEP 1008 H=1008:00 +CPU= 41.502
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_41.zip
  06:03:01 STEP 1009 H=1009:00 +CPU= 22.436
Uploading the intermediate file: upload_file_41.zip
  06:03:24 STEP 1010 H=1010:00 +CPU= 23.245
  06:03:59 STEP 1011 H=1011:00 +CPU= 35.535
  06:04:23 STEP 1012 H=1012:00 +CPU= 23.999
  06:04:47 STEP 1013 H=1013:00 +CPU= 23.263
  06:05:24 STEP 1014 H=1014:00 +CPU= 36.903
  06:05:46 STEP 1015 H=1015:00 +CPU= 22.374
  06:06:09 STEP 1016 H=1016:00 +CPU= 23.196
  06:06:45 STEP 1017 H=1017:00 +CPU= 35.471
  06:07:08 STEP 1018 H=1018:00 +CPU= 23.203
  06:07:30 STEP 1019 H=1019:00 +CPU= 22.595
  06:08:10 STEP 1020 H=1020:00 +CPU= 39.279
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001020
  06:08:32 STEP 1021 H=1021:00 +CPU= 22.476
  06:08:55 STEP 1022 H=1022:00 +CPU= 23.234
  06:09:31 STEP 1023 H=1023:00 +CPU= 35.506
  06:09:54 STEP 1024 H=1024:00 +CPU= 23.014
  06:10:17 STEP 1025 H=1025:00 +CPU= 22.644
  06:10:53 STEP 1026 H=1026:00 +CPU= 36.211
  06:11:16 STEP 1027 H=1027:00 +CPU= 23.025
  06:11:40 STEP 1028 H=1028:00 +CPU= 23.204
  06:12:15 STEP 1029 H=1029:00 +CPU= 35.442
  06:12:38 STEP 1030 H=1030:00 +CPU= 23.162
  06:13:02 STEP 1031 H=1031:00 +CPU= 23.243
  06:13:47 STEP 1032 H=1032:00 +CPU= 41.752
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_42.zip
Uploading the intermediate file: upload_file_42.zip
  06:14:10 STEP 1033 H=1033:00 +CPU= 22.471
  06:14:33 STEP 1034 H=1034:00 +CPU= 23.441
  06:15:09 STEP 1035 H=1035:00 +CPU= 35.758
  06:15:32 STEP 1036 H=1036:00 +CPU= 23.247
  06:15:56 STEP 1037 H=1037:00 +CPU= 23.516
  06:16:32 STEP 1038 H=1038:00 +CPU= 36.232
  06:16:54 STEP 1039 H=1039:00 +CPU= 22.353
  06:17:18 STEP 1040 H=1040:00 +CPU= 23.309
  06:17:53 STEP 1041 H=1041:00 +CPU= 35.508
  06:18:16 STEP 1042 H=1042:00 +CPU= 22.796
  06:18:38 STEP 1043 H=1043:00 +CPU= 22.401
  06:19:18 STEP 1044 H=1044:00 +CPU= 39.798
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001044
  06:19:41 STEP 1045 H=1045:00 +CPU= 22.677
  06:20:04 STEP 1046 H=1046:00 +CPU= 23.224
  06:20:40 STEP 1047 H=1047:00 +CPU= 35.447
  06:21:04 STEP 1048 H=1048:00 +CPU= 24.034
  06:21:29 STEP 1049 H=1049:00 +CPU= 24.870
  06:22:05 STEP 1050 H=1050:00 +CPU= 36.842
  06:22:29 STEP 1051 H=1051:00 +CPU= 22.557
  06:22:52 STEP 1052 H=1052:00 +CPU= 23.565
  06:23:28 STEP 1053 H=1053:00 +CPU= 35.754
  06:23:51 STEP 1054 H=1054:00 +CPU= 23.165
  06:24:16 STEP 1055 H=1055:00 +CPU= 23.787
  06:25:03 STEP 1056 H=1056:00 +CPU= 43.335
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_43.zip
Uploading the intermediate file: upload_file_43.zip
  06:25:27 STEP 1057 H=1057:00 +CPU= 24.118
Uploading trickle at timestep: 3801600
  06:25:51 STEP 1058 H=1058:00 +CPU= 23.416
  06:26:27 STEP 1059 H=1059:00 +CPU= 36.080
  06:26:50 STEP 1060 H=1060:00 +CPU= 23.336
  06:27:14 STEP 1061 H=1061:00 +CPU= 23.683
  06:27:52 STEP 1062 H=1062:00 +CPU= 37.076
  06:28:15 STEP 1063 H=1063:00 +CPU= 22.981
  06:28:38 STEP 1064 H=1064:00 +CPU= 23.561
  06:29:15 STEP 1065 H=1065:00 +CPU= 36.727
  06:29:39 STEP 1066 H=1066:00 +CPU= 23.885
  06:30:02 STEP 1067 H=1067:00 +CPU= 22.766
  06:30:50 STEP 1068 H=1068:00 +CPU= 39.852
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001068
  06:31:13 STEP 1069 H=1069:00 +CPU= 22.578
  06:31:37 STEP 1070 H=1070:00 +CPU= 24.202
  06:32:14 STEP 1071 H=1071:00 +CPU= 37.005
  06:32:38 STEP 1072 H=1072:00 +CPU= 23.950
  06:33:03 STEP 1073 H=1073:00 +CPU= 23.688
  06:33:41 STEP 1074 H=1074:00 +CPU= 37.288
  06:34:04 STEP 1075 H=1075:00 +CPU= 23.450
  06:34:28 STEP 1076 H=1076:00 +CPU= 24.042
  06:35:07 STEP 1077 H=1077:00 +CPU= 38.711
  06:35:32 STEP 1078 H=1078:00 +CPU= 24.723
  06:35:56 STEP 1079 H=1079:00 +CPU= 24.364
  06:36:47 STEP 1080 H=1080:00 +CPU= 43.427
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_44.zip
Uploading the intermediate file: upload_file_44.zip
  06:37:11 STEP 1081 H=1081:00 +CPU= 23.807
  06:37:35 STEP 1082 H=1082:00 +CPU= 24.392
  06:38:12 STEP 1083 H=1083:00 +CPU= 36.495
  06:38:38 STEP 1084 H=1084:00 +CPU= 25.383
  06:39:02 STEP 1085 H=1085:00 +CPU= 24.819
  06:39:40 STEP 1086 H=1086:00 +CPU= 37.114
  06:40:03 STEP 1087 H=1087:00 +CPU= 23.620
  06:40:28 STEP 1088 H=1088:00 +CPU= 24.620
  06:41:04 STEP 1089 H=1089:00 +CPU= 36.085
  06:41:27 STEP 1090 H=1090:00 +CPU= 23.307
  06:41:50 STEP 1091 H=1091:00 +CPU= 22.961
  06:42:30 STEP 1092 H=1092:00 +CPU= 39.708
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001092
  06:42:53 STEP 1093 H=1093:00 +CPU= 23.228
  06:43:19 STEP 1094 H=1094:00 +CPU= 25.437
  06:43:55 STEP 1095 H=1095:00 +CPU= 36.575
  06:44:20 STEP 1096 H=1096:00 +CPU= 23.798
  06:44:44 STEP 1097 H=1097:00 +CPU= 23.446
  06:45:21 STEP 1098 H=1098:00 +CPU= 37.060
  06:45:43 STEP 1099 H=1099:00 +CPU= 22.650
  06:46:08 STEP 1100 H=1100:00 +CPU= 24.613
  06:46:45 STEP 1101 H=1101:00 +CPU= 36.734
  06:47:11 STEP 1102 H=1102:00 +CPU= 25.498
  06:47:36 STEP 1103 H=1103:00 +CPU= 24.517
  06:48:21 STEP 1104 H=1104:00 +CPU= 44.484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_45.zip
  06:48:45 STEP 1105 H=1105:00 +CPU= 22.983
Uploading the intermediate file: upload_file_45.zip
  06:49:09 STEP 1106 H=1106:00 +CPU= 24.439
  06:49:47 STEP 1107 H=1107:00 +CPU= 37.869
  06:50:11 STEP 1108 H=1108:00 +CPU= 23.196
  06:50:34 STEP 1109 H=1109:00 +CPU= 22.772
  06:51:11 STEP 1110 H=1110:00 +CPU= 37.441
  06:51:34 STEP 1111 H=1111:00 +CPU= 22.787
  06:51:58 STEP 1112 H=1112:00 +CPU= 23.488
  06:52:34 STEP 1113 H=1113:00 +CPU= 36.572
  06:52:58 STEP 1114 H=1114:00 +CPU= 23.320
  06:53:20 STEP 1115 H=1115:00 +CPU= 22.818
  06:54:00 STEP 1116 H=1116:00 +CPU= 40.047
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001116
  06:54:23 STEP 1117 H=1117:00 +CPU= 22.403
  06:54:46 STEP 1118 H=1118:00 +CPU= 23.115
  06:55:22 STEP 1119 H=1119:00 +CPU= 36.321
  06:55:46 STEP 1120 H=1120:00 +CPU= 23.226
  06:56:08 STEP 1121 H=1121:00 +CPU= 22.566
  06:56:45 STEP 1122 H=1122:00 +CPU= 36.790
  06:57:07 STEP 1123 H=1123:00 +CPU= 22.531
  06:57:31 STEP 1124 H=1124:00 +CPU= 23.293
  06:58:08 STEP 1125 H=1125:00 +CPU= 36.928
  06:58:31 STEP 1126 H=1126:00 +CPU= 23.162
  06:58:58 STEP 1127 H=1127:00 +CPU= 23.220
  06:59:40 STEP 1128 H=1128:00 +CPU= 42.231
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_46.zip
Uploading the intermediate file: upload_file_46.zip
  07:00:02 STEP 1129 H=1129:00 +CPU= 22.299
  07:00:27 STEP 1130 H=1130:00 +CPU= 24.242
  07:01:02 STEP 1131 H=1131:00 +CPU= 35.774
  07:01:26 STEP 1132 H=1132:00 +CPU= 23.369
  07:01:48 STEP 1133 H=1133:00 +CPU= 22.406
  07:02:25 STEP 1134 H=1134:00 +CPU= 36.946
  07:02:48 STEP 1135 H=1135:00 +CPU= 22.471
  07:03:11 STEP 1136 H=1136:00 +CPU= 23.396
  07:03:47 STEP 1137 H=1137:00 +CPU= 35.859
  07:04:11 STEP 1138 H=1138:00 +CPU= 23.777
  07:04:35 STEP 1139 H=1139:00 +CPU= 23.374
  07:05:14 STEP 1140 H=1140:00 +CPU= 39.860
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001140
  07:05:37 STEP 1141 H=1141:00 +CPU= 22.627
  07:06:01 STEP 1142 H=1142:00 +CPU= 23.626
  07:06:37 STEP 1143 H=1143:00 +CPU= 36.064
  07:07:00 STEP 1144 H=1144:00 +CPU= 23.237
  07:07:24 STEP 1145 H=1145:00 +CPU= 23.519
  07:08:01 STEP 1146 H=1146:00 +CPU= 36.730
  07:08:23 STEP 1147 H=1147:00 +CPU= 22.481
  07:08:47 STEP 1148 H=1148:00 +CPU= 23.255
  07:09:23 STEP 1149 H=1149:00 +CPU= 36.110
  07:09:46 STEP 1150 H=1150:00 +CPU= 23.308
  07:10:12 STEP 1151 H=1151:00 +CPU= 23.312
  07:11:07 STEP 1152 H=1152:00 +CPU= 44.075
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_47.zip
Uploading the intermediate file: upload_file_47.zip
  07:11:29 STEP 1153 H=1153:00 +CPU= 22.467
  07:11:53 STEP 1154 H=1154:00 +CPU= 23.416
  07:12:30 STEP 1155 H=1155:00 +CPU= 36.224
  07:12:53 STEP 1156 H=1156:00 +CPU= 23.005
  07:13:15 STEP 1157 H=1157:00 +CPU= 21.939
  07:13:52 STEP 1158 H=1158:00 +CPU= 36.855
  07:14:15 STEP 1159 H=1159:00 +CPU= 22.686
  07:14:40 STEP 1160 H=1160:00 +CPU= 24.180
  07:15:16 STEP 1161 H=1161:00 +CPU= 36.268
  07:15:40 STEP 1162 H=1162:00 +CPU= 24.081
  07:16:03 STEP 1163 H=1163:00 +CPU= 22.711
  07:16:43 STEP 1164 H=1164:00 +CPU= 39.917
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001164
  07:17:06 STEP 1165 H=1165:00 +CPU= 22.685
  07:17:29 STEP 1166 H=1166:00 +CPU= 23.265
  07:18:06 STEP 1167 H=1167:00 +CPU= 36.465
  07:18:31 STEP 1168 H=1168:00 +CPU= 25.109
  07:18:54 STEP 1169 H=1169:00 +CPU= 22.622
  07:19:32 STEP 1170 H=1170:00 +CPU= 37.160
  07:19:56 STEP 1171 H=1171:00 +CPU= 24.174
  07:20:22 STEP 1172 H=1172:00 +CPU= 25.270
  07:20:59 STEP 1173 H=1173:00 +CPU= 36.289
  07:21:22 STEP 1174 H=1174:00 +CPU= 23.278
  07:21:47 STEP 1175 H=1175:00 +CPU= 24.707
  07:22:36 STEP 1176 H=1176:00 +CPU= 43.331
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_48.zip
Uploading the intermediate file: upload_file_48.zip
  07:22:59 STEP 1177 H=1177:00 +CPU= 23.369
  07:23:23 STEP 1178 H=1178:00 +CPU= 24.004
  07:24:00 STEP 1179 H=1179:00 +CPU= 37.019
  07:24:24 STEP 1180 H=1180:00 +CPU= 23.498
  07:24:48 STEP 1181 H=1181:00 +CPU= 23.814
  07:25:28 STEP 1182 H=1182:00 +CPU= 39.325
  07:25:51 STEP 1183 H=1183:00 +CPU= 22.931
  07:26:14 STEP 1184 H=1184:00 +CPU= 23.391
  07:26:51 STEP 1185 H=1185:00 +CPU= 36.273
  07:27:14 STEP 1186 H=1186:00 +CPU= 23.783
  07:27:37 STEP 1187 H=1187:00 +CPU= 22.755
  07:28:18 STEP 1188 H=1188:00 +CPU= 40.271
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001188
  07:28:41 STEP 1189 H=1189:00 +CPU= 22.957
  07:29:05 STEP 1190 H=1190:00 +CPU= 23.746
  07:29:40 STEP 1191 H=1191:00 +CPU= 35.913
  07:30:04 STEP 1192 H=1192:00 +CPU= 23.326
  07:30:27 STEP 1193 H=1193:00 +CPU= 23.106
  07:31:04 STEP 1194 H=1194:00 +CPU= 36.687
  07:31:27 STEP 1195 H=1195:00 +CPU= 22.877
  07:31:50 STEP 1196 H=1196:00 +CPU= 23.515
  07:32:26 STEP 1197 H=1197:00 +CPU= 36.100
  07:32:50 STEP 1198 H=1198:00 +CPU= 24.132
  07:33:14 STEP 1199 H=1199:00 +CPU= 23.835
  07:33:58 STEP 1200 H=1200:00 +CPU= 42.542
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_49.zip
Uploading the intermediate file: upload_file_49.zip
  07:34:22 STEP 1201 H=1201:00 +CPU= 23.109
  07:34:46 STEP 1202 H=1202:00 +CPU= 24.371
  07:35:22 STEP 1203 H=1203:00 +CPU= 36.316
  07:35:47 STEP 1204 H=1204:00 +CPU= 25.001
  07:36:12 STEP 1205 H=1205:00 +CPU= 24.647
  07:36:50 STEP 1206 H=1206:00 +CPU= 37.520
  07:37:13 STEP 1207 H=1207:00 +CPU= 22.991
  07:37:37 STEP 1208 H=1208:00 +CPU= 24.662
  07:38:14 STEP 1209 H=1209:00 +CPU= 36.240
  07:38:37 STEP 1210 H=1210:00 +CPU= 23.328
  07:39:00 STEP 1211 H=1211:00 +CPU= 22.893
  07:39:40 STEP 1212 H=1212:00 +CPU= 40.488
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001212
  07:40:03 STEP 1213 H=1213:00 +CPU= 22.820
  07:40:27 STEP 1214 H=1214:00 +CPU= 23.618
  07:41:03 STEP 1215 H=1215:00 +CPU= 36.036
  07:41:26 STEP 1216 H=1216:00 +CPU= 23.264
  07:41:50 STEP 1217 H=1217:00 +CPU= 23.825
  07:42:27 STEP 1218 H=1218:00 +CPU= 37.002
  07:42:50 STEP 1219 H=1219:00 +CPU= 22.668
  07:43:13 STEP 1220 H=1220:00 +CPU= 23.712
  07:43:50 STEP 1221 H=1221:00 +CPU= 36.083
  07:44:14 STEP 1222 H=1222:00 +CPU= 23.721
  07:44:38 STEP 1223 H=1223:00 +CPU= 23.475
  07:45:31 STEP 1224 H=1224:00 +CPU= 42.713
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_50.zip
Uploading the intermediate file: upload_file_50.zip
  07:45:55 STEP 1225 H=1225:00 +CPU= 23.977
  07:46:19 STEP 1226 H=1226:00 +CPU= 23.942
  07:46:55 STEP 1227 H=1227:00 +CPU= 36.062
  07:47:20 STEP 1228 H=1228:00 +CPU= 23.283
  07:47:43 STEP 1229 H=1229:00 +CPU= 23.530
  07:48:20 STEP 1230 H=1230:00 +CPU= 37.033
  07:48:43 STEP 1231 H=1231:00 +CPU= 23.098
  07:49:07 STEP 1232 H=1232:00 +CPU= 23.525
  07:49:43 STEP 1233 H=1233:00 +CPU= 36.349
  07:50:07 STEP 1234 H=1234:00 +CPU= 23.841
  07:50:30 STEP 1235 H=1235:00 +CPU= 22.760
  07:51:10 STEP 1236 H=1236:00 +CPU= 40.515
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001236
  07:51:33 STEP 1237 H=1237:00 +CPU= 22.511
  07:51:56 STEP 1238 H=1238:00 +CPU= 23.332
  07:52:32 STEP 1239 H=1239:00 +CPU= 35.943
  07:52:56 STEP 1240 H=1240:00 +CPU= 23.779
  07:53:19 STEP 1241 H=1241:00 +CPU= 22.798
  07:53:56 STEP 1242 H=1242:00 +CPU= 36.790
  07:54:19 STEP 1243 H=1243:00 +CPU= 22.972
  07:54:43 STEP 1244 H=1244:00 +CPU= 24.045
  07:55:19 STEP 1245 H=1245:00 +CPU= 36.352
  07:55:43 STEP 1246 H=1246:00 +CPU= 24.511
  07:56:11 STEP 1247 H=1247:00 +CPU= 23.605
  07:56:56 STEP 1248 H=1248:00 +CPU= 43.494
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_51.zip
  07:57:20 STEP 1249 H=1249:00 +CPU= 23.414
Uploading the intermediate file: upload_file_51.zip
  07:57:44 STEP 1250 H=1250:00 +CPU= 23.806
  07:58:20 STEP 1251 H=1251:00 +CPU= 36.696
  07:58:44 STEP 1252 H=1252:00 +CPU= 23.897
  07:59:07 STEP 1253 H=1253:00 +CPU= 22.704
  07:59:44 STEP 1254 H=1254:00 +CPU= 36.661
  08:00:08 STEP 1255 H=1255:00 +CPU= 23.825
  08:00:31 STEP 1256 H=1256:00 +CPU= 23.663
  08:01:07 STEP 1257 H=1257:00 +CPU= 36.004
  08:01:31 STEP 1258 H=1258:00 +CPU= 23.313
  08:01:53 STEP 1259 H=1259:00 +CPU= 22.541
  08:02:33 STEP 1260 H=1260:00 +CPU= 39.578
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001260
  08:02:56 STEP 1261 H=1261:00 +CPU= 22.571
  08:03:19 STEP 1262 H=1262:00 +CPU= 23.560
  08:03:56 STEP 1263 H=1263:00 +CPU= 36.898
  08:04:20 STEP 1264 H=1264:00 +CPU= 23.395
  08:04:42 STEP 1265 H=1265:00 +CPU= 22.748
  08:05:19 STEP 1266 H=1266:00 +CPU= 36.447
  08:05:42 STEP 1267 H=1267:00 +CPU= 23.012
  08:06:06 STEP 1268 H=1268:00 +CPU= 23.274
  08:06:41 STEP 1269 H=1269:00 +CPU= 35.721
  08:07:05 STEP 1270 H=1270:00 +CPU= 23.170
  08:07:28 STEP 1271 H=1271:00 +CPU= 23.191
  08:08:15 STEP 1272 H=1272:00 +CPU= 46.359
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_52.zip
Uploading the intermediate file: upload_file_52.zip
  08:08:40 STEP 1273 H=1273:00 +CPU= 24.924
  08:09:04 STEP 1274 H=1274:00 +CPU= 23.805
  08:09:39 STEP 1275 H=1275:00 +CPU= 35.890
  08:10:03 STEP 1276 H=1276:00 +CPU= 23.380
  08:10:27 STEP 1277 H=1277:00 +CPU= 24.024
  08:11:04 STEP 1278 H=1278:00 +CPU= 36.883
  08:11:27 STEP 1279 H=1279:00 +CPU= 22.545
  08:11:51 STEP 1280 H=1280:00 +CPU= 23.516
  08:12:27 STEP 1281 H=1281:00 +CPU= 36.483
  08:12:51 STEP 1282 H=1282:00 +CPU= 24.068
  08:13:16 STEP 1283 H=1283:00 +CPU= 24.818
  08:13:59 STEP 1284 H=1284:00 +CPU= 40.890
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001284
  08:14:22 STEP 1285 H=1285:00 +CPU= 23.099
  08:14:45 STEP 1286 H=1286:00 +CPU= 23.356
  08:15:21 STEP 1287 H=1287:00 +CPU= 35.891
  08:15:44 STEP 1288 H=1288:00 +CPU= 23.210
  08:16:07 STEP 1289 H=1289:00 +CPU= 22.742
  08:16:43 STEP 1290 H=1290:00 +CPU= 36.318
  08:17:06 STEP 1291 H=1291:00 +CPU= 23.164
  08:17:30 STEP 1292 H=1292:00 +CPU= 23.715
  08:18:06 STEP 1293 H=1293:00 +CPU= 36.067
  08:18:30 STEP 1294 H=1294:00 +CPU= 24.110
  08:18:54 STEP 1295 H=1295:00 +CPU= 23.477
  08:19:36 STEP 1296 H=1296:00 +CPU= 41.867
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_53.zip
Uploading the intermediate file: upload_file_53.zip
  08:19:59 STEP 1297 H=1297:00 +CPU= 22.594
Uploading trickle at timestep: 4665600
  08:20:23 STEP 1298 H=1298:00 +CPU= 24.034
  08:21:00 STEP 1299 H=1299:00 +CPU= 36.801
  08:21:23 STEP 1300 H=1300:00 +CPU= 23.523
  08:21:47 STEP 1301 H=1301:00 +CPU= 23.695
  08:22:24 STEP 1302 H=1302:00 +CPU= 37.009
  08:22:47 STEP 1303 H=1303:00 +CPU= 22.542
  08:23:11 STEP 1304 H=1304:00 +CPU= 24.420
  08:23:48 STEP 1305 H=1305:00 +CPU= 36.605
  08:24:12 STEP 1306 H=1306:00 +CPU= 23.894
  08:24:36 STEP 1307 H=1307:00 +CPU= 24.212
  08:25:17 STEP 1308 H=1308:00 +CPU= 40.375
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001308
  08:25:40 STEP 1309 H=1309:00 +CPU= 23.867
  08:26:04 STEP 1310 H=1310:00 +CPU= 23.652
  08:26:40 STEP 1311 H=1311:00 +CPU= 36.131
  08:27:04 STEP 1312 H=1312:00 +CPU= 23.253
  08:27:27 STEP 1313 H=1313:00 +CPU= 23.332
  08:28:06 STEP 1314 H=1314:00 +CPU= 38.660
  08:28:29 STEP 1315 H=1315:00 +CPU= 22.900
  08:28:52 STEP 1316 H=1316:00 +CPU= 23.500
  08:29:29 STEP 1317 H=1317:00 +CPU= 35.967
  08:29:52 STEP 1318 H=1318:00 +CPU= 23.528
  08:30:18 STEP 1319 H=1319:00 +CPU= 24.143
  08:31:07 STEP 1320 H=1320:00 +CPU= 42.694
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_54.zip
Uploading the intermediate file: upload_file_54.zip
  08:31:29 STEP 1321 H=1321:00 +CPU= 22.653
  08:31:53 STEP 1322 H=1322:00 +CPU= 23.625
  08:32:29 STEP 1323 H=1323:00 +CPU= 35.751
  08:32:52 STEP 1324 H=1324:00 +CPU= 23.355
  08:33:16 STEP 1325 H=1325:00 +CPU= 23.135
  08:33:53 STEP 1326 H=1326:00 +CPU= 37.091
  08:34:16 STEP 1327 H=1327:00 +CPU= 22.350
  08:34:39 STEP 1328 H=1328:00 +CPU= 23.876
  08:35:17 STEP 1329 H=1329:00 +CPU= 37.043
  08:35:41 STEP 1330 H=1330:00 +CPU= 24.947
  08:36:05 STEP 1331 H=1331:00 +CPU= 23.471
  08:36:45 STEP 1332 H=1332:00 +CPU= 39.957
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001332
  08:37:08 STEP 1333 H=1333:00 +CPU= 22.942
  08:37:32 STEP 1334 H=1334:00 +CPU= 24.015
  08:38:09 STEP 1335 H=1335:00 +CPU= 36.691
  08:38:33 STEP 1336 H=1336:00 +CPU= 23.967
  08:38:55 STEP 1337 H=1337:00 +CPU= 22.783
  08:39:32 STEP 1338 H=1338:00 +CPU= 36.777
  08:39:56 STEP 1339 H=1339:00 +CPU= 22.496
  08:40:19 STEP 1340 H=1340:00 +CPU= 23.496
  08:40:56 STEP 1341 H=1341:00 +CPU= 36.976
  08:41:37 STEP 1342 H=1342:00 +CPU= 24.051
  08:42:01 STEP 1343 H=1343:00 +CPU= 24.307
  08:42:45 STEP 1344 H=1344:00 +CPU= 42.419
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_55.zip
  08:43:06 STEP 1345 H=1345:00 +CPU= 21.487
Uploading the intermediate file: upload_file_55.zip
  08:43:32 STEP 1346 H=1346:00 +CPU= 23.939
  08:44:08 STEP 1347 H=1347:00 +CPU= 36.161
  08:44:31 STEP 1348 H=1348:00 +CPU= 22.846
  08:44:54 STEP 1349 H=1349:00 +CPU= 22.383
  08:45:31 STEP 1350 H=1350:00 +CPU= 37.131
  08:45:53 STEP 1351 H=1351:00 +CPU= 22.375
  08:46:17 STEP 1352 H=1352:00 +CPU= 23.210
  08:46:54 STEP 1353 H=1353:00 +CPU= 37.104
  08:47:18 STEP 1354 H=1354:00 +CPU= 24.221
  08:47:42 STEP 1355 H=1355:00 +CPU= 23.487
  08:48:30 STEP 1356 H=1356:00 +CPU= 42.004
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001356
  08:48:54 STEP 1357 H=1357:00 +CPU= 23.772
  08:49:18 STEP 1358 H=1358:00 +CPU= 23.963
  08:49:55 STEP 1359 H=1359:00 +CPU= 37.288
  08:50:19 STEP 1360 H=1360:00 +CPU= 23.274
  08:50:41 STEP 1361 H=1361:00 +CPU= 22.544
  08:51:18 STEP 1362 H=1362:00 +CPU= 36.720
  08:51:40 STEP 1363 H=1363:00 +CPU= 22.526
  08:52:04 STEP 1364 H=1364:00 +CPU= 23.407
  08:52:40 STEP 1365 H=1365:00 +CPU= 36.337
  08:53:05 STEP 1366 H=1366:00 +CPU= 25.351
  08:53:38 STEP 1367 H=1367:00 +CPU= 22.918
  08:54:26 STEP 1368 H=1368:00 +CPU= 42.423
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_56.zip
Uploading the intermediate file: upload_file_56.zip
  08:54:49 STEP 1369 H=1369:00 +CPU= 23.211
  08:55:14 STEP 1370 H=1370:00 +CPU= 24.862
  08:55:51 STEP 1371 H=1371:00 +CPU= 36.980
  08:56:14 STEP 1372 H=1372:00 +CPU= 23.658
  08:56:40 STEP 1373 H=1373:00 +CPU= 25.277
  08:57:18 STEP 1374 H=1374:00 +CPU= 38.588
  08:57:41 STEP 1375 H=1375:00 +CPU= 22.495
  08:58:05 STEP 1376 H=1376:00 +CPU= 24.004
  08:58:44 STEP 1377 H=1377:00 +CPU= 38.047
  08:59:07 STEP 1378 H=1378:00 +CPU= 23.262
  08:59:31 STEP 1379 H=1379:00 +CPU= 22.968
  09:00:12 STEP 1380 H=1380:00 +CPU= 40.362
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001380
  09:00:35 STEP 1381 H=1381:00 +CPU= 23.011
  09:01:00 STEP 1382 H=1382:00 +CPU= 25.201
  09:01:37 STEP 1383 H=1383:00 +CPU= 36.498
  09:06:56 STEP 1384 H=1384:00 +CPU= 24.291
  09:07:23 STEP 1385 H=1385:00 +CPU= 22.404
  09:08:01 STEP 1386 H=1386:00 +CPU= 37.899
  09:08:24 STEP 1387 H=1387:00 +CPU= 22.814
  09:08:53 STEP 1388 H=1388:00 +CPU= 23.320
  09:09:32 STEP 1389 H=1389:00 +CPU= 35.748
  09:09:55 STEP 1390 H=1390:00 +CPU= 23.092
  09:10:19 STEP 1391 H=1391:00 +CPU= 23.330
  09:11:03 STEP 1392 H=1392:00 +CPU= 42.868
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_57.zip
Uploading the intermediate file: upload_file_57.zip
  09:11:25 STEP 1393 H=1393:00 +CPU= 22.626
  09:11:50 STEP 1394 H=1394:00 +CPU= 23.426
  09:12:26 STEP 1395 H=1395:00 +CPU= 36.067
  09:12:50 STEP 1396 H=1396:00 +CPU= 23.529
  09:13:12 STEP 1397 H=1397:00 +CPU= 22.361
  09:13:50 STEP 1398 H=1398:00 +CPU= 37.569
  09:14:14 STEP 1399 H=1399:00 +CPU= 24.161
  09:14:38 STEP 1400 H=1400:00 +CPU= 23.898
  09:15:15 STEP 1401 H=1401:00 +CPU= 36.962
  09:15:39 STEP 1402 H=1402:00 +CPU= 23.802
  09:16:02 STEP 1403 H=1403:00 +CPU= 22.697
  09:16:43 STEP 1404 H=1404:00 +CPU= 39.731
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001404
  09:17:06 STEP 1405 H=1405:00 +CPU= 23.348
  09:17:30 STEP 1406 H=1406:00 +CPU= 23.264
  09:18:06 STEP 1407 H=1407:00 +CPU= 36.119
  09:18:29 STEP 1408 H=1408:00 +CPU= 23.396
  09:18:52 STEP 1409 H=1409:00 +CPU= 22.618
  09:19:28 STEP 1410 H=1410:00 +CPU= 36.435
  09:19:51 STEP 1411 H=1411:00 +CPU= 23.084
  09:20:15 STEP 1412 H=1412:00 +CPU= 23.583
  09:20:51 STEP 1413 H=1413:00 +CPU= 36.258
  09:21:15 STEP 1414 H=1414:00 +CPU= 23.255
  09:21:40 STEP 1415 H=1415:00 +CPU= 24.497
  09:22:34 STEP 1416 H=1416:00 +CPU= 42.689
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_58.zip
Uploading the intermediate file: upload_file_58.zip
  09:22:57 STEP 1417 H=1417:00 +CPU= 22.736
  09:23:20 STEP 1418 H=1418:00 +CPU= 23.557
  09:23:57 STEP 1419 H=1419:00 +CPU= 36.579
  09:24:20 STEP 1420 H=1420:00 +CPU= 22.992
  09:24:43 STEP 1421 H=1421:00 +CPU= 23.359
  09:25:22 STEP 1422 H=1422:00 +CPU= 37.967
  09:25:46 STEP 1423 H=1423:00 +CPU= 23.163
  09:26:10 STEP 1424 H=1424:00 +CPU= 24.230
  09:26:49 STEP 1425 H=1425:00 +CPU= 38.302
  09:27:13 STEP 1426 H=1426:00 +CPU= 23.502
  09:27:35 STEP 1427 H=1427:00 +CPU= 22.669
  09:28:19 STEP 1428 H=1428:00 +CPU= 40.428
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001428
  09:28:41 STEP 1429 H=1429:00 +CPU= 22.551
  09:29:05 STEP 1430 H=1430:00 +CPU= 23.342
  09:29:40 STEP 1431 H=1431:00 +CPU= 35.866
  09:30:04 STEP 1432 H=1432:00 +CPU= 23.244
  09:30:26 STEP 1433 H=1433:00 +CPU= 22.743
  09:31:04 STEP 1434 H=1434:00 +CPU= 37.317
  09:31:26 STEP 1435 H=1435:00 +CPU= 22.632
  09:31:50 STEP 1436 H=1436:00 +CPU= 23.419
  09:32:27 STEP 1437 H=1437:00 +CPU= 36.349
  09:32:50 STEP 1438 H=1438:00 +CPU= 23.397
  09:33:16 STEP 1439 H=1439:00 +CPU= 23.376
  09:34:03 STEP 1440 H=1440:00 +CPU= 42.303
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844_59.zip
  09:34:27 STEP 1441 H=1441:00 +CPU= 22.716
Uploading the intermediate file: upload_file_59.zip
  09:34:51 STEP 1442 H=1442:00 +CPU= 24.837
  09:35:29 STEP 1443 H=1443:00 +CPU= 37.045
  09:35:53 STEP 1444 H=1444:00 +CPU= 24.074
  09:36:16 STEP 1445 H=1445:00 +CPU= 22.574
  09:36:54 STEP 1446 H=1446:00 +CPU= 37.426
  09:37:18 STEP 1447 H=1447:00 +CPU= 23.003
  09:37:42 STEP 1448 H=1448:00 +CPU= 24.025
  09:38:18 STEP 1449 H=1449:00 +CPU= 36.513
  09:38:42 STEP 1450 H=1450:00 +CPU= 23.439
  09:39:05 STEP 1451 H=1451:00 +CPU= 22.780
  09:39:46 STEP 1452 H=1452:00 +CPU= 41.134
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001452
  09:40:09 STEP 1453 H=1453:00 +CPU= 22.858
  09:40:33 STEP 1454 H=1454:00 +CPU= 23.171
  09:41:09 STEP 1455 H=1455:00 +CPU= 36.121
  09:41:32 STEP 1456 H=1456:00 +CPU= 23.465
  09:41:55 STEP 1457 H=1457:00 +CPU= 22.585
  09:42:33 STEP 1458 H=1458:00 +CPU= 37.460
  09:42:56 STEP 1459 H=1459:00 +CPU= 22.811
  09:43:19 STEP 1460 H=1460:00 +CPU= 23.303
  09:43:55 STEP 1461 H=1461:00 +CPU= 36.051
  09:44:19 STEP 1462 H=1462:00 +CPU= 23.245
  09:44:41 STEP 1463 H=1463:00 +CPU= 22.555
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0377
(argv4) batchid: 982
(argv5) wuid: 12199021
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199021
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_69ba2fc08a9fdc0dc309823adaa162d0 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0377_2013050100_123_982_12199021.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0377_2013050100_123_982_12199021.zip
ic_ancil_file: ic_ancil_12199021
ifsdata_file: ifsdata_12199021
climate_data_file: clim_data_12199021
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e08d0a41749b77d419bcd0b91783a0f6 to: /var/lib/boinc-client/slots/1/ic_ancil_12199021.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12199021.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ca669d573b9914d7c2b1554c7d2b85cc to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12199021.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12199021.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12199021.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12199021.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12199021.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12199021.xml
-- Model is restarting --
Adjusting last_iter, 1463, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12199021.xml
last_cpu_time: 42797
upload_file_number: 60
last_iter: 1439
last_upload: 5184000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0377_2013050100_123_982_12199021_2_r1856909844
The child process has been launched with process id: 438414
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094709:1673916429:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
 ABORT!    1 !! *** WAVE MODEL HAS ABORTED 
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094805:1673916485:55.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3847 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094805:1673916485:55.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094805:1673916485:55.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094805:1673916485:55.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094805:1673916485:55.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094805:1673916485:55.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094805:1673916485:55.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094805:1673916485:55.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        WVCOUPLE 
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094805:1673916485:55.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         WAVEMDL 
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094805:1673916485:55.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          INITMDL 
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094805:1673916485:55.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           USERIN 
 SDL_TRACEBACK: Calling INTEL_TRBK, THRD =            1
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/1/oifs_43r3_model.exe' : sigcontextptr=0x7ffdc9ea79a0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with addr2line-cmd
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/1/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x81d89e 0x15f8aca 0x15d51c8 0x166b07f 0x15d7ff0 0x8711e0 0x7e0165 0x55e4ad 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x81d89e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x15f8aca] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x15d51c8] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x166b07f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x15d7ff0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x8711e0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x7e0165] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x55e4ad] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x5471c9] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x546412] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x545e28] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x40708d] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40220f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4021a2] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x1dca390] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40206e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 SDL_TRACEBACK: Done INTEL_TRBK, THRD =            1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094806:1673916486:56.532] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3847MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094806:1673916486:56.532] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094806:1673916486:56.532] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094806:1673916486:56.532] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x750006b08e], nsigs = 1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094806:1673916486:56.532] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094807:1673916487:57.532] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1
[EC_DRHOOK:debian:1:1:438414:438414] [20230117:094807:1673916487:57.532] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC903B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008C2880  sdl_mod_mp_sdl_sr         100  sdl_mod.F90
oifs_43r3_model.e  000000000081D8AF  abor1_                     41  abor1.F90
oifs_43r3_model.e  00000000015F8ACA  mpabort_                   50  mpabort.F
oifs_43r3_model.e  00000000015D51C8  abort1_                    44  abort1.F
oifs_43r3_model.e  000000000166B07F  userin_                   553  userin.F
oifs_43r3_model.e  00000000015D7FF0  initmdl_                  452  initmdl.F
oifs_43r3_model.e  00000000008711E0  wavemdl_                  393  wavemdl.F
oifs_43r3_model.e  00000000007E0165  wvcouple_                 260  wvcouple.F90
oifs_43r3_model.e  000000000055E4AD  cnt4_                    1283  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:23:09 1538037 22298427 oifs_43r3_ps_0377_2013050100_123_982_12199021_2 4,665,600 38,105 0.0082
09 May 2023 16:23:09 1538037 22298427 oifs_43r3_ps_0377_2013050100_123_982_12199021_2 4,665,600 38,105 0.0082
09 May 2023 16:22:59 1538037 22298427 oifs_43r3_ps_0377_2013050100_123_982_12199021_2 3,801,600 31,315 0.0082
09 May 2023 16:22:49 1538037 22298427 oifs_43r3_ps_0377_2013050100_123_982_12199021_2 2,592,000 21,624 0.0083
09 May 2023 16:22:41 1538037 22298427 oifs_43r3_ps_0377_2013050100_123_982_12199021_2 1,728,000 14,425 0.0083
09 May 2023 16:22:32 1538037 22298427 oifs_43r3_ps_0377_2013050100_123_982_12199021_2 864,000 7,218 0.0084


©2024 climateprediction.net