climateprediction.net home page
Task 22251901

Task 22251901

Name oifs_43r3_ps_0032_1982050100_123_951_12167676_0
Workunit 12167676
Created 20 Dec 2022, 21:04:39 UTC
Sent 12 Jan 2023, 21:39:14 UTC
Report deadline 11 Feb 2023, 21:39:14 UTC
Received 16 Jan 2023, 21:20:34 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1493665
Run time 16 hours 41 min 16 sec
CPU time 14 hours 40 min 35 sec
Validate state Invalid
Credit 6,894.72
Device peak FLOPS 4.47 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,704.36 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,681.90 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
 the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_43.zip
Uploading the intermediate file: upload_file_43.zip
  06:52:55 STEP 1057 H=1057:00 +CPU= 26.604
  06:53:26 STEP 1058 H=1058:00 +CPU= 28.218
  06:54:12 STEP 1059 H=1059:00 +CPU= 42.658
  06:54:43 STEP 1060 H=1060:00 +CPU= 27.685
  06:55:12 STEP 1061 H=1061:00 +CPU= 26.811
  06:56:00 STEP 1062 H=1062:00 +CPU= 43.299
  06:56:29 STEP 1063 H=1063:00 +CPU= 27.146
  06:57:00 STEP 1064 H=1064:00 +CPU= 28.099
  06:57:47 STEP 1065 H=1065:00 +CPU= 43.244
  06:58:19 STEP 1066 H=1066:00 +CPU= 28.323
  06:58:49 STEP 1067 H=1067:00 +CPU= 27.236
  06:59:41 STEP 1068 H=1068:00 +CPU= 47.759
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001068
  07:00:11 STEP 1069 H=1069:00 +CPU= 27.376
  07:00:41 STEP 1070 H=1070:00 +CPU= 27.707
  07:01:28 STEP 1071 H=1071:00 +CPU= 42.965
  07:01:58 STEP 1072 H=1072:00 +CPU= 27.916
  07:02:29 STEP 1073 H=1073:00 +CPU= 27.667
  07:03:16 STEP 1074 H=1074:00 +CPU= 43.593
  07:03:46 STEP 1075 H=1075:00 +CPU= 27.478
  07:04:16 STEP 1076 H=1076:00 +CPU= 27.777
  07:05:02 STEP 1077 H=1077:00 +CPU= 42.405
  07:05:33 STEP 1078 H=1078:00 +CPU= 27.707
  07:06:02 STEP 1079 H=1079:00 +CPU= 26.852
  07:06:56 STEP 1080 H=1080:00 +CPU= 49.764
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_44.zip
Uploading the intermediate file: upload_file_44.zip
  07:07:25 STEP 1081 H=1081:00 +CPU= 26.983
  07:07:56 STEP 1082 H=1082:00 +CPU= 28.268
  07:08:43 STEP 1083 H=1083:00 +CPU= 42.482
  07:09:14 STEP 1084 H=1084:00 +CPU= 27.997
  07:09:44 STEP 1085 H=1085:00 +CPU= 27.852
  07:10:31 STEP 1086 H=1086:00 +CPU= 43.616
  07:11:00 STEP 1087 H=1087:00 +CPU= 27.256
  07:11:30 STEP 1088 H=1088:00 +CPU= 27.847
  07:12:16 STEP 1089 H=1089:00 +CPU= 42.125
  07:12:47 STEP 1090 H=1090:00 +CPU= 28.155
  07:13:17 STEP 1091 H=1091:00 +CPU= 27.312
  07:14:08 STEP 1092 H=1092:00 +CPU= 47.122
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001092
  07:14:38 STEP 1093 H=1093:00 +CPU= 27.397
  07:15:09 STEP 1094 H=1094:00 +CPU= 28.267
  07:15:54 STEP 1095 H=1095:00 +CPU= 41.800
  07:16:23 STEP 1096 H=1096:00 +CPU= 26.908
  07:16:51 STEP 1097 H=1097:00 +CPU= 26.160
  07:17:37 STEP 1098 H=1098:00 +CPU= 44.057
  07:18:06 STEP 1099 H=1099:00 +CPU= 26.861
  07:18:36 STEP 1100 H=1100:00 +CPU= 28.005
  07:19:22 STEP 1101 H=1101:00 +CPU= 43.146
  07:19:51 STEP 1102 H=1102:00 +CPU= 27.467
  07:20:22 STEP 1103 H=1103:00 +CPU= 28.215
  07:21:16 STEP 1104 H=1104:00 +CPU= 49.883
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_45.zip
Uploading the intermediate file: upload_file_45.zip
  07:21:46 STEP 1105 H=1105:00 +CPU= 27.547
  07:22:17 STEP 1106 H=1106:00 +CPU= 28.213
  07:23:03 STEP 1107 H=1107:00 +CPU= 42.753
  07:23:34 STEP 1108 H=1108:00 +CPU= 27.790
  07:24:04 STEP 1109 H=1109:00 +CPU= 26.610
  07:24:52 STEP 1110 H=1110:00 +CPU= 43.879
  07:25:21 STEP 1111 H=1111:00 +CPU= 27.336
  07:25:52 STEP 1112 H=1112:00 +CPU= 27.810
  07:26:38 STEP 1113 H=1113:00 +CPU= 42.898
  07:27:09 STEP 1114 H=1114:00 +CPU= 28.550
  07:27:40 STEP 1115 H=1115:00 +CPU= 27.611
  07:28:39 STEP 1116 H=1116:00 +CPU= 47.700
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001116
  07:29:08 STEP 1117 H=1117:00 +CPU= 27.002
  07:29:38 STEP 1118 H=1118:00 +CPU= 27.807
  07:30:24 STEP 1119 H=1119:00 +CPU= 41.779
  07:30:53 STEP 1120 H=1120:00 +CPU= 27.446
  07:31:23 STEP 1121 H=1121:00 +CPU= 27.330
  07:32:10 STEP 1122 H=1122:00 +CPU= 44.373
  07:32:39 STEP 1123 H=1123:00 +CPU= 27.260
  07:33:09 STEP 1124 H=1124:00 +CPU= 27.982
  07:34:00 STEP 1125 H=1125:00 +CPU= 42.511
  07:34:30 STEP 1126 H=1126:00 +CPU= 27.841
  07:35:00 STEP 1127 H=1127:00 +CPU= 27.489
  07:35:52 STEP 1128 H=1128:00 +CPU= 47.974
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_46.zip
Uploading the intermediate file: upload_file_46.zip
  07:36:21 STEP 1129 H=1129:00 +CPU= 26.465
  07:36:50 STEP 1130 H=1130:00 +CPU= 27.492
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1982050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0032
(argv4) batchid: 951
(argv5) wuid: 12167676
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/2
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_31c3615dab5d729eba03d42bbeb32b75 to: /var/lib/boinc/slots/2/oifs_43r3_ps_0032_1982050100_123_951_12167676.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_0032_1982050100_123_951_12167676.zip
ic_ancil_file: ic_ancil_12167676
ifsdata_file: ifsdata_12167676
climate_data_file: clim_data_12167676
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4e9cb166e954d87982d106778ac7c6a6 to: /var/lib/boinc/slots/2/ic_ancil_12167676.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/2/ic_ancil_12167676.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_7f02ec0281fa1eac315ddd3530915bad to: /var/lib/boinc/slots/2/ifsdata/ifsdata_12167676.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/2/ifsdata/ifsdata_12167676.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/2/159l_2/clim_data_12167676.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/2/159l_2/clim_data_12167676.zip
Checking for progress XML file: /var/lib/boinc/slots/2/progress_file_12167676.xml
Opened progress file ok : /var/lib/boinc/slots/2/progress_file_12167676.xml
-- Model is restarting --
Adjusting last_iter, 1131, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/2/progress_file_12167676.xml
last_cpu_time: 39775
upload_file_number: 47
last_iter: 1127
last_upload: 4060800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990
The child process has been launched with process id: 18068
Executing the command: /var/lib/boinc/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:18068:18068] [20230116:075237:1673880757:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:53:30 STEP 1128 H=1128:00 +CPU= 46.370
  07:53:58 STEP 1129 H=1129:00 +CPU= 25.751
  07:54:28 STEP 1130 H=1130:00 +CPU= 28.339
  07:55:12 STEP 1131 H=1131:00 +CPU= 41.154
  07:55:40 STEP 1132 H=1132:00 +CPU= 26.912
  07:56:08 STEP 1133 H=1133:00 +CPU= 26.367
  07:56:54 STEP 1134 H=1134:00 +CPU= 42.489
  07:57:21 STEP 1135 H=1135:00 +CPU= 25.846
  07:57:50 STEP 1136 H=1136:00 +CPU= 26.915
  07:58:34 STEP 1137 H=1137:00 +CPU= 40.736
  07:59:03 STEP 1138 H=1138:00 +CPU= 27.251
  07:59:31 STEP 1139 H=1139:00 +CPU= 26.195
  08:00:20 STEP 1140 H=1140:00 +CPU= 45.600
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001140
  08:00:47 STEP 1141 H=1141:00 +CPU= 25.855
  08:01:16 STEP 1142 H=1142:00 +CPU= 26.902
  08:02:00 STEP 1143 H=1143:00 +CPU= 41.067
  08:02:29 STEP 1144 H=1144:00 +CPU= 26.989
  08:02:57 STEP 1145 H=1145:00 +CPU= 26.265
  08:03:47 STEP 1146 H=1146:00 +CPU= 42.439
  08:04:14 STEP 1147 H=1147:00 +CPU= 25.532
  08:04:43 STEP 1148 H=1148:00 +CPU= 26.411
  08:05:27 STEP 1149 H=1149:00 +CPU= 41.010
  08:05:56 STEP 1150 H=1150:00 +CPU= 26.943
  08:06:25 STEP 1151 H=1151:00 +CPU= 26.771
  08:07:17 STEP 1152 H=1152:00 +CPU= 47.792
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_47.zip
Uploading the intermediate file: upload_file_47.zip
  08:07:45 STEP 1153 H=1153:00 +CPU= 26.189
  08:08:15 STEP 1154 H=1154:00 +CPU= 27.189
  08:09:00 STEP 1155 H=1155:00 +CPU= 41.626
  08:09:28 STEP 1156 H=1156:00 +CPU= 26.419
  08:09:56 STEP 1157 H=1157:00 +CPU= 25.699
  08:10:41 STEP 1158 H=1158:00 +CPU= 41.760
  08:11:09 STEP 1159 H=1159:00 +CPU= 26.113
  08:11:38 STEP 1160 H=1160:00 +CPU= 26.974
  08:12:23 STEP 1161 H=1161:00 +CPU= 40.764
  08:12:52 STEP 1162 H=1162:00 +CPU= 27.970
  08:13:21 STEP 1163 H=1163:00 +CPU= 27.696
  08:14:11 STEP 1164 H=1164:00 +CPU= 45.991
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001164
  08:14:50 STEP 1165 H=1165:00 +CPU= 26.493
  08:15:20 STEP 1166 H=1166:00 +CPU= 27.780
  08:16:05 STEP 1167 H=1167:00 +CPU= 40.886
  08:16:35 STEP 1168 H=1168:00 +CPU= 28.828
  08:17:03 STEP 1169 H=1169:00 +CPU= 25.791
  08:17:49 STEP 1170 H=1170:00 +CPU= 42.317
  08:18:16 STEP 1171 H=1171:00 +CPU= 25.582
  08:18:45 STEP 1172 H=1172:00 +CPU= 26.775
  08:19:29 STEP 1173 H=1173:00 +CPU= 41.379
  08:19:59 STEP 1174 H=1174:00 +CPU= 26.960
  08:20:28 STEP 1175 H=1175:00 +CPU= 27.070
  08:21:19 STEP 1176 H=1176:00 +CPU= 47.651
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_48.zip
Uploading the intermediate file: upload_file_48.zip
  08:21:47 STEP 1177 H=1177:00 +CPU= 25.676
  08:22:16 STEP 1178 H=1178:00 +CPU= 26.620
  08:23:00 STEP 1179 H=1179:00 +CPU= 40.734
  08:23:30 STEP 1180 H=1180:00 +CPU= 26.722
  08:23:58 STEP 1181 H=1181:00 +CPU= 25.843
  08:24:43 STEP 1182 H=1182:00 +CPU= 42.146
  08:25:11 STEP 1183 H=1183:00 +CPU= 26.129
  08:25:42 STEP 1184 H=1184:00 +CPU= 28.402
  08:26:26 STEP 1185 H=1185:00 +CPU= 41.808
  08:26:55 STEP 1186 H=1186:00 +CPU= 26.639
  08:27:22 STEP 1187 H=1187:00 +CPU= 25.479
  08:28:12 STEP 1188 H=1188:00 +CPU= 46.064
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001188
  08:28:40 STEP 1189 H=1189:00 +CPU= 25.663
  08:29:08 STEP 1190 H=1190:00 +CPU= 26.600
  08:29:54 STEP 1191 H=1191:00 +CPU= 41.603
  08:30:23 STEP 1192 H=1192:00 +CPU= 26.737
  08:30:51 STEP 1193 H=1193:00 +CPU= 26.032
  08:31:37 STEP 1194 H=1194:00 +CPU= 42.931
  08:32:06 STEP 1195 H=1195:00 +CPU= 26.121
  08:32:35 STEP 1196 H=1196:00 +CPU= 26.603
  08:33:19 STEP 1197 H=1197:00 +CPU= 40.674
  08:33:48 STEP 1198 H=1198:00 +CPU= 26.883
  08:34:18 STEP 1199 H=1199:00 +CPU= 27.127
  08:35:10 STEP 1200 H=1200:00 +CPU= 47.779
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_49.zip
Uploading the intermediate file: upload_file_49.zip
  08:35:37 STEP 1201 H=1201:00 +CPU= 25.809
  08:36:06 STEP 1202 H=1202:00 +CPU= 26.787
  08:36:50 STEP 1203 H=1203:00 +CPU= 41.119
  08:37:20 STEP 1204 H=1204:00 +CPU= 27.197
  08:37:48 STEP 1205 H=1205:00 +CPU= 26.409
  08:38:33 STEP 1206 H=1206:00 +CPU= 41.881
  08:39:01 STEP 1207 H=1207:00 +CPU= 25.944
  08:39:30 STEP 1208 H=1208:00 +CPU= 26.722
  08:40:14 STEP 1209 H=1209:00 +CPU= 40.983
  08:40:49 STEP 1210 H=1210:00 +CPU= 26.977
  08:41:19 STEP 1211 H=1211:00 +CPU= 25.979
  08:42:08 STEP 1212 H=1212:00 +CPU= 45.803
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001212
  08:42:37 STEP 1213 H=1213:00 +CPU= 26.681
  08:43:06 STEP 1214 H=1214:00 +CPU= 26.981
  08:43:51 STEP 1215 H=1215:00 +CPU= 41.482
  08:44:20 STEP 1216 H=1216:00 +CPU= 26.793
  08:44:48 STEP 1217 H=1217:00 +CPU= 26.207
  08:45:33 STEP 1218 H=1218:00 +CPU= 42.092
  08:46:02 STEP 1219 H=1219:00 +CPU= 26.024
  08:46:31 STEP 1220 H=1220:00 +CPU= 27.094
  08:47:16 STEP 1221 H=1221:00 +CPU= 41.255
  08:47:44 STEP 1222 H=1222:00 +CPU= 26.518
  08:48:13 STEP 1223 H=1223:00 +CPU= 25.869
  08:49:02 STEP 1224 H=1224:00 +CPU= 45.486
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_50.zip
Uploading the intermediate file: upload_file_50.zip
  08:49:28 STEP 1225 H=1225:00 +CPU= 24.699
  08:49:56 STEP 1226 H=1226:00 +CPU= 25.204
  08:50:38 STEP 1227 H=1227:00 +CPU= 38.952
  08:51:05 STEP 1228 H=1228:00 +CPU= 25.423
  08:51:32 STEP 1229 H=1229:00 +CPU= 24.581
  08:52:15 STEP 1230 H=1230:00 +CPU= 40.493
  08:52:42 STEP 1231 H=1231:00 +CPU= 24.720
  08:53:09 STEP 1232 H=1232:00 +CPU= 25.329
  08:53:51 STEP 1233 H=1233:00 +CPU= 38.820
  08:54:19 STEP 1234 H=1234:00 +CPU= 25.715
  08:54:45 STEP 1235 H=1235:00 +CPU= 24.593
  08:55:41 STEP 1236 H=1236:00 +CPU= 43.820
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001236
  08:56:20 STEP 1237 H=1237:00 +CPU= 25.670
  08:56:58 STEP 1238 H=1238:00 +CPU= 26.271
  08:57:52 STEP 1239 H=1239:00 +CPU= 40.188
  08:58:25 STEP 1240 H=1240:00 +CPU= 25.991
  08:58:57 STEP 1241 H=1241:00 +CPU= 25.076
  08:59:49 STEP 1242 H=1242:00 +CPU= 40.730
  09:00:20 STEP 1243 H=1243:00 +CPU= 24.768
  09:00:53 STEP 1244 H=1244:00 +CPU= 26.109
  09:01:41 STEP 1245 H=1245:00 +CPU= 39.989
  09:02:13 STEP 1246 H=1246:00 +CPU= 25.801
  09:02:45 STEP 1247 H=1247:00 +CPU= 25.751
  09:03:41 STEP 1248 H=1248:00 +CPU= 46.663
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_51.zip
Uploading the intermediate file: upload_file_51.zip
  09:04:13 STEP 1249 H=1249:00 +CPU= 25.028
  09:04:46 STEP 1250 H=1250:00 +CPU= 25.916
  09:05:36 STEP 1251 H=1251:00 +CPU= 40.345
  09:06:08 STEP 1252 H=1252:00 +CPU= 25.945
  09:06:40 STEP 1253 H=1253:00 +CPU= 25.132
  09:07:30 STEP 1254 H=1254:00 +CPU= 40.777
  09:08:02 STEP 1255 H=1255:00 +CPU= 25.162
  09:08:34 STEP 1256 H=1256:00 +CPU= 26.164
  09:09:24 STEP 1257 H=1257:00 +CPU= 39.690
  09:09:58 STEP 1258 H=1258:00 +CPU= 25.917
  09:10:29 STEP 1259 H=1259:00 +CPU= 25.169
  09:11:23 STEP 1260 H=1260:00 +CPU= 44.599
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001260
  09:11:58 STEP 1261 H=1261:00 +CPU= 25.353
  09:12:31 STEP 1262 H=1262:00 +CPU= 25.619
  09:13:21 STEP 1263 H=1263:00 +CPU= 40.361
  09:13:53 STEP 1264 H=1264:00 +CPU= 26.149
  09:14:25 STEP 1265 H=1265:00 +CPU= 24.900
  09:15:17 STEP 1266 H=1266:00 +CPU= 41.024
  09:15:49 STEP 1267 H=1267:00 +CPU= 25.103
  09:16:24 STEP 1268 H=1268:00 +CPU= 26.321
  09:17:13 STEP 1269 H=1269:00 +CPU= 40.079
  09:17:46 STEP 1270 H=1270:00 +CPU= 25.961
  09:18:24 STEP 1271 H=1271:00 +CPU= 26.243
  09:19:21 STEP 1272 H=1272:00 +CPU= 46.974
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_52.zip
Uploading the intermediate file: upload_file_52.zip
  09:19:50 STEP 1273 H=1273:00 +CPU= 25.482
  09:20:21 STEP 1274 H=1274:00 +CPU= 25.941
  09:21:10 STEP 1275 H=1275:00 +CPU= 40.203
  09:21:41 STEP 1276 H=1276:00 +CPU= 26.268
  09:22:12 STEP 1277 H=1277:00 +CPU= 25.265
  09:23:00 STEP 1278 H=1278:00 +CPU= 41.281
  09:23:31 STEP 1279 H=1279:00 +CPU= 25.377
  09:24:02 STEP 1280 H=1280:00 +CPU= 26.061
  09:24:50 STEP 1281 H=1281:00 +CPU= 40.114
  09:25:20 STEP 1282 H=1282:00 +CPU= 26.018
  09:25:53 STEP 1283 H=1283:00 +CPU= 25.242
  09:26:50 STEP 1284 H=1284:00 +CPU= 45.479
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001284
  09:27:19 STEP 1285 H=1285:00 +CPU= 25.219
  09:27:51 STEP 1286 H=1286:00 +CPU= 25.959
  09:28:39 STEP 1287 H=1287:00 +CPU= 40.564
  09:29:11 STEP 1288 H=1288:00 +CPU= 26.182
  09:29:41 STEP 1289 H=1289:00 +CPU= 25.438
  09:30:32 STEP 1290 H=1290:00 +CPU= 41.412
  09:31:03 STEP 1291 H=1291:00 +CPU= 25.482
  09:31:35 STEP 1292 H=1292:00 +CPU= 26.190
  09:32:25 STEP 1293 H=1293:00 +CPU= 40.608
  09:32:56 STEP 1294 H=1294:00 +CPU= 26.063
  09:33:27 STEP 1295 H=1295:00 +CPU= 26.186
  09:34:22 STEP 1296 H=1296:00 +CPU= 46.537
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_53.zip
Uploading the intermediate file: upload_file_53.zip
  09:34:52 STEP 1297 H=1297:00 +CPU= 25.309
  09:35:23 STEP 1298 H=1298:00 +CPU= 26.160
  09:36:10 STEP 1299 H=1299:00 +CPU= 40.175
  09:36:45 STEP 1300 H=1300:00 +CPU= 26.602
  09:37:21 STEP 1301 H=1301:00 +CPU= 25.818
  09:38:18 STEP 1302 H=1302:00 +CPU= 42.167
  09:38:49 STEP 1303 H=1303:00 +CPU= 25.675
  09:39:21 STEP 1304 H=1304:00 +CPU= 26.357
  09:40:12 STEP 1305 H=1305:00 +CPU= 40.890
  09:40:43 STEP 1306 H=1306:00 +CPU= 26.087
  09:41:13 STEP 1307 H=1307:00 +CPU= 25.241
  09:42:09 STEP 1308 H=1308:00 +CPU= 45.195
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001308
  09:42:40 STEP 1309 H=1309:00 +CPU= 25.682
  09:43:11 STEP 1310 H=1310:00 +CPU= 26.037
  09:44:00 STEP 1311 H=1311:00 +CPU= 40.637
  09:44:32 STEP 1312 H=1312:00 +CPU= 26.213
  09:45:03 STEP 1313 H=1313:00 +CPU= 25.532
  09:45:53 STEP 1314 H=1314:00 +CPU= 41.844
  09:46:24 STEP 1315 H=1315:00 +CPU= 25.749
  09:46:55 STEP 1316 H=1316:00 +CPU= 26.308
  09:47:44 STEP 1317 H=1317:00 +CPU= 40.521
  09:48:17 STEP 1318 H=1318:00 +CPU= 26.669
  09:48:51 STEP 1319 H=1319:00 +CPU= 26.565
  09:49:50 STEP 1320 H=1320:00 +CPU= 47.221
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_54.zip
Uploading the intermediate file: upload_file_54.zip
  09:50:22 STEP 1321 H=1321:00 +CPU= 25.861
  09:50:55 STEP 1322 H=1322:00 +CPU= 26.641
  09:51:45 STEP 1323 H=1323:00 +CPU= 40.954
  09:52:17 STEP 1324 H=1324:00 +CPU= 26.455
  09:52:48 STEP 1325 H=1325:00 +CPU= 25.251
  09:53:38 STEP 1326 H=1326:00 +CPU= 41.743
  09:54:08 STEP 1327 H=1327:00 +CPU= 25.609
  09:54:39 STEP 1328 H=1328:00 +CPU= 26.484
  09:55:29 STEP 1329 H=1329:00 +CPU= 40.976
  09:56:02 STEP 1330 H=1330:00 +CPU= 26.388
  09:56:35 STEP 1331 H=1331:00 +CPU= 25.851
  09:57:29 STEP 1332 H=1332:00 +CPU= 45.325
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001332
  09:57:59 STEP 1333 H=1333:00 +CPU= 25.623
  09:58:31 STEP 1334 H=1334:00 +CPU= 26.607
  09:59:20 STEP 1335 H=1335:00 +CPU= 40.946
  09:59:52 STEP 1336 H=1336:00 +CPU= 26.774
  10:00:23 STEP 1337 H=1337:00 +CPU= 25.769
  10:01:14 STEP 1338 H=1338:00 +CPU= 42.343
  10:01:48 STEP 1339 H=1339:00 +CPU= 25.825
  10:02:20 STEP 1340 H=1340:00 +CPU= 26.604
  10:03:08 STEP 1341 H=1341:00 +CPU= 40.732
  10:03:43 STEP 1342 H=1342:00 +CPU= 26.575
  10:04:17 STEP 1343 H=1343:00 +CPU= 26.777
  10:05:15 STEP 1344 H=1344:00 +CPU= 47.426
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_55.zip
Uploading the intermediate file: upload_file_55.zip
  10:05:46 STEP 1345 H=1345:00 +CPU= 25.521
  10:06:17 STEP 1346 H=1346:00 +CPU= 26.579
  10:07:08 STEP 1347 H=1347:00 +CPU= 41.338
  10:07:39 STEP 1348 H=1348:00 +CPU= 26.236
  10:08:11 STEP 1349 H=1349:00 +CPU= 25.818
  10:09:03 STEP 1350 H=1350:00 +CPU= 42.530
  10:09:36 STEP 1351 H=1351:00 +CPU= 25.918
  10:10:09 STEP 1352 H=1352:00 +CPU= 26.703
  10:10:58 STEP 1353 H=1353:00 +CPU= 41.128
  10:11:30 STEP 1354 H=1354:00 +CPU= 26.616
  10:12:04 STEP 1355 H=1355:00 +CPU= 26.117
  10:12:57 STEP 1356 H=1356:00 +CPU= 45.595
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001356
  10:13:28 STEP 1357 H=1357:00 +CPU= 25.745
  10:14:03 STEP 1358 H=1358:00 +CPU= 27.006
  10:14:56 STEP 1359 H=1359:00 +CPU= 41.024
  10:15:31 STEP 1360 H=1360:00 +CPU= 26.811
  10:16:02 STEP 1361 H=1361:00 +CPU= 25.996
  10:16:51 STEP 1362 H=1362:00 +CPU= 42.311
  10:17:22 STEP 1363 H=1363:00 +CPU= 25.755
  10:17:55 STEP 1364 H=1364:00 +CPU= 26.644
  10:18:44 STEP 1365 H=1365:00 +CPU= 41.084
  10:19:15 STEP 1366 H=1366:00 +CPU= 26.912
  10:19:52 STEP 1367 H=1367:00 +CPU= 27.078
  10:20:52 STEP 1368 H=1368:00 +CPU= 47.598
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_56.zip
Uploading the intermediate file: upload_file_56.zip
Uploading trickle at timestep: 4924800
  10:21:23 STEP 1369 H=1369:00 +CPU= 25.806
  10:21:58 STEP 1370 H=1370:00 +CPU= 26.956
  10:22:55 STEP 1371 H=1371:00 +CPU= 41.853
  10:23:28 STEP 1372 H=1372:00 +CPU= 26.780
  10:24:00 STEP 1373 H=1373:00 +CPU= 25.892
  10:24:52 STEP 1374 H=1374:00 +CPU= 42.515
  10:25:24 STEP 1375 H=1375:00 +CPU= 25.691
  10:25:58 STEP 1376 H=1376:00 +CPU= 26.590
  10:26:47 STEP 1377 H=1377:00 +CPU= 40.671
  10:27:19 STEP 1378 H=1378:00 +CPU= 26.490
  10:27:52 STEP 1379 H=1379:00 +CPU= 26.033
  10:28:46 STEP 1380 H=1380:00 +CPU= 45.497
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001380
  10:29:17 STEP 1381 H=1381:00 +CPU= 25.887
  10:29:51 STEP 1382 H=1382:00 +CPU= 26.912
  10:30:49 STEP 1383 H=1383:00 +CPU= 41.596
  10:31:21 STEP 1384 H=1384:00 +CPU= 26.866
  10:31:52 STEP 1385 H=1385:00 +CPU= 26.209
  10:32:42 STEP 1386 H=1386:00 +CPU= 42.399
  10:33:15 STEP 1387 H=1387:00 +CPU= 26.262
  10:33:49 STEP 1388 H=1388:00 +CPU= 27.014
  10:34:43 STEP 1389 H=1389:00 +CPU= 41.584
  10:35:18 STEP 1390 H=1390:00 +CPU= 27.194
  10:35:50 STEP 1391 H=1391:00 +CPU= 26.934
  10:36:49 STEP 1392 H=1392:00 +CPU= 48.577
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_57.zip
Uploading the intermediate file: upload_file_57.zip
  10:37:20 STEP 1393 H=1393:00 +CPU= 26.221
  10:37:53 STEP 1394 H=1394:00 +CPU= 26.924
  10:38:43 STEP 1395 H=1395:00 +CPU= 41.734
  10:39:16 STEP 1396 H=1396:00 +CPU= 27.026
  10:39:47 STEP 1397 H=1397:00 +CPU= 26.236
  10:40:38 STEP 1398 H=1398:00 +CPU= 42.580
  10:41:09 STEP 1399 H=1399:00 +CPU= 26.196
  10:41:42 STEP 1400 H=1400:00 +CPU= 27.178
  10:42:34 STEP 1401 H=1401:00 +CPU= 41.800
  10:43:12 STEP 1402 H=1402:00 +CPU= 27.641
  10:43:51 STEP 1403 H=1403:00 +CPU= 26.783
  10:44:57 STEP 1404 H=1404:00 +CPU= 46.853
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001404
  10:45:34 STEP 1405 H=1405:00 +CPU= 27.143
  10:46:11 STEP 1406 H=1406:00 +CPU= 27.852
  10:47:10 STEP 1407 H=1407:00 +CPU= 42.141
  10:47:46 STEP 1408 H=1408:00 +CPU= 27.659
  10:48:19 STEP 1409 H=1409:00 +CPU= 26.357
  10:49:12 STEP 1410 H=1410:00 +CPU= 42.071
  10:49:43 STEP 1411 H=1411:00 +CPU= 26.039
  10:50:20 STEP 1412 H=1412:00 +CPU= 27.154
  10:51:13 STEP 1413 H=1413:00 +CPU= 41.752
  10:51:48 STEP 1414 H=1414:00 +CPU= 27.043
  10:52:21 STEP 1415 H=1415:00 +CPU= 26.611
  10:53:21 STEP 1416 H=1416:00 +CPU= 47.666
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_58.zip
Uploading the intermediate file: upload_file_58.zip
  10:53:58 STEP 1417 H=1417:00 +CPU= 26.283
  10:54:33 STEP 1418 H=1418:00 +CPU= 27.282
  10:55:22 STEP 1419 H=1419:00 +CPU= 41.216
  10:55:55 STEP 1420 H=1420:00 +CPU= 26.856
  10:56:26 STEP 1421 H=1421:00 +CPU= 25.999
  10:57:17 STEP 1422 H=1422:00 +CPU= 42.231
  10:57:48 STEP 1423 H=1423:00 +CPU= 25.828
  10:58:21 STEP 1424 H=1424:00 +CPU= 26.917
  10:59:11 STEP 1425 H=1425:00 +CPU= 41.708
  10:59:43 STEP 1426 H=1426:00 +CPU= 26.735
  11:00:14 STEP 1427 H=1427:00 +CPU= 25.876
  11:01:09 STEP 1428 H=1428:00 +CPU= 46.058
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001428
  11:01:41 STEP 1429 H=1429:00 +CPU= 26.164
  11:02:13 STEP 1430 H=1430:00 +CPU= 26.979
  11:03:03 STEP 1431 H=1431:00 +CPU= 41.618
  11:03:34 STEP 1432 H=1432:00 +CPU= 26.897
  11:04:07 STEP 1433 H=1433:00 +CPU= 26.003
  11:04:58 STEP 1434 H=1434:00 +CPU= 42.753
  11:05:30 STEP 1435 H=1435:00 +CPU= 25.866
  11:06:02 STEP 1436 H=1436:00 +CPU= 27.175
  11:06:52 STEP 1437 H=1437:00 +CPU= 41.067
  11:07:24 STEP 1438 H=1438:00 +CPU= 26.873
  11:07:56 STEP 1439 H=1439:00 +CPU= 26.983
  11:08:56 STEP 1440 H=1440:00 +CPU= 48.080
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_59.zip
Uploading the intermediate file: upload_file_59.zip
  11:09:33 STEP 1441 H=1441:00 +CPU= 26.471
  11:10:05 STEP 1442 H=1442:00 +CPU= 27.278
  11:10:54 STEP 1443 H=1443:00 +CPU= 41.141
  11:11:26 STEP 1444 H=1444:00 +CPU= 26.824
  11:12:01 STEP 1445 H=1445:00 +CPU= 26.235
  11:12:52 STEP 1446 H=1446:00 +CPU= 42.709
  11:13:23 STEP 1447 H=1447:00 +CPU= 26.423
  11:13:55 STEP 1448 H=1448:00 +CPU= 26.625
  11:14:44 STEP 1449 H=1449:00 +CPU= 41.186
  11:15:21 STEP 1450 H=1450:00 +CPU= 27.148
  11:15:51 STEP 1451 H=1451:00 +CPU= 26.043
  11:16:44 STEP 1452 H=1452:00 +CPU= 45.273
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001452
  11:17:16 STEP 1453 H=1453:00 +CPU= 26.094
  11:17:47 STEP 1454 H=1454:00 +CPU= 26.536
  11:18:35 STEP 1455 H=1455:00 +CPU= 42.224
  11:19:07 STEP 1456 H=1456:00 +CPU= 27.300
  11:19:38 STEP 1457 H=1457:00 +CPU= 26.083
  11:20:26 STEP 1458 H=1458:00 +CPU= 42.049
  11:21:00 STEP 1459 H=1459:00 +CPU= 25.806
  11:21:31 STEP 1460 H=1460:00 +CPU= 27.286
  11:22:21 STEP 1461 H=1461:00 +CPU= 41.101
  11:22:52 STEP 1462 H=1462:00 +CPU= 26.805
  11:23:24 STEP 1463 H=1463:00 +CPU= 26.819
  11:24:25 STEP 1464 H=1464:00 +CPU= 47.947
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_60.zip
Uploading the intermediate file: upload_file_60.zip
  11:24:55 STEP 1465 H=1465:00 +CPU= 26.032
  11:25:27 STEP 1466 H=1466:00 +CPU= 26.872
  11:26:17 STEP 1467 H=1467:00 +CPU= 41.077
  11:26:49 STEP 1468 H=1468:00 +CPU= 26.663
  11:27:20 STEP 1469 H=1469:00 +CPU= 25.999
  11:28:10 STEP 1470 H=1470:00 +CPU= 42.791
  11:28:40 STEP 1471 H=1471:00 +CPU= 25.546
  11:29:11 STEP 1472 H=1472:00 +CPU= 27.202
  11:30:02 STEP 1473 H=1473:00 +CPU= 41.468
  11:30:36 STEP 1474 H=1474:00 +CPU= 27.122
  11:31:11 STEP 1475 H=1475:00 +CPU= 26.174
  11:32:08 STEP 1476 H=1476:00 +CPU= 45.808
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001476
  11:32:39 STEP 1477 H=1477:00 +CPU= 25.854
  11:33:11 STEP 1478 H=1478:00 +CPU= 26.813
  11:34:02 STEP 1479 H=1479:00 +CPU= 40.931
  11:34:33 STEP 1480 H=1480:00 +CPU= 26.471
  11:35:03 STEP 1481 H=1481:00 +CPU= 26.261
  11:36:01 STEP 1482 H=1482:00 +CPU= 42.017
  11:36:40 STEP 1483 H=1483:00 +CPU= 26.309
  11:37:13 STEP 1484 H=1484:00 +CPU= 27.092
  11:38:11 STEP 1485 H=1485:00 +CPU= 41.829
  11:38:50 STEP 1486 H=1486:00 +CPU= 27.450
  11:39:27 STEP 1487 H=1487:00 +CPU= 27.313
  11:40:29 STEP 1488 H=1488:00 +CPU= 47.912
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_61.zip
Uploading the intermediate file: upload_file_61.zip
  11:41:01 STEP 1489 H=1489:00 +CPU= 25.682
  11:41:42 STEP 1490 H=1490:00 +CPU= 27.481
  11:42:36 STEP 1491 H=1491:00 +CPU= 41.150
  11:43:07 STEP 1492 H=1492:00 +CPU= 26.546
  11:43:38 STEP 1493 H=1493:00 +CPU= 25.805
  11:44:28 STEP 1494 H=1494:00 +CPU= 41.864
  11:44:58 STEP 1495 H=1495:00 +CPU= 25.953
  11:45:30 STEP 1496 H=1496:00 +CPU= 26.738
  11:46:19 STEP 1497 H=1497:00 +CPU= 40.791
  11:46:51 STEP 1498 H=1498:00 +CPU= 26.433
  11:47:21 STEP 1499 H=1499:00 +CPU= 26.166
  11:48:17 STEP 1500 H=1500:00 +CPU= 45.583
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001500
  11:48:48 STEP 1501 H=1501:00 +CPU= 25.781
  11:49:20 STEP 1502 H=1502:00 +CPU= 26.848
  11:50:10 STEP 1503 H=1503:00 +CPU= 41.352
  11:50:42 STEP 1504 H=1504:00 +CPU= 26.875
  11:51:12 STEP 1505 H=1505:00 +CPU= 25.971
  11:52:03 STEP 1506 H=1506:00 +CPU= 41.878
  11:52:33 STEP 1507 H=1507:00 +CPU= 25.639
  11:53:05 STEP 1508 H=1508:00 +CPU= 26.641
  11:53:52 STEP 1509 H=1509:00 +CPU= 40.811
  11:54:26 STEP 1510 H=1510:00 +CPU= 26.985
  11:54:57 STEP 1511 H=1511:00 +CPU= 26.692
  11:55:57 STEP 1512 H=1512:00 +CPU= 48.370
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990_62.zip
Uploading the intermediate file: upload_file_62.zip
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1982050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0032
(argv4) batchid: 951
(argv5) wuid: 12167676
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/2
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167676
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_31c3615dab5d729eba03d42bbeb32b75 to: /var/lib/boinc/slots/2/oifs_43r3_ps_0032_1982050100_123_951_12167676.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_0032_1982050100_123_951_12167676.zip
ic_ancil_file: ic_ancil_12167676
ifsdata_file: ifsdata_12167676
climate_data_file: clim_data_12167676
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4e9cb166e954d87982d106778ac7c6a6 to: /var/lib/boinc/slots/2/ic_ancil_12167676.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/2/ic_ancil_12167676.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_7f02ec0281fa1eac315ddd3530915bad to: /var/lib/boinc/slots/2/ifsdata/ifsdata_12167676.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/2/ifsdata/ifsdata_12167676.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/2/159l_2/clim_data_12167676.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/2/159l_2/clim_data_12167676.zip
Checking for progress XML file: /var/lib/boinc/slots/2/progress_file_12167676.xml
Opened progress file ok : /var/lib/boinc/slots/2/progress_file_12167676.xml
-- Model is restarting --
Adjusting last_iter, 1513, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/2/progress_file_12167676.xml
last_cpu_time: 52024
upload_file_number: 63
last_iter: 1511
last_upload: 5443200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0032_1982050100_123_951_12167676_0_r920939990
The child process has been launched with process id: 15780
Executing the command: /var/lib/boinc/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:15780:15780] [20230116:130206:1673899326:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  13:03:16 STEP 1512 H=1512:00 +CPU= 48.357
  13:03:50 STEP 1513 H=1513:00 +CPU= 26.927
  13:04:25 STEP 1514 H=1514:00 +CPU= 27.870
  13:05:18 STEP 1515 H=1515:00 +CPU= 42.377
  13:05:53 STEP 1516 H=1516:00 +CPU= 28.153
  13:06:31 STEP 1517 H=1517:00 +CPU= 27.201
  13:07:44 STEP 1518 H=1518:00 +CPU= 44.662
  13:08:20 STEP 1519 H=1519:00 +CPU= 27.459
  13:08:57 STEP 1520 H=1520:00 +CPU= 28.138
  13:09:53 STEP 1521 H=1521:00 +CPU= 42.597
  13:10:31 STEP 1522 H=1522:00 +CPU= 27.853
  13:11:04 STEP 1523 H=1523:00 +CPU= 27.094
  13:12:06 STEP 1524 H=1524:00 +CPU= 44.778
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001524
  13:12:39 STEP 1525 H=1525:00 +CPU= 25.102
  13:13:14 STEP 1526 H=1526:00 +CPU= 26.211
  13:14:06 STEP 1527 H=1527:00 +CPU= 39.808
  13:14:40 STEP 1528 H=1528:00 +CPU= 26.221
  13:15:13 STEP 1529 H=1529:00 +CPU= 25.149
  13:16:06 STEP 1530 H=1530:00 +CPU= 40.930
  13:16:40 STEP 1531 H=1531:00 +CPU= 25.122
  13:17:15 STEP 1532 H=1532:00 +CPU= 25.935
  13:18:08 STEP 1533 H=1533:00 +CPU= 39.658
  13:18:43 STEP 1534 H=1534:00 +CPU= 25.970
  13:19:19 STEP 1535 H=1535:00 +CPU= 26.134
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:22:40 1493665 22251901 oifs_43r3_ps_0032_1982050100_123_951_12167676_0 4,924,800 47,402 0.0096
09 May 2023 16:22:20 1493665 22251901 oifs_43r3_ps_0032_1982050100_123_951_12167676_0 3,456,000 34,127 0.0099
09 May 2023 16:19:05 1493665 22251901 oifs_43r3_ps_0032_1982050100_123_951_12167676_0 1,209,600 11,823 0.0098


©2024 climateprediction.net