climateprediction.net home page
Task 22276567

Task 22276567

Name oifs_43r3_ps_0269_2005050100_123_974_12190913_0
Workunit 12190913
Created 23 Dec 2022, 10:08:00 UTC
Sent 11 Jan 2023, 11:55:12 UTC
Report deadline 10 Feb 2023, 11:55:12 UTC
Received 15 Jan 2023, 1:40:57 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1491253
Run time 18 hours 4 min 17 sec
CPU time 18 hours 4 min 17 sec
Validate state Invalid
Credit 9,676.80
Device peak FLOPS 6.20 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,417.55 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,702.91 MB
Stderr 
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
pport/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:91585:91585] [20230114:222756:1673695676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:91585:91585] [20230114:222756:1673695676:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Lian:1:1:91585:91585] [20230114:222756:1673695676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:91585:91585] [20230114:222756:1673695676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:91585:91585] [20230114:222756:1673695676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:91585:91585] [20230114:222756:1673695676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:91585:91585] [20230114:222756:1673695676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:91585:91585] [20230114:222756:1673695676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:91585:91585] [20230114:222756:1673695676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:91585:91585] [20230114:222756:1673695676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:91585:91585] [20230114:222756:1673695676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:91585:91585] [20230114:222756:1673695676:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:29:26 STEP 2256 H=2256:00 +CPU= 55.388
  22:29:59 STEP 2257 H=2257:00 +CPU= 31.381
  22:30:32 STEP 2258 H=2258:00 +CPU= 31.770
  22:31:18 STEP 2259 H=2259:00 +CPU= 44.942
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:32:04 STEP 2260 H=2260:00 +CPU= 32.808
  22:32:38 STEP 2261 H=2261:00 +CPU= 31.780
  22:33:30 STEP 2262 H=2262:00 +CPU= 49.058
  22:34:04 STEP 2263 H=2263:00 +CPU= 31.811
  22:34:40 STEP 2264 H=2264:00 +CPU= 33.308
  22:35:32 STEP 2265 H=2265:00 +CPU= 48.280
  22:36:07 STEP 2266 H=2266:00 +CPU= 32.654
  22:36:39 STEP 2267 H=2267:00 +CPU= 29.788
  22:37:33 STEP 2268 H=2268:00 +CPU= 49.688
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002268
  22:38:06 STEP 2269 H=2269:00 +CPU= 29.597
  22:38:38 STEP 2270 H=2270:00 +CPU= 30.219
  22:39:22 STEP 2271 H=2271:00 +CPU= 40.776
  22:39:48 STEP 2272 H=2272:00 +CPU= 23.968
  22:40:14 STEP 2273 H=2273:00 +CPU= 23.868
  22:40:53 STEP 2274 H=2274:00 +CPU= 36.599
  22:41:18 STEP 2275 H=2275:00 +CPU= 23.730
  22:41:44 STEP 2276 H=2276:00 +CPU= 24.184
  22:42:22 STEP 2277 H=2277:00 +CPU= 35.622
  22:42:48 STEP 2278 H=2278:00 +CPU= 24.336
  22:43:14 STEP 2279 H=2279:00 +CPU= 24.609
  22:44:00 STEP 2280 H=2280:00 +CPU= 41.944
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_94.zip
Uploading the intermediate file: upload_file_94.zip
  22:44:25 STEP 2281 H=2281:00 +CPU= 23.591
  22:44:52 STEP 2282 H=2282:00 +CPU= 24.911
  22:45:32 STEP 2283 H=2283:00 +CPU= 37.402
  22:46:00 STEP 2284 H=2284:00 +CPU= 25.613
  22:46:27 STEP 2285 H=2285:00 +CPU= 24.966
  22:47:08 STEP 2286 H=2286:00 +CPU= 38.298
  22:47:35 STEP 2287 H=2287:00 +CPU= 25.078
  22:48:02 STEP 2288 H=2288:00 +CPU= 25.766
  22:48:41 STEP 2289 H=2289:00 +CPU= 37.238
  22:49:09 STEP 2290 H=2290:00 +CPU= 25.678
  22:49:36 STEP 2291 H=2291:00 +CPU= 25.412
  22:50:19 STEP 2292 H=2292:00 +CPU= 41.084
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002292
  22:50:45 STEP 2293 H=2293:00 +CPU= 24.686
  22:51:10 STEP 2294 H=2294:00 +CPU= 23.898
  22:51:47 STEP 2295 H=2295:00 +CPU= 35.720
  22:52:12 STEP 2296 H=2296:00 +CPU= 24.681
  22:52:38 STEP 2297 H=2297:00 +CPU= 24.899
  22:53:18 STEP 2298 H=2298:00 +CPU= 38.474
  22:53:44 STEP 2299 H=2299:00 +CPU= 24.811
  22:54:10 STEP 2300 H=2300:00 +CPU= 25.205
  22:54:49 STEP 2301 H=2301:00 +CPU= 37.479
  22:55:20 STEP 2302 H=2302:00 +CPU= 28.735
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:56:01 STEP 2303 H=2303:00 +CPU= 28.484
  22:56:55 STEP 2304 H=2304:00 +CPU= 51.204
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_95.zip
Uploading the intermediate file: upload_file_95.zip
  22:57:26 STEP 2305 H=2305:00 +CPU= 29.140
  22:57:58 STEP 2306 H=2306:00 +CPU= 29.966
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:59:30 STEP 2307 H=2307:00 +CPU= 45.993
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:00:14 STEP 2308 H=2308:00 +CPU= 30.408
  23:00:48 STEP 2309 H=2309:00 +CPU= 30.686
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:02:13 STEP 2310 H=2310:00 +CPU= 46.695
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:03:17 STEP 2311 H=2311:00 +CPU= 28.455
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:04:13 STEP 2312 H=2312:00 +CPU= 30.375
  23:05:02 STEP 2313 H=2313:00 +CPU= 44.704
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:05:47 STEP 2314 H=2314:00 +CPU= 31.074
  23:06:19 STEP 2315 H=2315:00 +CPU= 29.530
  23:07:10 STEP 2316 H=2316:00 +CPU= 48.912
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002316
  23:07:43 STEP 2317 H=2317:00 +CPU= 29.861
  23:08:15 STEP 2318 H=2318:00 +CPU= 29.873
  23:09:02 STEP 2319 H=2319:00 +CPU= 43.759
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0269
(argv4) batchid: 974
(argv5) wuid: 12190913
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c92de2d8afafecd16c64bd94df6f29d5 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0269_2005050100_123_974_12190913.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0269_2005050100_123_974_12190913.zip
ic_ancil_file: ic_ancil_12190913
ifsdata_file: ifsdata_12190913
climate_data_file: clim_data_12190913
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3bc68b92bd78065a5e482a446699dde5 to: /var/lib/boinc-client/slots/6/ic_ancil_12190913.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12190913.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e3c4383f5a0bbeff0da476e15570cbd8 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12190913.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12190913.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12190913.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12190913.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12190913.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12190913.xml
-- Model is restarting --
Adjusting last_iter, 2320, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12190913.xml
last_cpu_time: 57865
upload_file_number: 96
last_iter: 2303
last_upload: 8294400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706
The child process has been launched with process id: 96839
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:96839:96839] [20230114:231126:1673698286:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:12:11 STEP 2304 H=2304:00 +CPU= 37.220
  23:12:33 STEP 2305 H=2305:00 +CPU= 21.454
  23:12:56 STEP 2306 H=2306:00 +CPU= 21.387
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:13:43 STEP 2307 H=2307:00 +CPU= 33.849
  23:14:06 STEP 2308 H=2308:00 +CPU= 21.602
  23:14:28 STEP 2309 H=2309:00 +CPU= 21.122
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:15:15 STEP 2310 H=2310:00 +CPU= 33.646
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:15:50 STEP 2311 H=2311:00 +CPU= 21.684
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:16:25 STEP 2312 H=2312:00 +CPU= 22.245
  23:17:00 STEP 2313 H=2313:00 +CPU= 32.799
  23:17:24 STEP 2314 H=2314:00 +CPU= 22.294
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:18:00 STEP 2315 H=2315:00 +CPU= 22.157
  23:18:38 STEP 2316 H=2316:00 +CPU= 36.168
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002316
  23:19:01 STEP 2317 H=2317:00 +CPU= 21.188
  23:19:24 STEP 2318 H=2318:00 +CPU= 22.116
  23:19:58 STEP 2319 H=2319:00 +CPU= 32.198
  23:20:21 STEP 2320 H=2320:00 +CPU= 21.833
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:20:56 STEP 2321 H=2321:00 +CPU= 21.749
  23:21:30 STEP 2322 H=2322:00 +CPU= 32.584
  23:21:55 STEP 2323 H=2323:00 +CPU= 22.482
  23:22:18 STEP 2324 H=2324:00 +CPU= 21.892
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:23:16 STEP 2325 H=2325:00 +CPU= 32.967
  23:23:40 STEP 2326 H=2326:00 +CPU= 21.998
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:24:15 STEP 2327 H=2327:00 +CPU= 22.324
  23:25:00 STEP 2328 H=2328:00 +CPU= 38.477
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_96.zip
Uploading the intermediate file: upload_file_96.zip
  23:25:23 STEP 2329 H=2329:00 +CPU= 21.312
  23:25:50 STEP 2330 H=2330:00 +CPU= 21.798
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:26:41 STEP 2331 H=2331:00 +CPU= 32.522
  23:27:04 STEP 2332 H=2332:00 +CPU= 21.516
  23:27:26 STEP 2333 H=2333:00 +CPU= 21.368
  23:28:01 STEP 2334 H=2334:00 +CPU= 33.021
  23:28:23 STEP 2335 H=2335:00 +CPU= 21.137
  23:28:46 STEP 2336 H=2336:00 +CPU= 21.741
  23:29:20 STEP 2337 H=2337:00 +CPU= 32.440
  23:29:44 STEP 2338 H=2338:00 +CPU= 22.334
  23:30:07 STEP 2339 H=2339:00 +CPU= 21.319
  23:30:48 STEP 2340 H=2340:00 +CPU= 37.199
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002340
  23:31:11 STEP 2341 H=2341:00 +CPU= 21.562
  23:31:33 STEP 2342 H=2342:00 +CPU= 21.900
  23:32:08 STEP 2343 H=2343:00 +CPU= 32.419
  23:32:31 STEP 2344 H=2344:00 +CPU= 21.932
  23:32:53 STEP 2345 H=2345:00 +CPU= 21.244
  23:33:28 STEP 2346 H=2346:00 +CPU= 32.966
  23:33:49 STEP 2347 H=2347:00 +CPU= 20.818
  23:34:13 STEP 2348 H=2348:00 +CPU= 22.235
  23:34:47 STEP 2349 H=2349:00 +CPU= 32.114
  23:35:08 STEP 2350 H=2350:00 +CPU= 20.479
  23:35:29 STEP 2351 H=2351:00 +CPU= 20.764
  23:36:16 STEP 2352 H=2352:00 +CPU= 35.647
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_97.zip
Uploading the intermediate file: upload_file_97.zip
  23:36:36 STEP 2353 H=2353:00 +CPU= 20.355
  23:36:58 STEP 2354 H=2354:00 +CPU= 21.186
  23:37:29 STEP 2355 H=2355:00 +CPU= 30.366
  23:37:50 STEP 2356 H=2356:00 +CPU= 19.924
  23:42:48 STEP 2357 H=2357:00 +CPU= 22.189
  23:43:15 STEP 2358 H=2358:00 +CPU= 25.923
  23:43:31 STEP 2359 H=2359:00 +CPU= 16.415
  23:43:48 STEP 2360 H=2360:00 +CPU= 16.541
  23:44:13 STEP 2361 H=2361:00 +CPU= 24.416
  23:44:29 STEP 2362 H=2362:00 +CPU= 16.361
  23:44:45 STEP 2363 H=2363:00 +CPU= 16.020
  23:45:12 STEP 2364 H=2364:00 +CPU= 27.248
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002364
  23:45:28 STEP 2365 H=2365:00 +CPU= 15.470
  23:45:44 STEP 2366 H=2366:00 +CPU= 16.022
  23:46:09 STEP 2367 H=2367:00 +CPU= 25.439
  23:46:26 STEP 2368 H=2368:00 +CPU= 16.336
  23:46:42 STEP 2369 H=2369:00 +CPU= 15.740
  23:47:08 STEP 2370 H=2370:00 +CPU= 26.333
  23:47:24 STEP 2371 H=2371:00 +CPU= 15.769
  23:47:40 STEP 2372 H=2372:00 +CPU= 16.377
  23:48:04 STEP 2373 H=2373:00 +CPU= 23.931
  23:48:20 STEP 2374 H=2374:00 +CPU= 16.010
  23:48:38 STEP 2375 H=2375:00 +CPU= 17.432
  23:49:08 STEP 2376 H=2376:00 +CPU= 28.656
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_98.zip
Uploading the intermediate file: upload_file_98.zip
  23:49:24 STEP 2377 H=2377:00 +CPU= 16.312
  23:49:41 STEP 2378 H=2378:00 +CPU= 16.462
  23:50:05 STEP 2379 H=2379:00 +CPU= 24.252
  23:50:22 STEP 2380 H=2380:00 +CPU= 16.587
  23:50:38 STEP 2381 H=2381:00 +CPU= 16.367
  23:51:03 STEP 2382 H=2382:00 +CPU= 24.767
  23:51:20 STEP 2383 H=2383:00 +CPU= 16.834
  23:51:36 STEP 2384 H=2384:00 +CPU= 16.157
  23:52:01 STEP 2385 H=2385:00 +CPU= 24.734
  23:52:17 STEP 2386 H=2386:00 +CPU= 16.096
  23:52:33 STEP 2387 H=2387:00 +CPU= 16.462
  23:53:00 STEP 2388 H=2388:00 +CPU= 26.624
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002388
  23:53:17 STEP 2389 H=2389:00 +CPU= 16.900
  23:53:33 STEP 2390 H=2390:00 +CPU= 16.254
  23:53:59 STEP 2391 H=2391:00 +CPU= 25.405
  23:54:15 STEP 2392 H=2392:00 +CPU= 16.076
  23:54:31 STEP 2393 H=2393:00 +CPU= 15.933
  23:54:57 STEP 2394 H=2394:00 +CPU= 25.752
  23:55:13 STEP 2395 H=2395:00 +CPU= 15.717
  23:55:29 STEP 2396 H=2396:00 +CPU= 16.225
  23:55:54 STEP 2397 H=2397:00 +CPU= 25.130
  23:56:11 STEP 2398 H=2398:00 +CPU= 16.535
  23:56:27 STEP 2399 H=2399:00 +CPU= 16.372
  23:56:57 STEP 2400 H=2400:00 +CPU= 29.465
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_99.zip
Uploading the intermediate file: upload_file_99.zip
  23:57:13 STEP 2401 H=2401:00 +CPU= 15.915
  23:57:29 STEP 2402 H=2402:00 +CPU= 15.972
  23:57:53 STEP 2403 H=2403:00 +CPU= 24.043
  23:58:09 STEP 2404 H=2404:00 +CPU= 16.109
  23:58:25 STEP 2405 H=2405:00 +CPU= 16.275
  23:58:50 STEP 2406 H=2406:00 +CPU= 24.502
  23:59:06 STEP 2407 H=2407:00 +CPU= 15.976
  23:59:22 STEP 2408 H=2408:00 +CPU= 15.893
  23:59:46 STEP 2409 H=2409:00 +CPU= 24.611
  00:00:03 STEP 2410 H=2410:00 +CPU= 16.893
  00:00:21 STEP 2411 H=2411:00 +CPU= 17.060
  00:00:50 STEP 2412 H=2412:00 +CPU= 29.480
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002412
  00:01:07 STEP 2413 H=2413:00 +CPU= 16.238
  00:01:23 STEP 2414 H=2414:00 +CPU= 16.537
  00:01:49 STEP 2415 H=2415:00 +CPU= 25.280
  00:02:07 STEP 2416 H=2416:00 +CPU= 17.901
  00:02:23 STEP 2417 H=2417:00 +CPU= 16.605
  00:02:51 STEP 2418 H=2418:00 +CPU= 27.254
  00:03:08 STEP 2419 H=2419:00 +CPU= 16.767
  00:03:25 STEP 2420 H=2420:00 +CPU= 17.334
  00:03:50 STEP 2421 H=2421:00 +CPU= 24.882
  00:04:07 STEP 2422 H=2422:00 +CPU= 17.197
  00:04:24 STEP 2423 H=2423:00 +CPU= 17.131
  00:04:55 STEP 2424 H=2424:00 +CPU= 30.616
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_100.zip
Uploading the intermediate file: upload_file_100.zip
  00:05:12 STEP 2425 H=2425:00 +CPU= 16.516
  00:05:29 STEP 2426 H=2426:00 +CPU= 16.611
  00:05:53 STEP 2427 H=2427:00 +CPU= 24.366
  00:06:09 STEP 2428 H=2428:00 +CPU= 16.279
  00:06:26 STEP 2429 H=2429:00 +CPU= 16.590
  00:06:51 STEP 2430 H=2430:00 +CPU= 24.464
  00:07:07 STEP 2431 H=2431:00 +CPU= 16.215
  00:07:23 STEP 2432 H=2432:00 +CPU= 16.220
  00:07:49 STEP 2433 H=2433:00 +CPU= 25.919
  00:08:06 STEP 2434 H=2434:00 +CPU= 16.669
  00:08:23 STEP 2435 H=2435:00 +CPU= 16.981
  00:08:52 STEP 2436 H=2436:00 +CPU= 28.826
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002436
  00:09:10 STEP 2437 H=2437:00 +CPU= 18.050
  00:09:26 STEP 2438 H=2438:00 +CPU= 16.577
  00:09:51 STEP 2439 H=2439:00 +CPU= 24.301
  00:10:07 STEP 2440 H=2440:00 +CPU= 15.935
  00:10:23 STEP 2441 H=2441:00 +CPU= 16.317
  00:10:48 STEP 2442 H=2442:00 +CPU= 25.437
  00:11:04 STEP 2443 H=2443:00 +CPU= 15.480
  00:11:20 STEP 2444 H=2444:00 +CPU= 16.314
  00:11:46 STEP 2445 H=2445:00 +CPU= 25.122
  00:12:02 STEP 2446 H=2446:00 +CPU= 16.104
  00:12:19 STEP 2447 H=2447:00 +CPU= 17.143
  00:12:49 STEP 2448 H=2448:00 +CPU= 29.486
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_101.zip
Uploading the intermediate file: upload_file_101.zip
  00:13:05 STEP 2449 H=2449:00 +CPU= 16.524
  00:13:22 STEP 2450 H=2450:00 +CPU= 16.266
  00:13:46 STEP 2451 H=2451:00 +CPU= 24.597
  00:14:02 STEP 2452 H=2452:00 +CPU= 15.383
  00:14:18 STEP 2453 H=2453:00 +CPU= 15.931
  00:14:42 STEP 2454 H=2454:00 +CPU= 24.491
  00:14:59 STEP 2455 H=2455:00 +CPU= 16.457
  00:15:15 STEP 2456 H=2456:00 +CPU= 16.942
  00:15:39 STEP 2457 H=2457:00 +CPU= 23.692
  00:15:55 STEP 2458 H=2458:00 +CPU= 15.689
  00:16:10 STEP 2459 H=2459:00 +CPU= 15.389
  00:16:39 STEP 2460 H=2460:00 +CPU= 28.083
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002460
  00:16:55 STEP 2461 H=2461:00 +CPU= 16.100
  00:17:12 STEP 2462 H=2462:00 +CPU= 16.914
  00:17:35 STEP 2463 H=2463:00 +CPU= 23.474
  00:17:52 STEP 2464 H=2464:00 +CPU= 16.557
  00:18:08 STEP 2465 H=2465:00 +CPU= 15.810
  00:18:32 STEP 2466 H=2466:00 +CPU= 24.336
  00:18:48 STEP 2467 H=2467:00 +CPU= 15.397
  00:19:04 STEP 2468 H=2468:00 +CPU= 16.663
  00:19:29 STEP 2469 H=2469:00 +CPU= 24.761
  00:19:45 STEP 2470 H=2470:00 +CPU= 15.878
  00:20:03 STEP 2471 H=2471:00 +CPU= 17.699
  00:20:32 STEP 2472 H=2472:00 +CPU= 29.612
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_102.zip
Uploading the intermediate file: upload_file_102.zip
  00:20:49 STEP 2473 H=2473:00 +CPU= 16.508
  00:21:06 STEP 2474 H=2474:00 +CPU= 16.659
  00:21:31 STEP 2475 H=2475:00 +CPU= 24.944
  00:21:46 STEP 2476 H=2476:00 +CPU= 15.703
  00:22:02 STEP 2477 H=2477:00 +CPU= 15.627
  00:22:27 STEP 2478 H=2478:00 +CPU= 24.898
  00:22:44 STEP 2479 H=2479:00 +CPU= 16.797
  00:23:01 STEP 2480 H=2480:00 +CPU= 16.711
  00:23:25 STEP 2481 H=2481:00 +CPU= 24.093
  00:23:41 STEP 2482 H=2482:00 +CPU= 15.955
  00:23:58 STEP 2483 H=2483:00 +CPU= 17.251
  00:24:25 STEP 2484 H=2484:00 +CPU= 26.962
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002484
  00:24:41 STEP 2485 H=2485:00 +CPU= 15.781
  00:24:57 STEP 2486 H=2486:00 +CPU= 15.939
  00:25:21 STEP 2487 H=2487:00 +CPU= 23.806
  00:25:37 STEP 2488 H=2488:00 +CPU= 15.752
  00:25:52 STEP 2489 H=2489:00 +CPU= 15.381
  00:26:17 STEP 2490 H=2490:00 +CPU= 24.490
  00:26:34 STEP 2491 H=2491:00 +CPU= 16.977
  00:26:51 STEP 2492 H=2492:00 +CPU= 17.194
  00:27:15 STEP 2493 H=2493:00 +CPU= 24.474
  00:27:32 STEP 2494 H=2494:00 +CPU= 16.848
  00:27:49 STEP 2495 H=2495:00 +CPU= 16.818
  00:28:19 STEP 2496 H=2496:00 +CPU= 29.728
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_103.zip
Uploading the intermediate file: upload_file_103.zip
  00:28:35 STEP 2497 H=2497:00 +CPU= 15.944
  00:28:52 STEP 2498 H=2498:00 +CPU= 16.936
  00:29:17 STEP 2499 H=2499:00 +CPU= 24.838
  00:29:33 STEP 2500 H=2500:00 +CPU= 16.395
  00:29:49 STEP 2501 H=2501:00 +CPU= 15.978
  00:30:15 STEP 2502 H=2502:00 +CPU= 25.741
  00:30:30 STEP 2503 H=2503:00 +CPU= 15.181
  00:30:46 STEP 2504 H=2504:00 +CPU= 16.104
  00:31:11 STEP 2505 H=2505:00 +CPU= 24.762
  00:31:28 STEP 2506 H=2506:00 +CPU= 16.325
  00:31:43 STEP 2507 H=2507:00 +CPU= 15.754
  00:32:12 STEP 2508 H=2508:00 +CPU= 28.146
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002508
  00:32:28 STEP 2509 H=2509:00 +CPU= 15.978
  00:32:45 STEP 2510 H=2510:00 +CPU= 17.457
  00:33:11 STEP 2511 H=2511:00 +CPU= 25.783
  00:33:28 STEP 2512 H=2512:00 +CPU= 17.340
  00:33:44 STEP 2513 H=2513:00 +CPU= 15.857
  00:34:10 STEP 2514 H=2514:00 +CPU= 25.293
  00:34:26 STEP 2515 H=2515:00 +CPU= 16.136
  00:34:41 STEP 2516 H=2516:00 +CPU= 15.669
  00:35:06 STEP 2517 H=2517:00 +CPU= 24.383
  00:35:22 STEP 2518 H=2518:00 +CPU= 15.773
  00:35:38 STEP 2519 H=2519:00 +CPU= 16.237
  00:36:08 STEP 2520 H=2520:00 +CPU= 29.518
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_104.zip
Uploading the intermediate file: upload_file_104.zip
  00:36:24 STEP 2521 H=2521:00 +CPU= 16.421
  00:36:40 STEP 2522 H=2522:00 +CPU= 16.337
  00:37:07 STEP 2523 H=2523:00 +CPU= 26.126
  00:37:23 STEP 2524 H=2524:00 +CPU= 15.985
  00:37:40 STEP 2525 H=2525:00 +CPU= 16.877
  00:38:05 STEP 2526 H=2526:00 +CPU= 25.170
  00:38:20 STEP 2527 H=2527:00 +CPU= 15.616
  00:38:37 STEP 2528 H=2528:00 +CPU= 16.176
  00:39:01 STEP 2529 H=2529:00 +CPU= 23.998
  00:39:17 STEP 2530 H=2530:00 +CPU= 16.137
  00:39:32 STEP 2531 H=2531:00 +CPU= 15.320
  00:39:59 STEP 2532 H=2532:00 +CPU= 26.654
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002532
  00:40:15 STEP 2533 H=2533:00 +CPU= 15.802
  00:40:31 STEP 2534 H=2534:00 +CPU= 16.191
  00:40:55 STEP 2535 H=2535:00 +CPU= 24.183
  00:41:12 STEP 2536 H=2536:00 +CPU= 16.758
  00:41:28 STEP 2537 H=2537:00 +CPU= 15.648
  00:41:53 STEP 2538 H=2538:00 +CPU= 25.049
  00:42:09 STEP 2539 H=2539:00 +CPU= 16.312
  00:42:26 STEP 2540 H=2540:00 +CPU= 16.778
  00:42:50 STEP 2541 H=2541:00 +CPU= 24.047
  00:43:07 STEP 2542 H=2542:00 +CPU= 16.618
  00:43:24 STEP 2543 H=2543:00 +CPU= 17.327
  00:43:54 STEP 2544 H=2544:00 +CPU= 29.118
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_105.zip
Uploading the intermediate file: upload_file_105.zip
  00:44:10 STEP 2545 H=2545:00 +CPU= 16.642
Uploading trickle at timestep: 9158400
  00:44:26 STEP 2546 H=2546:00 +CPU= 16.240
  00:44:50 STEP 2547 H=2547:00 +CPU= 23.720
  00:45:06 STEP 2548 H=2548:00 +CPU= 16.102
  00:45:22 STEP 2549 H=2549:00 +CPU= 15.845
  00:45:48 STEP 2550 H=2550:00 +CPU= 25.344
  00:46:04 STEP 2551 H=2551:00 +CPU= 15.866
  00:46:19 STEP 2552 H=2552:00 +CPU= 15.862
  00:46:44 STEP 2553 H=2553:00 +CPU= 24.005
  00:47:01 STEP 2554 H=2554:00 +CPU= 17.038
  00:47:17 STEP 2555 H=2555:00 +CPU= 16.342
  00:47:44 STEP 2556 H=2556:00 +CPU= 27.223
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002556
  00:48:01 STEP 2557 H=2557:00 +CPU= 16.211
  00:48:18 STEP 2558 H=2558:00 +CPU= 16.898
  00:48:42 STEP 2559 H=2559:00 +CPU= 24.220
  00:48:58 STEP 2560 H=2560:00 +CPU= 16.250
  00:49:15 STEP 2561 H=2561:00 +CPU= 16.319
  00:49:39 STEP 2562 H=2562:00 +CPU= 23.829
  00:49:54 STEP 2563 H=2563:00 +CPU= 15.550
  00:50:10 STEP 2564 H=2564:00 +CPU= 16.246
  00:50:36 STEP 2565 H=2565:00 +CPU= 25.288
  00:50:52 STEP 2566 H=2566:00 +CPU= 16.142
  00:51:08 STEP 2567 H=2567:00 +CPU= 15.631
  00:51:37 STEP 2568 H=2568:00 +CPU= 29.072
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_106.zip
Uploading the intermediate file: upload_file_106.zip
  00:51:53 STEP 2569 H=2569:00 +CPU= 15.801
  00:52:09 STEP 2570 H=2570:00 +CPU= 15.807
  00:52:33 STEP 2571 H=2571:00 +CPU= 24.764
  00:52:50 STEP 2572 H=2572:00 +CPU= 16.546
  00:53:06 STEP 2573 H=2573:00 +CPU= 15.865
  00:53:30 STEP 2574 H=2574:00 +CPU= 23.725
  00:53:46 STEP 2575 H=2575:00 +CPU= 16.034
  00:54:02 STEP 2576 H=2576:00 +CPU= 15.791
  00:54:25 STEP 2577 H=2577:00 +CPU= 23.413
  00:54:42 STEP 2578 H=2578:00 +CPU= 16.528
  00:54:59 STEP 2579 H=2579:00 +CPU= 16.928
  00:55:26 STEP 2580 H=2580:00 +CPU= 26.742
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002580
  00:55:41 STEP 2581 H=2581:00 +CPU= 15.614
  00:55:57 STEP 2582 H=2582:00 +CPU= 15.971
  00:56:21 STEP 2583 H=2583:00 +CPU= 23.768
  00:56:37 STEP 2584 H=2584:00 +CPU= 16.166
  00:56:53 STEP 2585 H=2585:00 +CPU= 15.408
  00:57:17 STEP 2586 H=2586:00 +CPU= 24.589
  00:57:33 STEP 2587 H=2587:00 +CPU= 15.934
  00:57:49 STEP 2588 H=2588:00 +CPU= 15.972
  00:58:14 STEP 2589 H=2589:00 +CPU= 24.937
  00:58:30 STEP 2590 H=2590:00 +CPU= 15.285
  00:58:46 STEP 2591 H=2591:00 +CPU= 16.571
  00:59:15 STEP 2592 H=2592:00 +CPU= 28.767
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_107.zip
Uploading the intermediate file: upload_file_107.zip
  00:59:31 STEP 2593 H=2593:00 +CPU= 15.431
  00:59:48 STEP 2594 H=2594:00 +CPU= 16.857
  01:00:12 STEP 2595 H=2595:00 +CPU= 24.080
  01:00:29 STEP 2596 H=2596:00 +CPU= 17.520
  01:00:46 STEP 2597 H=2597:00 +CPU= 16.669
  01:01:11 STEP 2598 H=2598:00 +CPU= 25.114
  01:01:29 STEP 2599 H=2599:00 +CPU= 18.087
  01:01:47 STEP 2600 H=2600:00 +CPU= 17.590
  01:02:13 STEP 2601 H=2601:00 +CPU= 26.203
  01:02:30 STEP 2602 H=2602:00 +CPU= 16.708
  01:02:46 STEP 2603 H=2603:00 +CPU= 15.624
  01:03:14 STEP 2604 H=2604:00 +CPU= 27.551
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002604
  01:03:29 STEP 2605 H=2605:00 +CPU= 15.206
  01:03:45 STEP 2606 H=2606:00 +CPU= 15.794
  01:04:10 STEP 2607 H=2607:00 +CPU= 25.050
  01:04:27 STEP 2608 H=2608:00 +CPU= 16.470
  01:04:43 STEP 2609 H=2609:00 +CPU= 16.000
  01:05:06 STEP 2610 H=2610:00 +CPU= 23.030
  01:05:21 STEP 2611 H=2611:00 +CPU= 14.888
  01:05:37 STEP 2612 H=2612:00 +CPU= 15.911
  01:06:01 STEP 2613 H=2613:00 +CPU= 23.386
  01:06:17 STEP 2614 H=2614:00 +CPU= 16.263
  01:06:32 STEP 2615 H=2615:00 +CPU= 15.157
  01:06:59 STEP 2616 H=2616:00 +CPU= 26.137
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_108.zip
Uploading the intermediate file: upload_file_108.zip
  01:07:13 STEP 2617 H=2617:00 +CPU= 14.118
  01:07:28 STEP 2618 H=2618:00 +CPU= 14.392
  01:07:50 STEP 2619 H=2619:00 +CPU= 22.031
  01:08:04 STEP 2620 H=2620:00 +CPU= 14.132
  01:08:18 STEP 2621 H=2621:00 +CPU= 14.279
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0269
(argv4) batchid: 974
(argv5) wuid: 12190913
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c92de2d8afafecd16c64bd94df6f29d5 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0269_2005050100_123_974_12190913.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0269_2005050100_123_974_12190913.zip
ic_ancil_file: ic_ancil_12190913
ifsdata_file: ifsdata_12190913
climate_data_file: clim_data_12190913
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3bc68b92bd78065a5e482a446699dde5 to: /var/lib/boinc-client/slots/6/ic_ancil_12190913.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12190913.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e3c4383f5a0bbeff0da476e15570cbd8 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12190913.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12190913.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12190913.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12190913.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12190913.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12190913.xml
-- Model is restarting --
Adjusting last_iter, 2622, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12190913.xml
last_cpu_time: 64374
upload_file_number: 109
last_iter: 2615
last_upload: 9417600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706
The child process has been launched with process id: 11439
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Lian:1:1:11439:11439] [20230115:112823:1673742503:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:29:00 STEP 2616 H=2616:00 +CPU= 34.198
  11:29:21 STEP 2617 H=2617:00 +CPU= 20.271
  11:29:43 STEP 2618 H=2618:00 +CPU= 21.192
  11:30:16 STEP 2619 H=2619:00 +CPU= 32.052
  11:30:39 STEP 2620 H=2620:00 +CPU= 21.963
  11:30:59 STEP 2621 H=2621:00 +CPU= 19.915
  11:31:31 STEP 2622 H=2622:00 +CPU= 31.308
  11:31:51 STEP 2623 H=2623:00 +CPU= 19.884
  11:32:13 STEP 2624 H=2624:00 +CPU= 20.930
  11:32:44 STEP 2625 H=2625:00 +CPU= 30.667
  11:33:05 STEP 2626 H=2626:00 +CPU= 20.397
  11:33:26 STEP 2627 H=2627:00 +CPU= 20.577
  11:34:01 STEP 2628 H=2628:00 +CPU= 34.364
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002628
  11:34:22 STEP 2629 H=2629:00 +CPU= 20.463
  11:34:44 STEP 2630 H=2630:00 +CPU= 21.505
  11:35:16 STEP 2631 H=2631:00 +CPU= 31.430
  11:35:37 STEP 2632 H=2632:00 +CPU= 20.796
  11:35:58 STEP 2633 H=2633:00 +CPU= 20.068
  11:36:29 STEP 2634 H=2634:00 +CPU= 31.264
  11:36:50 STEP 2635 H=2635:00 +CPU= 19.891
  11:37:11 STEP 2636 H=2636:00 +CPU= 20.746
  11:37:42 STEP 2637 H=2637:00 +CPU= 30.854
  11:38:03 STEP 2638 H=2638:00 +CPU= 20.163
  11:38:23 STEP 2639 H=2639:00 +CPU= 19.333
  11:38:56 STEP 2640 H=2640:00 +CPU= 32.938
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190913/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0269_2005050100_123_974_12190913_0_r2053826706_109.zip
Uploading the intermediate file: upload_file_109.zip
  11:39:14 STEP 2641 H=2641:00 +CPU= 17.685
  11:39:33 STEP 2642 H=2642:00 +CPU= 18.643
  11:40:02 STEP 2643 H=2643:00 +CPU= 28.162
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:18:52 1491253 22276567 oifs_43r3_ps_0269_2005050100_123_974_12190913_0 6,912,000 45,899 0.0066
09 May 2023 16:18:46 1491253 22276567 oifs_43r3_ps_0269_2005050100_123_974_12190913_0 6,048,000 40,775 0.0067
09 May 2023 16:18:42 1491253 22276567 oifs_43r3_ps_0269_2005050100_123_974_12190913_0 5,184,000 35,508 0.0068
09 May 2023 16:18:37 1491253 22276567 oifs_43r3_ps_0269_2005050100_123_974_12190913_0 4,320,000 29,949 0.0069
09 May 2023 16:17:56 1491253 22276567 oifs_43r3_ps_0269_2005050100_123_974_12190913_0 3,456,000 24,293 0.0070
09 May 2023 16:16:10 1491253 22276567 oifs_43r3_ps_0269_2005050100_123_974_12190913_0 864,000 6,271 0.0073


©2024 climateprediction.net