<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
slots/3/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMGGhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMSHhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMGGhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMSHhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMUAhq0f+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0647_2017050100_123_986_12203291_0_r48597888_24.zip
Uploading the intermediate file: upload_file_24.zip
16:36:38 STEP 601 H= 601:00 +CPU= 42.061
16:37:39 STEP 602 H= 602:00 +CPU= 46.088
16:38:51 STEP 603 H= 603:00 +CPU= 62.312
16:39:46 STEP 604 H= 604:00 +CPU= 42.572
16:40:33 STEP 605 H= 605:00 +CPU= 40.537
16:42:00 STEP 606 H= 606:00 +CPU= 67.161
16:42:49 STEP 607 H= 607:00 +CPU= 41.920
16:43:41 STEP 608 H= 608:00 +CPU= 42.875
16:45:00 STEP 609 H= 609:00 +CPU= 64.789
16:45:50 STEP 610 H= 610:00 +CPU= 41.582
16:46:57 STEP 611 H= 611:00 +CPU= 41.893
16:48:32 STEP 612 H= 612:00 +CPU= 73.880
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000612
16:49:20 STEP 613 H= 613:00 +CPU= 40.443
16:50:17 STEP 614 H= 614:00 +CPU= 44.715
16:51:26 STEP 615 H= 615:00 +CPU= 62.335
16:52:25 STEP 616 H= 616:00 +CPU= 45.158
16:53:07 STEP 617 H= 617:00 +CPU= 40.069
16:54:35 STEP 618 H= 618:00 +CPU= 70.032
16:55:21 STEP 619 H= 619:00 +CPU= 40.468
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0647
(argv4) batchid: 986
(argv5) wuid: 12203291
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5b3c6952db6bf3c4402f87d788604a6b to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0647_2017050100_123_986_12203291.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0647_2017050100_123_986_12203291.zip
ic_ancil_file: ic_ancil_12203291
ifsdata_file: ifsdata_12203291
climate_data_file: clim_data_12203291
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6996e0862fda2f9cc3e840f0281fe43d to: /var/lib/boinc-client/slots/3/ic_ancil_12203291.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12203291.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0934f6d48c31fe767dafc9d15b6ac064 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203291.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203291.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203291.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203291.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12203291.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12203291.xml
-- Model is restarting --
Adjusting last_iter, 620, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12203291.xml
last_cpu_time: 28370
upload_file_number: 25
last_iter: 599
last_upload: 2160000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0647_2017050100_123_986_12203291_0_r48597888
The child process has been launched with process id: 29042
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:29042:29042] [20230120:175739:1674266259:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0647
(argv4) batchid: 986
(argv5) wuid: 12203291
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5b3c6952db6bf3c4402f87d788604a6b to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0647_2017050100_123_986_12203291.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0647_2017050100_123_986_12203291.zip
ic_ancil_file: ic_ancil_12203291
ifsdata_file: ifsdata_12203291
climate_data_file: clim_data_12203291
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6996e0862fda2f9cc3e840f0281fe43d to: /var/lib/boinc-client/slots/3/ic_ancil_12203291.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12203291.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0934f6d48c31fe767dafc9d15b6ac064 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203291.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203291.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203291.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203291.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12203291.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12203291.xml
-- Model is restarting --
Adjusting last_iter, 600, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12203291.xml
last_cpu_time: 28392
upload_file_number: 25
last_iter: 599
last_upload: 2160000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0647_2017050100_123_986_12203291_0_r48597888
The child process has been launched with process id: 30008
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:30008:30008] [20230120:185837:1674269917:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0647
(argv4) batchid: 986
(argv5) wuid: 12203291
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5b3c6952db6bf3c4402f87d788604a6b to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0647_2017050100_123_986_12203291.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0647_2017050100_123_986_12203291.zip
ic_ancil_file: ic_ancil_12203291
ifsdata_file: ifsdata_12203291
climate_data_file: clim_data_12203291
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6996e0862fda2f9cc3e840f0281fe43d to: /var/lib/boinc-client/slots/3/ic_ancil_12203291.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12203291.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0934f6d48c31fe767dafc9d15b6ac064 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203291.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203291.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203291.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203291.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12203291.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12203291.xml
-- Model is restarting --
Adjusting last_iter, 600, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12203291.xml
last_cpu_time: 28415
upload_file_number: 25
last_iter: 599
last_upload: 2160000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0647_2017050100_123_986_12203291_0_r48597888
The child process has been launched with process id: 31483
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:31483:31483] [20230120:195913:1674273553:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
20:00:57 STEP 600 H= 600:00 +CPU= 64.542
20:01:54 STEP 601 H= 601:00 +CPU= 40.771
20:02:44 STEP 602 H= 602:00 +CPU= 34.276
20:04:06 STEP 603 H= 603:00 +CPU= 63.875
20:04:47 STEP 604 H= 604:00 +CPU= 39.377
20:05:35 STEP 605 H= 605:00 +CPU= 35.648
20:06:52 STEP 606 H= 606:00 +CPU= 64.988
20:07:44 STEP 607 H= 607:00 +CPU= 38.341
20:08:40 STEP 608 H= 608:00 +CPU= 44.081
20:09:48 STEP 609 H= 609:00 +CPU= 60.395
20:10:50 STEP 610 H= 610:00 +CPU= 43.371
20:11:39 STEP 611 H= 611:00 +CPU= 41.252
20:13:08 STEP 612 H= 612:00 +CPU= 69.942
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000612
20:13:58 STEP 613 H= 613:00 +CPU= 40.905
20:14:52 STEP 614 H= 614:00 +CPU= 43.444
20:16:10 STEP 615 H= 615:00 +CPU= 61.389
20:17:05 STEP 616 H= 616:00 +CPU= 43.408
20:17:48 STEP 617 H= 617:00 +CPU= 37.538
20:19:10 STEP 618 H= 618:00 +CPU= 67.879
20:20:00 STEP 619 H= 619:00 +CPU= 40.481
20:20:54 STEP 620 H= 620:00 +CPU= 43.790
20:22:06 STEP 621 H= 621:00 +CPU= 63.193
20:22:51 STEP 622 H= 622:00 +CPU= 39.606
20:23:45 STEP 623 H= 623:00 +CPU= 40.741
20:25:22 STEP 624 H= 624:00 +CPU= 75.145
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMGGhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMSHhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMGGhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMSHhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMUAhq0f+000612
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0647_2017050100_123_986_12203291_0_r48597888_25.zip
Uploading the intermediate file: upload_file_25.zip
20:26:16 STEP 625 H= 625:00 +CPU= 40.539
20:27:09 STEP 626 H= 626:00 +CPU= 43.690
20:28:21 STEP 627 H= 627:00 +CPU= 60.311
20:29:15 STEP 628 H= 628:00 +CPU= 43.856
20:30:05 STEP 629 H= 629:00 +CPU= 41.187
20:31:30 STEP 630 H= 630:00 +CPU= 66.080
20:32:16 STEP 631 H= 631:00 +CPU= 40.463
20:33:20 STEP 632 H= 632:00 +CPU= 44.108
20:34:37 STEP 633 H= 633:00 +CPU= 63.805
20:35:29 STEP 634 H= 634:00 +CPU= 43.642
20:36:28 STEP 635 H= 635:00 +CPU= 44.001
20:37:47 STEP 636 H= 636:00 +CPU= 70.479
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000636
20:38:47 STEP 637 H= 637:00 +CPU= 43.764
20:39:30 STEP 638 H= 638:00 +CPU= 38.454
20:40:57 STEP 639 H= 639:00 +CPU= 64.882
20:41:47 STEP 640 H= 640:00 +CPU= 42.336
20:42:37 STEP 641 H= 641:00 +CPU= 42.139
20:43:56 STEP 642 H= 642:00 +CPU= 64.030
20:44:42 STEP 643 H= 643:00 +CPU= 39.363
20:45:44 STEP 644 H= 644:00 +CPU= 45.172
20:47:08 STEP 645 H= 645:00 +CPU= 65.821
20:47:58 STEP 646 H= 646:00 +CPU= 42.122
20:48:55 STEP 647 H= 647:00 +CPU= 44.289
20:50:39 STEP 648 H= 648:00 +CPU= 77.316
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0647_2017050100_123_986_12203291_0_r48597888_26.zip
Uploading the intermediate file: upload_file_26.zip
20:51:34 STEP 649 H= 649:00 +CPU= 41.507
20:52:24 STEP 650 H= 650:00 +CPU= 43.034
20:53:43 STEP 651 H= 651:00 +CPU= 63.160
20:54:30 STEP 652 H= 652:00 +CPU= 42.635
20:55:31 STEP 653 H= 653:00 +CPU= 42.837
20:56:52 STEP 654 H= 654:00 +CPU= 66.295
20:57:49 STEP 655 H= 655:00 +CPU= 42.675
20:58:41 STEP 656 H= 656:00 +CPU= 43.609
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0647
(argv4) batchid: 986
(argv5) wuid: 12203291
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5b3c6952db6bf3c4402f87d788604a6b to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0647_2017050100_123_986_12203291.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0647_2017050100_123_986_12203291.zip
ic_ancil_file: ic_ancil_12203291
ifsdata_file: ifsdata_12203291
climate_data_file: clim_data_12203291
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6996e0862fda2f9cc3e840f0281fe43d to: /var/lib/boinc-client/slots/3/ic_ancil_12203291.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12203291.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0934f6d48c31fe767dafc9d15b6ac064 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203291.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203291.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203291.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203291.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12203291.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12203291.xml
-- Model is restarting --
Adjusting last_iter, 657, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12203291.xml
last_cpu_time: 31303
upload_file_number: 27
last_iter: 647
last_upload: 2332800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0647_2017050100_123_986_12203291_0_r48597888
The child process has been launched with process id: 1032
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:1032:1032] [20230120:220033:1674280833:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
22:02:21 STEP 648 H= 648:00 +CPU= 66.040
22:03:19 STEP 649 H= 649:00 +CPU= 41.873
22:04:18 STEP 650 H= 650:00 +CPU= 35.554
22:05:39 STEP 651 H= 651:00 +CPU= 66.272
22:06:17 STEP 652 H= 652:00 +CPU= 37.208
22:07:11 STEP 653 H= 653:00 +CPU= 36.469
22:08:22 STEP 654 H= 654:00 +CPU= 66.011
22:08:59 STEP 655 H= 655:00 +CPU= 34.422
22:09:57 STEP 656 H= 656:00 +CPU= 41.259
22:11:08 STEP 657 H= 657:00 +CPU= 64.072
22:11:48 STEP 658 H= 658:00 +CPU= 33.342
22:12:43 STEP 659 H= 659:00 +CPU= 42.357
22:13:52 STEP 660 H= 660:00 +CPU= 67.196
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000660
22:14:40 STEP 661 H= 661:00 +CPU= 34.280
22:15:30 STEP 662 H= 662:00 +CPU= 42.816
22:16:29 STEP 663 H= 663:00 +CPU= 57.549
22:17:35 STEP 664 H= 664:00 +CPU= 41.555
22:18:22 STEP 665 H= 665:00 +CPU= 42.277
22:19:39 STEP 666 H= 666:00 +CPU= 58.154
22:20:29 STEP 667 H= 667:00 +CPU= 42.451
22:21:12 STEP 668 H= 668:00 +CPU= 42.605
22:22:24 STEP 669 H= 669:00 +CPU= 57.392
22:23:14 STEP 670 H= 670:00 +CPU= 43.433
22:23:56 STEP 671 H= 671:00 +CPU= 41.165
22:25:33 STEP 672 H= 672:00 +CPU= 63.688
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0647_2017050100_123_986_12203291_0_r48597888_27.zip
Uploading the intermediate file: upload_file_27.zip
22:26:22 STEP 673 H= 673:00 +CPU= 42.005
22:26:59 STEP 674 H= 674:00 +CPU= 36.566
22:28:22 STEP 675 H= 675:00 +CPU= 62.859
22:29:09 STEP 676 H= 676:00 +CPU= 42.150
22:29:42 STEP 677 H= 677:00 +CPU= 32.474
22:31:09 STEP 678 H= 678:00 +CPU= 67.202
22:31:51 STEP 679 H= 679:00 +CPU= 39.936
22:32:55 STEP 680 H= 680:00 +CPU= 39.060
22:34:10 STEP 681 H= 681:00 +CPU= 64.234
22:34:59 STEP 682 H= 682:00 +CPU= 41.428
22:35:56 STEP 683 H= 683:00 +CPU= 42.367
22:37:12 STEP 684 H= 684:00 +CPU= 69.336
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000684
22:38:11 STEP 685 H= 685:00 +CPU= 41.036
22:38:56 STEP 686 H= 686:00 +CPU= 40.904
22:40:24 STEP 687 H= 687:00 +CPU= 66.603
22:41:15 STEP 688 H= 688:00 +CPU= 43.143
22:42:09 STEP 689 H= 689:00 +CPU= 43.379
22:43:20 STEP 690 H= 690:00 +CPU= 62.221
22:44:16 STEP 691 H= 691:00 +CPU= 43.024
22:44:58 STEP 692 H= 692:00 +CPU= 39.394
22:46:22 STEP 693 H= 693:00 +CPU= 62.958
22:47:14 STEP 694 H= 694:00 +CPU= 42.099
22:47:58 STEP 695 H= 695:00 +CPU= 40.854
22:49:34 STEP 696 H= 696:00 +CPU= 75.049
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0647_2017050100_123_986_12203291_0_r48597888_28.zip
Uploading the intermediate file: upload_file_28.zip
22:50:27 STEP 697 H= 697:00 +CPU= 40.453
22:51:25 STEP 698 H= 698:00 +CPU= 44.024
22:52:33 STEP 699 H= 699:00 +CPU= 61.129
22:53:32 STEP 700 H= 700:00 +CPU= 44.044
22:54:26 STEP 701 H= 701:00 +CPU= 43.087
22:55:42 STEP 702 H= 702:00 +CPU= 63.466
22:56:38 STEP 703 H= 703:00 +CPU= 44.847
22:57:22 STEP 704 H= 704:00 +CPU= 40.865
22:58:49 STEP 705 H= 705:00 +CPU= 65.971
22:59:35 STEP 706 H= 706:00 +CPU= 41.776
23:00:44 STEP 707 H= 707:00 +CPU= 44.080
23:02:10 STEP 708 H= 708:00 +CPU= 70.377
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000708
23:02:59 STEP 709 H= 709:00 +CPU= 41.756
23:03:57 STEP 710 H= 710:00 +CPU= 46.153
23:05:02 STEP 711 H= 711:00 +CPU= 59.314
23:06:09 STEP 712 H= 712:00 +CPU= 44.817
23:06:52 STEP 713 H= 713:00 +CPU= 39.357
23:08:25 STEP 714 H= 714:00 +CPU= 68.910
23:09:11 STEP 715 H= 715:00 +CPU= 39.671
23:10:01 STEP 716 H= 716:00 +CPU= 42.465
23:11:16 STEP 717 H= 717:00 +CPU= 62.082
23:12:04 STEP 718 H= 718:00 +CPU= 40.516
23:12:59 STEP 719 H= 719:00 +CPU= 42.360
23:14:37 STEP 720 H= 720:00 +CPU= 76.389
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0647
(argv4) batchid: 986
(argv5) wuid: 12203291
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5b3c6952db6bf3c4402f87d788604a6b to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0647_2017050100_123_986_12203291.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0647_2017050100_123_986_12203291.zip
ic_ancil_file: ic_ancil_12203291
ifsdata_file: ifsdata_12203291
climate_data_file: clim_data_12203291
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6996e0862fda2f9cc3e840f0281fe43d to: /var/lib/boinc-client/slots/3/ic_ancil_12203291.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12203291.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0934f6d48c31fe767dafc9d15b6ac064 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203291.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203291.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203291.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203291.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12203291.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12203291.xml
-- Model is restarting --
Adjusting last_iter, 720, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12203291.xml
last_cpu_time: 34890
upload_file_number: 29
last_iter: 719
last_upload: 2505600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0647_2017050100_123_986_12203291_0_r48597888
The child process has been launched with process id: 17470
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-v110-2:1:1:17470:17470] [20230121:120945:1674331785:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
12:12:48 STEP 720 H= 720:00 +CPU= 68.230
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203291/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0647_2017050100_123_986_12203291_0_r48597888_29.zip
Uploading the intermediate file: upload_file_29.zip
12:14:11 STEP 721 H= 721:00 +CPU= 36.936
12:15:03 STEP 722 H= 722:00 +CPU= 42.561
12:16:04 STEP 723 H= 723:00 +CPU= 55.881
12:17:03 STEP 724 H= 724:00 +CPU= 38.590
12:17:46 STEP 725 H= 725:00 +CPU= 40.248
12:18:43 STEP 726 H= 726:00 +CPU= 55.552
12:19:48 STEP 727 H= 727:00 +CPU= 38.490
12:20:35 STEP 728 H= 728:00 +CPU= 40.738
12:21:31 STEP 729 H= 729:00 +CPU= 55.159
12:22:32 STEP 730 H= 730:00 +CPU= 39.871
12:23:18 STEP 731 H= 731:00 +CPU= 41.063
12:24:46 STEP 732 H= 732:00 +CPU= 64.362
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000732
12:25:43 STEP 733 H= 733:00 +CPU= 42.734
12:26:23 STEP 734 H= 734:00 +CPU= 38.958
12:27:32 STEP 735 H= 735:00 +CPU= 53.471
12:28:23 STEP 736 H= 736:00 +CPU= 43.249
12:29:01 STEP 737 H= 737:00 +CPU= 37.234
12:30:17 STEP 738 H= 738:00 +CPU= 56.682
12:31:10 STEP 739 H= 739:00 +CPU= 42.124
12:31:49 STEP 740 H= 740:00 +CPU= 37.864
12:33:15 STEP 741 H= 741:00 +CPU= 62.358
12:34:00 STEP 742 H= 742:00 +CPU= 41.092
12:35:07 STEP 743 H= 743:00 +CPU= 40.645
12:36:39 STEP 744 H= 744:00 +CPU= 73.205
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
