climateprediction.net home page
Task 22311967

Task 22311967

Name oifs_43r3_bl_a3t1_2016092300_15_991_12211720_0
Workunit 12211720
Created 13 Feb 2023, 21:05:23 UTC
Sent 13 Feb 2023, 21:09:12 UTC
Report deadline 14 Apr 2023, 21:09:12 UTC
Received 16 Feb 2023, 1:06:21 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 5 (0x00000005) Unknown error code
Computer ID 1537658
Run time 10 hours 10 min 53 sec
CPU time 10 hours 10 min 53 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 2.62 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 4,783.64 MB
Peak swap size 5,930.79 MB
Peak disk usage 2,230.85 MB
Stderr 
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 5 (0x5, -251)</message>
<stderr_txt>
.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.198189194057345E-07 0.000000000000000E+00 0.273309426807427E-05
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.884863620141587E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.457431400828819E-09 0.000000000000000E+00 0.154399557857648E-06
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.233683459758845E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =   288 STEPO   0AAA00AAA
 20:57:57 STEP  288 H=  72:00 +CPU= 48.468
 NSTEP =   289 STEPO   0AAA00AAA
 20:58:42 STEP  289 H=  72:15 +CPU= 44.388
ISTEP=     290 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=     290 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =   290 STEPO   0AAA00AAA
 20:59:26 STEP  290 H=  72:30 +CPU= 43.949
 NSTEP =   291 STEPO   0AAA00AAA
 21:00:09 STEP  291 H=  72:45 +CPU= 43.568
 NSTEP =   292 STEPO   0AAA00AAA
 21:00:54 STEP  292 H=  73:00 +CPU= 44.657
 NSTEP =   293 STEPO   0AAA00AAA
 21:01:38 STEP  293 H=  73:15 +CPU= 44.144
 NSTEP =   294 STEPO   0AAA00AAA
 21:02:23 STEP  294 H=  73:30 +CPU= 44.494
ISTEP=     295 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=     295 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =   295 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 20:57:57 0AAA00AAA STEPO      289    48.212   48.212   48.521      0:54      1:01 0.10446467725561E-05       2GB       0MB
 20:58:42 0AAA00AAA STEPO      290    44.388   44.388   44.549      1:39      1:46 0.10446467725561E-05       2GB       0MB
 20:59:26 0AAA00AAA STEPO      291    43.950   43.950   44.086      2:23      2:30 0.10446467725561E-05       2GB       0MB
 21:00:09 0AAA00AAA STEPO      292    43.568   43.568   43.729      3:06      3:14 0.10446467725561E-05       2GB       0MB
 21:00:54 0AAA00AAA STEPO      293    44.657   44.657   44.702      3:51      3:58 0.10446467725561E-05       2GB       0MB
 21:01:38 0AAA00AAA STEPO      294    44.144   44.144   44.185      4:35      4:43 0.10446467725561E-05       2GB       0MB
 21:02:23 0AAA00AAA STEPO      295    44.494   44.494   44.603      5:20      5:27 0.10446467725561E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "     288",
 CTIME   = "00030000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,         276, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/39/progress_file_12211720.xml

<running_values>
  <last_cpu_time>27059.300000</last_cpu_time>
  <upload_file_number>3</upload_file_number>
  <last_iter>294</last_iter>
  <last_upload>259200</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3t1
(argv4) batchid: 991
(argv5) wuid: 12211720
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/39
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/39/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/39/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8a660c23fc0cf7a33fe79a39cb63b992 to: /var/lib/boinc-client/slots/39/oifs_43r3_bl_a3t1_2016092300_15_991_12211720.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/39/oifs_43r3_bl_a3t1_2016092300_15_991_12211720.zip
ic_ancil_file: ic_ancil_12211720
ifsdata_file: ifsdata_12211720
climate_data_file: clim_data_12211720
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/39/ic_ancil_12211720.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/39/ic_ancil_12211720.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c442047aac50823c47ff0af11442c9b4 to: /var/lib/boinc-client/slots/39/ifsdata/ifsdata_12211720.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/39/ifsdata/ifsdata_12211720.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/39/159l_2/clim_data_12211720.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/39/159l_2/clim_data_12211720.zip
Checking for progress XML file: /var/lib/boinc-client/slots/39/progress_file_12211720.xml
Opened progress file ok : /var/lib/boinc-client/slots/39/progress_file_12211720.xml
-- Model is restarting --
Adjusting last_iter, 294, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/39/progress_file_12211720.xml
last_cpu_time: 27059
upload_file_number: 3
last_iter: 289
last_upload: 259200
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a3t1_2016092300_15_991_12211720_0_r2123028782
The child process has been launched with process id: 72779
Executing the command: /var/lib/boinc-client/slots/39/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:72779:72779] [20230215:210351:1676495031:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3t1
(argv4) batchid: 991
(argv5) wuid: 12211720
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/39
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/39/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/39/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8a660c23fc0cf7a33fe79a39cb63b992 to: /var/lib/boinc-client/slots/39/oifs_43r3_bl_a3t1_2016092300_15_991_12211720.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/39/oifs_43r3_bl_a3t1_2016092300_15_991_12211720.zip
ic_ancil_file: ic_ancil_12211720
ifsdata_file: ifsdata_12211720
climate_data_file: clim_data_12211720
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/39/ic_ancil_12211720.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/39/ic_ancil_12211720.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c442047aac50823c47ff0af11442c9b4 to: /var/lib/boinc-client/slots/39/ifsdata/ifsdata_12211720.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/39/ifsdata/ifsdata_12211720.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/39/159l_2/clim_data_12211720.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/39/159l_2/clim_data_12211720.zip
Checking for progress XML file: /var/lib/boinc-client/slots/39/progress_file_12211720.xml
Opened progress file ok : /var/lib/boinc-client/slots/39/progress_file_12211720.xml
-- Model is restarting --
Adjusting last_iter, 288, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/39/progress_file_12211720.xml
last_cpu_time: 27077
upload_file_number: 3
last_iter: 289
last_upload: 259200
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a3t1_2016092300_15_991_12211720_0_r2123028782
The child process has been launched with process id: 72846
Executing the command: /var/lib/boinc-client/slots/39/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:72846:72846] [20230215:210523:1676495123:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:06:54 STEP  288 H=  72:00 +CPU= 53.277
  21:07:38 STEP  289 H=  72:15 +CPU= 44.072
  21:08:22 STEP  290 H=  72:30 +CPU= 43.415
  21:09:05 STEP  291 H=  72:45 +CPU= 42.950
  21:09:58 STEP  292 H=  73:00 +CPU= 47.496
  21:10:44 STEP  293 H=  73:15 +CPU= 45.304
  21:11:31 STEP  294 H=  73:30 +CPU= 47.650
  21:12:20 STEP  295 H=  73:45 +CPU= 45.464
  21:13:04 STEP  296 H=  74:00 +CPU= 44.050
  21:14:00 STEP  297 H=  74:15 +CPU= 47.634
  21:14:46 STEP  298 H=  74:30 +CPU= 45.011
  21:15:31 STEP  299 H=  74:45 +CPU= 45.014
  21:16:28 STEP  300 H=  75:00 +CPU= 54.712
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000300
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000300
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000300
  21:17:13 STEP  301 H=  75:15 +CPU= 44.392
  21:17:56 STEP  302 H=  75:30 +CPU= 42.687
  21:18:46 STEP  303 H=  75:45 +CPU= 45.370
  21:19:31 STEP  304 H=  76:00 +CPU= 44.832
  21:20:18 STEP  305 H=  76:15 +CPU= 45.012
  21:21:09 STEP  306 H=  76:30 +CPU= 44.630
  21:21:55 STEP  307 H=  76:45 +CPU= 45.514
  21:22:41 STEP  308 H=  77:00 +CPU= 46.471
  21:23:28 STEP  309 H=  77:15 +CPU= 45.959
  21:24:15 STEP  310 H=  77:30 +CPU= 46.981
  21:25:06 STEP  311 H=  77:45 +CPU= 47.867
  21:26:06 STEP  312 H=  78:00 +CPU= 58.256
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000312
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000312
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000312
  21:26:55 STEP  313 H=  78:15 +CPU= 47.826
  21:27:41 STEP  314 H=  78:30 +CPU= 45.446
  21:28:29 STEP  315 H=  78:45 +CPU= 46.850
  21:29:17 STEP  316 H=  79:00 +CPU= 47.860
  21:30:01 STEP  317 H=  79:15 +CPU= 43.349
  21:30:45 STEP  318 H=  79:30 +CPU= 43.745
  21:31:32 STEP  319 H=  79:45 +CPU= 46.794
  21:32:22 STEP  320 H=  80:00 +CPU= 49.705
  21:33:09 STEP  321 H=  80:15 +CPU= 46.229
  21:33:56 STEP  322 H=  80:30 +CPU= 47.040
  21:34:46 STEP  323 H=  80:45 +CPU= 48.094
  21:35:45 STEP  324 H=  81:00 +CPU= 58.761
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000324
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000324
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000324
  21:36:35 STEP  325 H=  81:15 +CPU= 45.984
  21:37:17 STEP  326 H=  81:30 +CPU= 42.023
  21:38:11 STEP  327 H=  81:45 +CPU= 43.810
  21:38:55 STEP  328 H=  82:00 +CPU= 43.921
  21:39:40 STEP  329 H=  82:15 +CPU= 44.794
  21:40:32 STEP  330 H=  82:30 +CPU= 45.257
  21:41:15 STEP  331 H=  82:45 +CPU= 42.780
  21:42:01 STEP  332 H=  83:00 +CPU= 45.740
  21:43:05 STEP  333 H=  83:15 +CPU= 50.916
  21:43:54 STEP  334 H=  83:30 +CPU= 47.366
  21:44:42 STEP  335 H=  83:45 +CPU= 47.065
  21:45:58 STEP  336 H=  84:00 +CPU= 67.046
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000336
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000336
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000336
  21:46:54 STEP  337 H=  84:15 +CPU= 46.617
  21:47:39 STEP  338 H=  84:30 +CPU= 44.710
  21:48:32 STEP  339 H=  84:45 +CPU= 51.681
  21:49:20 STEP  340 H=  85:00 +CPU= 47.315
  21:50:05 STEP  341 H=  85:15 +CPU= 44.376
  21:50:57 STEP  342 H=  85:30 +CPU= 48.368
  21:51:50 STEP  343 H=  85:45 +CPU= 47.109
  21:52:36 STEP  344 H=  86:00 +CPU= 45.763
  21:53:21 STEP  345 H=  86:15 +CPU= 44.993
  21:54:09 STEP  346 H=  86:30 +CPU= 47.154
  21:55:00 STEP  347 H=  86:45 +CPU= 47.730
  21:55:59 STEP  348 H=  87:00 +CPU= 56.620
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000348
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000348
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000348
  21:56:52 STEP  349 H=  87:15 +CPU= 48.647
  21:57:46 STEP  350 H=  87:30 +CPU= 47.822
  21:58:35 STEP  351 H=  87:45 +CPU= 47.436
  21:59:26 STEP  352 H=  88:00 +CPU= 49.964
  22:00:15 STEP  353 H=  88:15 +CPU= 47.927
  22:01:02 STEP  354 H=  88:30 +CPU= 46.702
  22:01:50 STEP  355 H=  88:45 +CPU= 47.479
  22:02:38 STEP  356 H=  89:00 +CPU= 48.514
  22:03:29 STEP  357 H=  89:15 +CPU= 49.712
  22:04:19 STEP  358 H=  89:30 +CPU= 47.984
  22:05:13 STEP  359 H=  89:45 +CPU= 53.008
  22:06:19 STEP  360 H=  90:00 +CPU= 64.425
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000360
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000360
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000360
  22:07:11 STEP  361 H=  90:15 +CPU= 51.029
  22:07:59 STEP  362 H=  90:30 +CPU= 47.723
  22:08:48 STEP  363 H=  90:45 +CPU= 48.204
  22:09:37 STEP  364 H=  91:00 +CPU= 48.793
  22:10:25 STEP  365 H=  91:15 +CPU= 47.195
  22:11:17 STEP  366 H=  91:30 +CPU= 49.818
  22:12:08 STEP  367 H=  91:45 +CPU= 49.852
  22:12:56 STEP  368 H=  92:00 +CPU= 47.283
  22:13:45 STEP  369 H=  92:15 +CPU= 49.006
  22:14:35 STEP  370 H=  92:30 +CPU= 46.895
  22:15:19 STEP  371 H=  92:45 +CPU= 44.221
  22:16:23 STEP  372 H=  93:00 +CPU= 55.908
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000372
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000372
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000372
  22:17:13 STEP  373 H=  93:15 +CPU= 46.330
  22:18:16 STEP  374 H=  93:30 +CPU= 44.849
  22:19:01 STEP  375 H=  93:45 +CPU= 43.067
  22:19:51 STEP  376 H=  94:00 +CPU= 44.874
  22:20:37 STEP  377 H=  94:15 +CPU= 45.287
  22:21:34 STEP  378 H=  94:30 +CPU= 46.986
  22:22:19 STEP  379 H=  94:45 +CPU= 44.591
  22:23:09 STEP  380 H=  95:00 +CPU= 49.678
  22:23:57 STEP  381 H=  95:15 +CPU= 45.933
  22:24:47 STEP  382 H=  95:30 +CPU= 47.173
  22:25:37 STEP  383 H=  95:45 +CPU= 48.743
  22:26:54 STEP  384 H=  96:00 +CPU= 66.593
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000384
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000384
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000300
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000300
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000300
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000324
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000324
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000324
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000372
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3t1_2016092300_15_991_12211720_0_r2123028782_3.zip
Uploading the intermediate file: upload_file_3.zip
  22:27:44 STEP  385 H=  96:15 +CPU= 47.207
  22:28:29 STEP  386 H=  96:30 +CPU= 45.216
  22:29:46 STEP  387 H=  96:45 +CPU= 49.728
  22:30:36 STEP  388 H=  97:00 +CPU= 48.277
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3t1
(argv4) batchid: 991
(argv5) wuid: 12211720
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/39
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/39/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/39/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8a660c23fc0cf7a33fe79a39cb63b992 to: /var/lib/boinc-client/slots/39/oifs_43r3_bl_a3t1_2016092300_15_991_12211720.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/39/oifs_43r3_bl_a3t1_2016092300_15_991_12211720.zip
ic_ancil_file: ic_ancil_12211720
ifsdata_file: ifsdata_12211720
climate_data_file: clim_data_12211720
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/39/ic_ancil_12211720.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/39/ic_ancil_12211720.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c442047aac50823c47ff0af11442c9b4 to: /var/lib/boinc-client/slots/39/ifsdata/ifsdata_12211720.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/39/ifsdata/ifsdata_12211720.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/39/159l_2/clim_data_12211720.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/39/159l_2/clim_data_12211720.zip
Checking for progress XML file: /var/lib/boinc-client/slots/39/progress_file_12211720.xml
Opened progress file ok : /var/lib/boinc-client/slots/39/progress_file_12211720.xml
-- Model is restarting --
Adjusting last_iter, 389, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/39/progress_file_12211720.xml
last_cpu_time: 31928
upload_file_number: 4
last_iter: 385
last_upload: 345600
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a3t1_2016092300_15_991_12211720_0_r2123028782
The child process has been launched with process id: 75403
Executing the command: /var/lib/boinc-client/slots/39/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:75403:75403] [20230215:223429:1676500469:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:35:32 STEP  384 H=  96:00 +CPU= 48.317
  22:36:13 STEP  385 H=  96:15 +CPU= 41.136
  22:37:01 STEP  386 H=  96:30 +CPU= 47.389
  22:37:49 STEP  387 H=  96:45 +CPU= 46.468
  22:38:34 STEP  388 H=  97:00 +CPU= 45.532
  22:39:24 STEP  389 H=  97:15 +CPU= 47.016
  22:40:09 STEP  390 H=  97:30 +CPU= 43.969
  22:40:53 STEP  391 H=  97:45 +CPU= 43.990
  22:41:41 STEP  392 H=  98:00 +CPU= 47.558
  22:42:32 STEP  393 H=  98:15 +CPU= 47.051
  22:43:24 STEP  394 H=  98:30 +CPU= 50.726
  22:44:18 STEP  395 H=  98:45 +CPU= 51.821
  22:45:22 STEP  396 H=  99:00 +CPU= 62.065
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000396
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000396
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000396
  22:46:09 STEP  397 H=  99:15 +CPU= 47.246
  22:46:56 STEP  398 H=  99:30 +CPU= 46.096
  22:47:43 STEP  399 H=  99:45 +CPU= 47.592
  22:48:33 STEP  400 H= 100:00 +CPU= 48.349
  22:49:19 STEP  401 H= 100:15 +CPU= 46.110
  22:50:06 STEP  402 H= 100:30 +CPU= 46.067
  22:50:53 STEP  403 H= 100:45 +CPU= 46.727
  22:51:41 STEP  404 H= 101:00 +CPU= 47.419
  22:52:27 STEP  405 H= 101:15 +CPU= 45.435
  22:53:11 STEP  406 H= 101:30 +CPU= 44.174
  22:53:56 STEP  407 H= 101:45 +CPU= 44.614
  22:54:54 STEP  408 H= 102:00 +CPU= 56.617
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000408
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000408
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000408
  22:55:41 STEP  409 H= 102:15 +CPU= 46.948
  22:56:28 STEP  410 H= 102:30 +CPU= 46.196
  22:57:14 STEP  411 H= 102:45 +CPU= 44.785
  22:58:02 STEP  412 H= 103:00 +CPU= 48.086
  22:58:45 STEP  413 H= 103:15 +CPU= 42.313
  22:59:29 STEP  414 H= 103:30 +CPU= 43.973
  23:00:19 STEP  415 H= 103:45 +CPU= 47.917
  23:01:04 STEP  416 H= 104:00 +CPU= 45.529
  23:01:52 STEP  417 H= 104:15 +CPU= 46.022
  23:02:44 STEP  418 H= 104:30 +CPU= 48.554
  23:03:31 STEP  419 H= 104:45 +CPU= 46.073
  23:04:31 STEP  420 H= 105:00 +CPU= 56.532
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000420
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000420
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000420
  23:05:18 STEP  421 H= 105:15 +CPU= 46.011
  23:06:07 STEP  422 H= 105:30 +CPU= 47.954
  23:06:56 STEP  423 H= 105:45 +CPU= 49.405
  23:07:52 STEP  424 H= 106:00 +CPU= 51.392
  23:08:38 STEP  425 H= 106:15 +CPU= 45.317
  23:09:25 STEP  426 H= 106:30 +CPU= 46.160
  23:10:15 STEP  427 H= 106:45 +CPU= 46.066
  23:11:03 STEP  428 H= 107:00 +CPU= 47.007
  23:11:52 STEP  429 H= 107:15 +CPU= 48.087
  23:12:40 STEP  430 H= 107:30 +CPU= 44.132
  23:13:24 STEP  431 H= 107:45 +CPU= 43.709
  23:14:34 STEP  432 H= 108:00 +CPU= 61.558
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000432
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000432
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000432
  23:15:19 STEP  433 H= 108:15 +CPU= 44.632
  23:16:13 STEP  434 H= 108:30 +CPU= 51.466
  23:16:58 STEP  435 H= 108:45 +CPU= 44.886
  23:17:44 STEP  436 H= 109:00 +CPU= 45.553
  23:18:32 STEP  437 H= 109:15 +CPU= 46.249
  23:19:18 STEP  438 H= 109:30 +CPU= 45.225
  23:20:06 STEP  439 H= 109:45 +CPU= 47.674
  23:20:53 STEP  440 H= 110:00 +CPU= 46.505
  23:21:41 STEP  441 H= 110:15 +CPU= 47.358
  23:22:29 STEP  442 H= 110:30 +CPU= 45.287
  23:23:16 STEP  443 H= 110:45 +CPU= 46.758
  23:24:26 STEP  444 H= 111:00 +CPU= 63.118
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000444
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000444
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000444
  23:25:17 STEP  445 H= 111:15 +CPU= 46.227
  23:26:03 STEP  446 H= 111:30 +CPU= 46.130
  23:26:50 STEP  447 H= 111:45 +CPU= 45.496
  23:27:38 STEP  448 H= 112:00 +CPU= 47.437
  23:28:25 STEP  449 H= 112:15 +CPU= 45.955
  23:29:11 STEP  450 H= 112:30 +CPU= 45.573
  23:30:03 STEP  451 H= 112:45 +CPU= 47.004
  23:30:57 STEP  452 H= 113:00 +CPU= 46.632
  23:31:44 STEP  453 H= 113:15 +CPU= 45.942
  23:32:34 STEP  454 H= 113:30 +CPU= 47.270
  23:33:20 STEP  455 H= 113:45 +CPU= 45.407
  23:34:28 STEP  456 H= 114:00 +CPU= 62.512
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000456
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000456
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000456
  23:35:15 STEP  457 H= 114:15 +CPU= 46.305
  23:36:15 STEP  458 H= 114:30 +CPU= 52.158
  23:37:02 STEP  459 H= 114:45 +CPU= 45.255
  23:37:59 STEP  460 H= 115:00 +CPU= 48.709
  23:38:47 STEP  461 H= 115:15 +CPU= 47.587
  23:39:41 STEP  462 H= 115:30 +CPU= 47.557
  23:40:28 STEP  463 H= 115:45 +CPU= 45.504
  23:41:40 STEP  464 H= 116:00 +CPU= 55.167
  23:43:01 STEP  465 H= 116:15 +CPU= 54.453
  23:44:48 STEP  466 H= 116:30 +CPU= 52.416
  23:46:05 STEP  467 H= 116:45 +CPU= 48.199
  23:47:34 STEP  468 H= 117:00 +CPU= 63.039
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000468
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000468
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000468
  23:48:59 STEP  469 H= 117:15 +CPU= 46.339
  23:50:02 STEP  470 H= 117:30 +CPU= 46.313
  23:50:50 STEP  471 H= 117:45 +CPU= 46.981
  23:51:47 STEP  472 H= 118:00 +CPU= 48.720
  23:52:40 STEP  473 H= 118:15 +CPU= 46.966
  23:53:30 STEP  474 H= 118:30 +CPU= 46.417
  23:54:40 STEP  475 H= 118:45 +CPU= 47.196
  23:56:28 STEP  476 H= 119:00 +CPU= 47.844
  23:57:33 STEP  477 H= 119:15 +CPU= 46.523
  23:58:28 STEP  478 H= 119:30 +CPU= 45.311
  23:59:46 STEP  479 H= 119:45 +CPU= 46.980
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3t1
(argv4) batchid: 991
(argv5) wuid: 12211720
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/39
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/39/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/39/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8a660c23fc0cf7a33fe79a39cb63b992 to: /var/lib/boinc-client/slots/39/oifs_43r3_bl_a3t1_2016092300_15_991_12211720.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/39/oifs_43r3_bl_a3t1_2016092300_15_991_12211720.zip
ic_ancil_file: ic_ancil_12211720
ifsdata_file: ifsdata_12211720
climate_data_file: clim_data_12211720
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/39/ic_ancil_12211720.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/39/ic_ancil_12211720.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c442047aac50823c47ff0af11442c9b4 to: /var/lib/boinc-client/slots/39/ifsdata/ifsdata_12211720.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/39/ifsdata/ifsdata_12211720.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/39/159l_2/clim_data_12211720.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/39/159l_2/clim_data_12211720.zip
Checking for progress XML file: /var/lib/boinc-client/slots/39/progress_file_12211720.xml
Opened progress file ok : /var/lib/boinc-client/slots/39/progress_file_12211720.xml
-- Model is restarting --
Adjusting last_iter, 480, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/39/progress_file_12211720.xml
last_cpu_time: 36560
upload_file_number: 4
last_iter: 481
last_upload: 345600
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a3t1_2016092300_15_991_12211720_0_r2123028782
The child process has been launched with process id: 78177
Executing the command: /var/lib/boinc-client/slots/39/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:78177:78177] [20230216:000249:1676505769:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMGGh7zg+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMSHh7zg+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211720/ICMUAh7zg+000468
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3t1_2016092300_15_991_12211720_0_r2123028782_4.zip
Uploading the intermediate file: upload_file_4.zip
  00:04:02 STEP  480 H= 120:00 +CPU= 52.987
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMGGh7zg+000480
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMSHh7zg+000480
Moving to projects directory: /var/lib/boinc-client/slots/39/ICMUAh7zg+000480
free(): invalid pointer
double free or corruption (out)

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net