climateprediction.net home page
Task 22298993

Task 22298993

Name oifs_43r3_ps_0987_2005050100_123_974_12191631_2
Workunit 12191631
Created 17 Jan 2023, 13:47:48 UTC
Sent 17 Jan 2023, 23:31:07 UTC
Report deadline 18 Mar 2023, 23:31:07 UTC
Received 20 Jan 2023, 21:24:29 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1532160
Run time 30 min 26 sec
CPU time 19 min 39 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 8.21 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,732.16 MB
Peak swap size 4,978.07 MB
Peak disk usage 1,244.03 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153295:153295] [20230120:125633:1674244593:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153295:153295] [20230120:125633:1674244593:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:57:20 STEP    0 H=   0:00 +CPU= 18.772
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000000
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000000
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000000
  12:57:46 STEP    1 H=   1:00 +CPU= 12.730
  12:57:57 STEP    2 H=   2:00 +CPU= 10.935
  12:58:15 STEP    3 H=   3:00 +CPU= 17.319
  12:58:30 STEP    4 H=   4:00 +CPU= 11.205
  12:58:41 STEP    5 H=   5:00 +CPU= 10.736
  12:59:01 STEP    6 H=   6:00 +CPU= 17.379
  12:59:15 STEP    7 H=   7:00 +CPU= 10.899
  12:59:28 STEP    8 H=   8:00 +CPU= 10.573
  12:59:46 STEP    9 H=   9:00 +CPU= 16.535
  12:59:58 STEP   10 H=  10:00 +CPU= 11.018
  13:00:15 STEP   11 H=  11:00 +CPU= 10.470
  13:00:38 STEP   12 H=  12:00 +CPU= 18.945
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000012
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000012
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000012
  13:00:49 STEP   13 H=  13:00 +CPU= 10.624
  13:01:00 STEP   14 H=  14:00 +CPU= 10.669
  13:01:17 STEP   15 H=  15:00 +CPU= 17.039
  13:01:29 STEP   16 H=  16:00 +CPU= 10.820
  13:01:40 STEP   17 H=  17:00 +CPU= 10.178
  13:02:01 STEP   18 H=  18:00 +CPU= 17.268
  13:02:12 STEP   19 H=  19:00 +CPU= 10.600
  13:02:23 STEP   20 H=  20:00 +CPU= 10.522
  13:02:44 STEP   21 H=  21:00 +CPU= 16.698
  13:02:56 STEP   22 H=  22:00 +CPU= 10.729
  13:03:19 STEP   23 H=  23:00 +CPU= 10.418
  13:03:46 STEP   24 H=  24:00 +CPU= 19.512
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000024
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000024
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191631/ICMGGhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191631/ICMSHhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191631/ICMUAhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191631/ICMGGhq0f+000012
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191631/ICMSHhq0f+000012
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191631/ICMUAhq0f+000012
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0987_2005050100_123_974_12191631_2_r1387658810_0.zip
Uploading the intermediate file: upload_file_0.zip
  13:04:00 STEP   25 H=  25:00 +CPU= 10.568
  13:04:14 STEP   26 H=  26:00 +CPU= 10.625
  13:04:40 STEP   27 H=  27:00 +CPU= 17.125
  13:04:52 STEP   28 H=  28:00 +CPU= 10.686
  13:05:03 STEP   29 H=  29:00 +CPU= 10.409
  13:05:42 STEP   30 H=  30:00 +CPU= 17.667
  13:05:52 STEP   31 H=  31:00 +CPU= 10.681
  13:06:03 STEP   32 H=  32:00 +CPU= 10.719
  13:06:22 STEP   33 H=  33:00 +CPU= 17.038
  13:06:32 STEP   34 H=  34:00 +CPU= 10.380
  13:06:49 STEP   35 H=  35:00 +CPU= 10.234
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0987
(argv4) batchid: 974
(argv5) wuid: 12191631
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191631
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fd35ed1ec2daa90105d717e124892787 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0987_2005050100_123_974_12191631.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0987_2005050100_123_974_12191631.zip
ic_ancil_file: ic_ancil_12191631
ifsdata_file: ifsdata_12191631
climate_data_file: clim_data_12191631
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8df6f9ed95b3acb256b9261ca3b0b2e6 to: /var/lib/boinc-client/slots/6/ic_ancil_12191631.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12191631.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0f0b5cfd4dc987411eeb973ab4b1d894 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12191631.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12191631.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12191631.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12191631.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12191631.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12191631.xml
-- Model is restarting --
Adjusting last_iter, 36, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12191631.xml
last_cpu_time: 471
upload_file_number: 1
last_iter: 23
last_upload: 86400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0987_2005050100_123_974_12191631_2_r1387658810
The child process has been launched with process id: 153321
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153321:153321] [20230120:130925:1674245365:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  13:10:15 STEP   24 H=  24:00 +CPU= 18.771
  13:10:25 STEP   25 H=  25:00 +CPU= 10.539
  13:10:39 STEP   26 H=  26:00 +CPU= 10.748
  13:11:05 STEP   27 H=  27:00 +CPU= 17.107
  13:11:22 STEP   28 H=  28:00 +CPU= 10.832
  13:11:41 STEP   29 H=  29:00 +CPU= 10.199
  13:12:08 STEP   30 H=  30:00 +CPU= 17.746
  13:12:24 STEP   31 H=  31:00 +CPU= 10.528
  13:12:42 STEP   32 H=  32:00 +CPU= 11.114
  13:13:11 STEP   33 H=  33:00 +CPU= 16.510
  13:13:27 STEP   34 H=  34:00 +CPU= 10.955
  13:13:42 STEP   35 H=  35:00 +CPU= 10.580
  13:14:07 STEP   36 H=  36:00 +CPU= 19.221
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000036
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000036
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000036
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0987
(argv4) batchid: 974
(argv5) wuid: 12191631
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191631
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fd35ed1ec2daa90105d717e124892787 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0987_2005050100_123_974_12191631.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0987_2005050100_123_974_12191631.zip
ic_ancil_file: ic_ancil_12191631
ifsdata_file: ifsdata_12191631
climate_data_file: clim_data_12191631
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8df6f9ed95b3acb256b9261ca3b0b2e6 to: /var/lib/boinc-client/slots/6/ic_ancil_12191631.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12191631.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0f0b5cfd4dc987411eeb973ab4b1d894 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12191631.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12191631.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12191631.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12191631.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12191631.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12191631.xml
-- Model is restarting --
Adjusting last_iter, 37, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12191631.xml
last_cpu_time: 649
upload_file_number: 1
last_iter: 23
last_upload: 86400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0987_2005050100_123_974_12191631_2_r1387658810
The child process has been launched with process id: 153348
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153348:153348] [20230120:132013:1674246013:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  13:20:40 STEP   24 H=  24:00 +CPU= 18.654
  13:20:54 STEP   25 H=  25:00 +CPU= 10.257
  13:21:11 STEP   26 H=  26:00 +CPU= 10.814
  13:21:36 STEP   27 H=  27:00 +CPU= 17.129
  13:21:54 STEP   28 H=  28:00 +CPU= 10.854
  13:22:07 STEP   29 H=  29:00 +CPU= 10.538
  13:22:57 STEP   30 H=  30:00 +CPU= 23.037
  13:23:16 STEP   31 H=  31:00 +CPU= 10.485
  13:23:29 STEP   32 H=  32:00 +CPU= 11.009
  13:23:50 STEP   33 H=  33:00 +CPU= 16.926
  13:24:02 STEP   34 H=  34:00 +CPU= 11.003
  13:24:13 STEP   35 H=  35:00 +CPU= 10.627
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0987
(argv4) batchid: 974
(argv5) wuid: 12191631
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191631
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fd35ed1ec2daa90105d717e124892787 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0987_2005050100_123_974_12191631.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0987_2005050100_123_974_12191631.zip
ic_ancil_file: ic_ancil_12191631
ifsdata_file: ifsdata_12191631
climate_data_file: clim_data_12191631
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8df6f9ed95b3acb256b9261ca3b0b2e6 to: /var/lib/boinc-client/slots/6/ic_ancil_12191631.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12191631.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0f0b5cfd4dc987411eeb973ab4b1d894 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12191631.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12191631.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12191631.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12191631.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12191631.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12191631.xml
-- Model is restarting --
Adjusting last_iter, 36, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12191631.xml
last_cpu_time: 817
upload_file_number: 1
last_iter: 23
last_upload: 86400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0987_2005050100_123_974_12191631_2_r1387658810
The child process has been launched with process id: 153371
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153371:153371] [20230120:133006:1674246606:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  13:30:30 STEP   24 H=  24:00 +CPU= 18.461
  13:30:48 STEP   25 H=  25:00 +CPU= 10.814
  13:31:02 STEP   26 H=  26:00 +CPU= 10.438
  13:31:36 STEP   27 H=  27:00 +CPU= 16.685
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0987
(argv4) batchid: 974
(argv5) wuid: 12191631
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191631
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fd35ed1ec2daa90105d717e124892787 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0987_2005050100_123_974_12191631.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0987_2005050100_123_974_12191631.zip
ic_ancil_file: ic_ancil_12191631
ifsdata_file: ifsdata_12191631
climate_data_file: clim_data_12191631
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8df6f9ed95b3acb256b9261ca3b0b2e6 to: /var/lib/boinc-client/slots/6/ic_ancil_12191631.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12191631.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0f0b5cfd4dc987411eeb973ab4b1d894 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12191631.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12191631.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12191631.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12191631.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12191631.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12191631.xml
-- Model is restarting --
Adjusting last_iter, 28, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12191631.xml
last_cpu_time: 878
upload_file_number: 1
last_iter: 23
last_upload: 86400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0987_2005050100_123_974_12191631_2_r1387658810
The child process has been launched with process id: 153388
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153388:153388] [20230120:133628:1674246988:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  13:36:51 STEP   24 H=  24:00 +CPU= 18.004
  13:37:06 STEP   25 H=  25:00 +CPU= 10.703
  13:37:24 STEP   26 H=  26:00 +CPU= 10.806
  13:37:43 STEP   27 H=  27:00 +CPU= 16.616
  13:37:57 STEP   28 H=  28:00 +CPU= 10.647
  13:38:10 STEP   29 H=  29:00 +CPU= 10.598
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0987
(argv4) batchid: 974
(argv5) wuid: 12191631
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191631
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fd35ed1ec2daa90105d717e124892787 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0987_2005050100_123_974_12191631.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0987_2005050100_123_974_12191631.zip
ic_ancil_file: ic_ancil_12191631
ifsdata_file: ifsdata_12191631
climate_data_file: clim_data_12191631
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8df6f9ed95b3acb256b9261ca3b0b2e6 to: /var/lib/boinc-client/slots/6/ic_ancil_12191631.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12191631.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0f0b5cfd4dc987411eeb973ab4b1d894 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12191631.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12191631.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12191631.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12191631.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12191631.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12191631.xml
-- Model is restarting --
Adjusting last_iter, 30, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12191631.xml
last_cpu_time: 973
upload_file_number: 1
last_iter: 23
last_upload: 86400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0987_2005050100_123_974_12191631_2_r1387658810
The child process has been launched with process id: 153402
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153402:153402] [20230120:134356:1674247436:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  13:44:18 STEP   24 H=  24:00 +CPU= 17.815
  13:44:38 STEP   25 H=  25:00 +CPU= 10.507
  13:44:56 STEP   26 H=  26:00 +CPU= 11.139
  13:45:19 STEP   27 H=  27:00 +CPU= 17.386
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0987
(argv4) batchid: 974
(argv5) wuid: 12191631
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191631
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fd35ed1ec2daa90105d717e124892787 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0987_2005050100_123_974_12191631.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0987_2005050100_123_974_12191631.zip
ic_ancil_file: ic_ancil_12191631
ifsdata_file: ifsdata_12191631
climate_data_file: clim_data_12191631
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8df6f9ed95b3acb256b9261ca3b0b2e6 to: /var/lib/boinc-client/slots/6/ic_ancil_12191631.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12191631.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0f0b5cfd4dc987411eeb973ab4b1d894 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12191631.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12191631.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12191631.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12191631.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12191631.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12191631.xml
-- Model is restarting --
Adjusting last_iter, 27, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12191631.xml
last_cpu_time: 1029
upload_file_number: 1
last_iter: 23
last_upload: 86400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0987_2005050100_123_974_12191631_2_r1387658810
The child process has been launched with process id: 153413
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:3900x-2:1:1:153413:153413] [20230120:134703:1674247623:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  13:47:30 STEP   24 H=  24:00 +CPU= 18.472
  13:47:43 STEP   25 H=  25:00 +CPU= 10.691
  13:48:00 STEP   26 H=  26:00 +CPU= 11.240
  13:48:31 STEP   27 H=  27:00 +CPU= 16.708
  13:48:42 STEP   28 H=  28:00 +CPU= 11.041
  13:48:59 STEP   29 H=  29:00 +CPU= 10.778
  13:49:21 STEP   30 H=  30:00 +CPU= 17.462
  13:49:32 STEP   31 H=  31:00 +CPU= 10.268
  13:49:44 STEP   32 H=  32:00 +CPU= 10.697
  13:50:09 STEP   33 H=  33:00 +CPU= 17.537
  13:50:23 STEP   34 H=  34:00 +CPU= 10.664
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net