<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ient/slots/1/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0007_2004050100_123_973_12189651_0_r307396617_62.zip
Uploading the intermediate file: upload_file_62.zip
17:59:30 STEP 1513 H=1513:00 +CPU= 59.109
18:00:25 STEP 1514 H=1514:00 +CPU= 53.916
18:02:42 STEP 1515 H=1515:00 +CPU= 95.202
18:04:23 STEP 1516 H=1516:00 +CPU= 63.150
18:05:27 STEP 1517 H=1517:00 +CPU= 54.482
18:07:25 STEP 1518 H=1518:00 +CPU= 94.388
18:08:57 STEP 1519 H=1519:00 +CPU= 62.132
18:10:21 STEP 1520 H=1520:00 +CPU= 60.616
18:13:04 STEP 1521 H=1521:00 +CPU=100.547
18:14:29 STEP 1522 H=1522:00 +CPU= 61.410
18:15:23 STEP 1523 H=1523:00 +CPU= 52.953
18:17:42 STEP 1524 H=1524:00 +CPU=104.439
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001524
18:19:21 STEP 1525 H=1525:00 +CPU= 62.143
18:20:47 STEP 1526 H=1526:00 +CPU= 60.093
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0007
(argv4) batchid: 973
(argv5) wuid: 12189651
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_562477c554a1128fc143bbcc3b36f885 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0007_2004050100_123_973_12189651.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0007_2004050100_123_973_12189651.zip
ic_ancil_file: ic_ancil_12189651
ifsdata_file: ifsdata_12189651
climate_data_file: clim_data_12189651
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e8188cd663c4618bd8869b26cc676cd4 to: /var/lib/boinc-client/slots/1/ic_ancil_12189651.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12189651.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2bfd2e03aaca2502e5542375a5fe0790 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12189651.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12189651.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12189651.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12189651.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12189651.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12189651.xml
-- Model is restarting --
Adjusting last_iter, 1527, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12189651.xml
last_cpu_time: 98104
upload_file_number: 63
last_iter: 1511
last_upload: 5443200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0007_2004050100_123_973_12189651_0_r307396617
The child process has been launched with process id: 488797
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:488797:488797] [20221226:182244:1672100564:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
18:24:53 STEP 1512 H=1512:00 +CPU=102.426
18:26:12 STEP 1513 H=1513:00 +CPU= 58.468
18:27:17 STEP 1514 H=1514:00 +CPU= 55.249
18:29:07 STEP 1515 H=1515:00 +CPU= 91.787
18:30:36 STEP 1516 H=1516:00 +CPU= 61.750
18:32:11 STEP 1517 H=1517:00 +CPU= 60.517
18:34:14 STEP 1518 H=1518:00 +CPU= 94.673
18:35:54 STEP 1519 H=1519:00 +CPU= 61.494
18:37:15 STEP 1520 H=1520:00 +CPU= 61.320
18:39:05 STEP 1521 H=1521:00 +CPU= 92.426
18:40:36 STEP 1522 H=1522:00 +CPU= 61.371
18:42:15 STEP 1523 H=1523:00 +CPU= 61.223
18:44:33 STEP 1524 H=1524:00 +CPU=103.861
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001524
18:46:15 STEP 1525 H=1525:00 +CPU= 61.437
18:47:43 STEP 1526 H=1526:00 +CPU= 60.871
18:50:01 STEP 1527 H=1527:00 +CPU= 95.294
18:51:44 STEP 1528 H=1528:00 +CPU= 64.089
18:53:14 STEP 1529 H=1529:00 +CPU= 59.604
18:55:16 STEP 1530 H=1530:00 +CPU= 94.918
18:56:25 STEP 1531 H=1531:00 +CPU= 56.643
18:57:31 STEP 1532 H=1532:00 +CPU= 56.888
18:59:22 STEP 1533 H=1533:00 +CPU= 91.355
19:00:57 STEP 1534 H=1534:00 +CPU= 61.262
19:02:46 STEP 1535 H=1535:00 +CPU= 62.201
19:05:20 STEP 1536 H=1536:00 +CPU=108.484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0007_2004050100_123_973_12189651_0_r307396617_63.zip
Uploading the intermediate file: upload_file_63.zip
19:06:58 STEP 1537 H=1537:00 +CPU= 61.667
19:08:19 STEP 1538 H=1538:00 +CPU= 59.522
19:10:34 STEP 1539 H=1539:00 +CPU= 95.378
19:12:20 STEP 1540 H=1540:00 +CPU= 63.698
19:13:47 STEP 1541 H=1541:00 +CPU= 59.561
19:15:43 STEP 1542 H=1542:00 +CPU= 93.963
19:17:04 STEP 1543 H=1543:00 +CPU= 59.482
19:18:29 STEP 1544 H=1544:00 +CPU= 60.669
19:21:07 STEP 1545 H=1545:00 +CPU= 98.889
19:22:23 STEP 1546 H=1546:00 +CPU= 58.835
19:23:50 STEP 1547 H=1547:00 +CPU= 58.373
19:27:00 STEP 1548 H=1548:00 +CPU=111.328
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001548
19:28:29 STEP 1549 H=1549:00 +CPU= 59.842
19:30:05 STEP 1550 H=1550:00 +CPU= 61.750
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0007
(argv4) batchid: 973
(argv5) wuid: 12189651
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_562477c554a1128fc143bbcc3b36f885 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0007_2004050100_123_973_12189651.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0007_2004050100_123_973_12189651.zip
ic_ancil_file: ic_ancil_12189651
ifsdata_file: ifsdata_12189651
climate_data_file: clim_data_12189651
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e8188cd663c4618bd8869b26cc676cd4 to: /var/lib/boinc-client/slots/1/ic_ancil_12189651.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12189651.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2bfd2e03aaca2502e5542375a5fe0790 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12189651.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12189651.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12189651.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12189651.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12189651.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12189651.xml
-- Model is restarting --
Adjusting last_iter, 1551, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12189651.xml
last_cpu_time: 100980
upload_file_number: 64
last_iter: 1535
last_upload: 5529600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0007_2004050100_123_973_12189651_0_r307396617
The child process has been launched with process id: 489307
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:489307:489307] [20221226:193205:1672104725:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
19:34:12 STEP 1536 H=1536:00 +CPU=101.733
19:35:38 STEP 1537 H=1537:00 +CPU= 59.342
19:36:57 STEP 1538 H=1538:00 +CPU= 59.274
19:38:52 STEP 1539 H=1539:00 +CPU= 91.284
19:40:16 STEP 1540 H=1540:00 +CPU= 59.709
19:41:54 STEP 1541 H=1541:00 +CPU= 60.203
19:43:56 STEP 1542 H=1542:00 +CPU= 93.610
19:45:11 STEP 1543 H=1543:00 +CPU= 56.454
19:46:38 STEP 1544 H=1544:00 +CPU= 61.381
19:49:01 STEP 1545 H=1545:00 +CPU= 96.518
19:50:13 STEP 1546 H=1546:00 +CPU= 56.578
19:51:06 STEP 1547 H=1547:00 +CPU= 52.295
19:53:35 STEP 1548 H=1548:00 +CPU=107.404
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001548
19:54:48 STEP 1549 H=1549:00 +CPU= 57.881
19:56:27 STEP 1550 H=1550:00 +CPU= 61.167
19:58:41 STEP 1551 H=1551:00 +CPU= 96.863
20:00:13 STEP 1552 H=1552:00 +CPU= 61.271
20:02:08 STEP 1553 H=1553:00 +CPU= 63.240
20:04:10 STEP 1554 H=1554:00 +CPU= 93.991
20:05:32 STEP 1555 H=1555:00 +CPU= 57.582
20:06:53 STEP 1556 H=1556:00 +CPU= 58.629
20:08:36 STEP 1557 H=1557:00 +CPU= 89.151
20:09:59 STEP 1558 H=1558:00 +CPU= 61.261
20:11:02 STEP 1559 H=1559:00 +CPU= 56.309
20:13:25 STEP 1560 H=1560:00 +CPU=108.621
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0007_2004050100_123_973_12189651_0_r307396617_64.zip
Uploading the intermediate file: upload_file_64.zip
20:15:00 STEP 1561 H=1561:00 +CPU= 61.672
20:16:21 STEP 1562 H=1562:00 +CPU= 60.260
20:18:32 STEP 1563 H=1563:00 +CPU= 94.353
20:19:59 STEP 1564 H=1564:00 +CPU= 60.997
20:21:35 STEP 1565 H=1565:00 +CPU= 60.645
20:24:28 STEP 1566 H=1566:00 +CPU=101.994
20:25:37 STEP 1567 H=1567:00 +CPU= 56.691
20:26:51 STEP 1568 H=1568:00 +CPU= 57.530
20:28:45 STEP 1569 H=1569:00 +CPU= 90.920
20:30:02 STEP 1570 H=1570:00 +CPU= 58.159
20:31:21 STEP 1571 H=1571:00 +CPU= 58.480
20:33:54 STEP 1572 H=1572:00 +CPU=105.951
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001572
20:35:24 STEP 1573 H=1573:00 +CPU= 59.859
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0007
(argv4) batchid: 973
(argv5) wuid: 12189651
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_562477c554a1128fc143bbcc3b36f885 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0007_2004050100_123_973_12189651.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0007_2004050100_123_973_12189651.zip
ic_ancil_file: ic_ancil_12189651
ifsdata_file: ifsdata_12189651
climate_data_file: clim_data_12189651
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e8188cd663c4618bd8869b26cc676cd4 to: /var/lib/boinc-client/slots/1/ic_ancil_12189651.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12189651.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2bfd2e03aaca2502e5542375a5fe0790 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12189651.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12189651.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12189651.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12189651.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12189651.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12189651.xml
-- Model is restarting --
Adjusting last_iter, 1574, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12189651.xml
last_cpu_time: 103743
upload_file_number: 65
last_iter: 1559
last_upload: 5616000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0007_2004050100_123_973_12189651_0_r307396617
The child process has been launched with process id: 489845
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:489845:489845] [20221226:203650:1672108610:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
20:39:24 STEP 1560 H=1560:00 +CPU=108.076
20:41:02 STEP 1561 H=1561:00 +CPU= 61.259
20:42:37 STEP 1562 H=1562:00 +CPU= 61.249
20:44:32 STEP 1563 H=1563:00 +CPU= 92.450
20:45:57 STEP 1564 H=1564:00 +CPU= 60.637
20:47:28 STEP 1565 H=1565:00 +CPU= 60.452
20:49:32 STEP 1566 H=1566:00 +CPU= 95.376
20:51:15 STEP 1567 H=1567:00 +CPU= 63.054
20:52:48 STEP 1568 H=1568:00 +CPU= 62.575
20:54:55 STEP 1569 H=1569:00 +CPU= 94.199
20:56:38 STEP 1570 H=1570:00 +CPU= 62.854
20:57:55 STEP 1571 H=1571:00 +CPU= 57.395
21:00:10 STEP 1572 H=1572:00 +CPU=103.460
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001572
21:01:35 STEP 1573 H=1573:00 +CPU= 58.463
21:03:08 STEP 1574 H=1574:00 +CPU= 60.838
21:05:13 STEP 1575 H=1575:00 +CPU= 93.916
21:06:48 STEP 1576 H=1576:00 +CPU= 60.779
21:07:41 STEP 1577 H=1577:00 +CPU= 52.177
21:09:54 STEP 1578 H=1578:00 +CPU= 96.748
21:10:49 STEP 1579 H=1579:00 +CPU= 53.600
21:12:07 STEP 1580 H=1580:00 +CPU= 59.561
21:14:13 STEP 1581 H=1581:00 +CPU= 95.609
21:15:56 STEP 1582 H=1582:00 +CPU= 61.297
21:16:53 STEP 1583 H=1583:00 +CPU= 54.942
21:19:07 STEP 1584 H=1584:00 +CPU=108.491
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0007_2004050100_123_973_12189651_0_r307396617_65.zip
Uploading the intermediate file: upload_file_65.zip
21:20:34 STEP 1585 H=1585:00 +CPU= 59.311
21:22:05 STEP 1586 H=1586:00 +CPU= 62.776
21:24:22 STEP 1587 H=1587:00 +CPU= 96.186
21:25:59 STEP 1588 H=1588:00 +CPU= 62.357
21:27:27 STEP 1589 H=1589:00 +CPU= 58.253
21:29:31 STEP 1590 H=1590:00 +CPU= 95.761
21:31:03 STEP 1591 H=1591:00 +CPU= 60.966
21:32:39 STEP 1592 H=1592:00 +CPU= 63.272
21:34:47 STEP 1593 H=1593:00 +CPU= 95.251
21:36:18 STEP 1594 H=1594:00 +CPU= 61.198
21:37:54 STEP 1595 H=1595:00 +CPU= 60.848
21:40:11 STEP 1596 H=1596:00 +CPU=102.480
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001596
21:41:43 STEP 1597 H=1597:00 +CPU= 59.669
21:43:18 STEP 1598 H=1598:00 +CPU= 61.224
21:45:34 STEP 1599 H=1599:00 +CPU= 95.614
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0007
(argv4) batchid: 973
(argv5) wuid: 12189651
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_562477c554a1128fc143bbcc3b36f885 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0007_2004050100_123_973_12189651.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0007_2004050100_123_973_12189651.zip
ic_ancil_file: ic_ancil_12189651
ifsdata_file: ifsdata_12189651
climate_data_file: clim_data_12189651
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e8188cd663c4618bd8869b26cc676cd4 to: /var/lib/boinc-client/slots/1/ic_ancil_12189651.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12189651.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2bfd2e03aaca2502e5542375a5fe0790 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12189651.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12189651.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12189651.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12189651.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12189651.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12189651.xml
-- Model is restarting --
Adjusting last_iter, 1600, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12189651.xml
last_cpu_time: 106684
upload_file_number: 66
last_iter: 1583
last_upload: 5702400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0007_2004050100_123_973_12189651_0_r307396617
The child process has been launched with process id: 490491
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:490491:490491] [20221226:214700:1672112820:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
21:48:56 STEP 1584 H=1584:00 +CPU=100.458
21:50:01 STEP 1585 H=1585:00 +CPU= 55.030
21:51:07 STEP 1586 H=1586:00 +CPU= 56.075
21:53:18 STEP 1587 H=1587:00 +CPU= 93.104
21:54:51 STEP 1588 H=1588:00 +CPU= 61.313
21:56:05 STEP 1589 H=1589:00 +CPU= 56.325
21:58:09 STEP 1590 H=1590:00 +CPU= 96.085
21:59:46 STEP 1591 H=1591:00 +CPU= 59.958
22:00:43 STEP 1592 H=1592:00 +CPU= 54.809
22:02:36 STEP 1593 H=1593:00 +CPU= 91.928
22:04:10 STEP 1594 H=1594:00 +CPU= 62.428
22:05:49 STEP 1595 H=1595:00 +CPU= 60.107
22:07:45 STEP 1596 H=1596:00 +CPU= 99.545
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001596
22:09:05 STEP 1597 H=1597:00 +CPU= 58.645
22:10:00 STEP 1598 H=1598:00 +CPU= 54.067
22:12:02 STEP 1599 H=1599:00 +CPU= 94.053
22:13:30 STEP 1600 H=1600:00 +CPU= 60.999
22:14:59 STEP 1601 H=1601:00 +CPU= 61.260
22:17:08 STEP 1602 H=1602:00 +CPU= 96.085
22:18:31 STEP 1603 H=1603:00 +CPU= 58.562
22:20:00 STEP 1604 H=1604:00 +CPU= 61.453
22:22:18 STEP 1605 H=1605:00 +CPU= 96.546
22:23:42 STEP 1606 H=1606:00 +CPU= 60.739
22:25:17 STEP 1607 H=1607:00 +CPU= 62.538
22:28:23 STEP 1608 H=1608:00 +CPU=115.037
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0007_2004050100_123_973_12189651_0_r307396617_66.zip
Uploading the intermediate file: upload_file_66.zip
22:29:50 STEP 1609 H=1609:00 +CPU= 57.270
22:31:14 STEP 1610 H=1610:00 +CPU= 60.120
22:33:30 STEP 1611 H=1611:00 +CPU= 95.231
22:35:09 STEP 1612 H=1612:00 +CPU= 62.892
22:36:46 STEP 1613 H=1613:00 +CPU= 60.374
22:39:09 STEP 1614 H=1614:00 +CPU= 98.821
22:40:26 STEP 1615 H=1615:00 +CPU= 57.241
22:41:25 STEP 1616 H=1616:00 +CPU= 54.285
22:43:38 STEP 1617 H=1617:00 +CPU= 96.164
22:44:53 STEP 1618 H=1618:00 +CPU= 58.057
22:45:57 STEP 1619 H=1619:00 +CPU= 54.523
22:48:19 STEP 1620 H=1620:00 +CPU=105.716
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001620
22:49:40 STEP 1621 H=1621:00 +CPU= 59.899
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0007
(argv4) batchid: 973
(argv5) wuid: 12189651
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189651
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_562477c554a1128fc143bbcc3b36f885 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0007_2004050100_123_973_12189651.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0007_2004050100_123_973_12189651.zip
ic_ancil_file: ic_ancil_12189651
ifsdata_file: ifsdata_12189651
climate_data_file: clim_data_12189651
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e8188cd663c4618bd8869b26cc676cd4 to: /var/lib/boinc-client/slots/1/ic_ancil_12189651.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12189651.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2bfd2e03aaca2502e5542375a5fe0790 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12189651.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12189651.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12189651.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12189651.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12189651.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12189651.xml
-- Model is restarting --
Adjusting last_iter, 1622, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12189651.xml
last_cpu_time: 109472
upload_file_number: 67
last_iter: 1607
last_upload: 5788800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0007_2004050100_123_973_12189651_0_r307396617
The child process has been launched with process id: 491058
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:491058:491058] [20221226:225147:1672116707:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
22:54:01 STEP 1608 H=1608:00 +CPU=101.218
22:55:25 STEP 1609 H=1609:00 +CPU= 59.909
22:56:24 STEP 1610 H=1610:00 +CPU= 55.730
22:58:29 STEP 1611 H=1611:00 +CPU= 93.837
22:59:43 STEP 1612 H=1612:00 +CPU= 58.212
23:01:10 STEP 1613 H=1613:00 +CPU= 59.378
23:03:02 STEP 1614 H=1614:00 +CPU= 93.205
23:04:36 STEP 1615 H=1615:00 +CPU= 61.064
23:06:14 STEP 1616 H=1616:00 +CPU= 62.261
23:09:13 STEP 1617 H=1617:00 +CPU=102.307
23:10:32 STEP 1618 H=1618:00 +CPU= 59.653
23:11:50 STEP 1619 H=1619:00 +CPU= 57.504
23:14:16 STEP 1620 H=1620:00 +CPU=104.601
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001620
23:15:25 STEP 1621 H=1621:00 +CPU= 56.185
23:16:32 STEP 1622 H=1622:00 +CPU= 57.509
23:18:49 STEP 1623 H=1623:00 +CPU= 96.836
23:20:34 STEP 1624 H=1624:00 +CPU= 62.807
23:21:31 STEP 1625 H=1625:00 +CPU= 53.850
23:23:28 STEP 1626 H=1626:00 +CPU= 93.170
23:25:06 STEP 1627 H=1627:00 +CPU= 60.410
23:26:07 STEP 1628 H=1628:00 +CPU= 55.170
23:27:32 STEP 1629 H=1629:00 +CPU= 83.903
23:28:24 STEP 1630 H=1630:00 +CPU= 52.251
23:29:20 STEP 1631 H=1631:00 +CPU= 54.436
Abort request received from BOINC client, ending the child process
</stderr_txt>
]]>
