climateprediction.net home page
Task 22294560

Task 22294560

Name oifs_43r3_ps_0460_2005050100_123_974_12191104_1
Workunit 12191104
Created 30 Dec 2022, 12:22:09 UTC
Sent 30 Dec 2022, 13:19:05 UTC
Report deadline 29 Jan 2023, 13:19:05 UTC
Received 5 Feb 2023, 17:02:27 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1506277
Run time 14 hours 45 min 16 sec
CPU time 12 hours 22 min 40 sec
Validate state Invalid
Credit 1,814.40
Device peak FLOPS 4.93 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,402.18 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,676.84 MB
Stderr 
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
853119875_41.zip
Uploading the intermediate file: upload_file_41.zip
  15:01:19 STEP 1009 H=1009:00 +CPU= 19.355
  15:01:39 STEP 1010 H=1010:00 +CPU= 20.025
  15:02:09 STEP 1011 H=1011:00 +CPU= 29.872
  15:02:29 STEP 1012 H=1012:00 +CPU= 19.850
  15:02:48 STEP 1013 H=1013:00 +CPU= 19.321
  15:03:17 STEP 1014 H=1014:00 +CPU= 29.008
  15:03:34 STEP 1015 H=1015:00 +CPU= 17.310
  15:03:52 STEP 1016 H=1016:00 +CPU= 17.806
  15:04:20 STEP 1017 H=1017:00 +CPU= 27.463
  15:04:39 STEP 1018 H=1018:00 +CPU= 18.489
  15:05:00 STEP 1019 H=1019:00 +CPU= 19.769
  15:05:53 STEP 1020 H=1020:00 +CPU= 33.708
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001020
  15:06:12 STEP 1021 H=1021:00 +CPU= 18.996
  15:06:31 STEP 1022 H=1022:00 +CPU= 19.623
  15:07:13 STEP 1023 H=1023:00 +CPU= 30.234
  15:07:34 STEP 1024 H=1024:00 +CPU= 19.818
  15:07:53 STEP 1025 H=1025:00 +CPU= 19.201
  15:08:24 STEP 1026 H=1026:00 +CPU= 30.613
  15:08:43 STEP 1027 H=1027:00 +CPU= 19.429
  15:09:03 STEP 1028 H=1028:00 +CPU= 19.951
  15:09:33 STEP 1029 H=1029:00 +CPU= 29.624
  15:09:53 STEP 1030 H=1030:00 +CPU= 19.913
  15:10:14 STEP 1031 H=1031:00 +CPU= 20.029
  15:11:33 STEP 1032 H=1032:00 +CPU= 37.033
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_42.zip
Uploading the intermediate file: upload_file_42.zip
  15:11:54 STEP 1033 H=1033:00 +CPU= 20.003
  15:12:14 STEP 1034 H=1034:00 +CPU= 20.100
  15:12:44 STEP 1035 H=1035:00 +CPU= 30.209
  15:13:05 STEP 1036 H=1036:00 +CPU= 20.254
  15:13:24 STEP 1037 H=1037:00 +CPU= 19.333
  15:13:56 STEP 1038 H=1038:00 +CPU= 30.790
  15:14:16 STEP 1039 H=1039:00 +CPU= 20.444
  15:14:36 STEP 1040 H=1040:00 +CPU= 19.022
  15:15:04 STEP 1041 H=1041:00 +CPU= 27.853
  15:15:22 STEP 1042 H=1042:00 +CPU= 18.235
  15:15:40 STEP 1043 H=1043:00 +CPU= 17.853
  15:16:20 STEP 1044 H=1044:00 +CPU= 30.700
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001044
  15:16:42 STEP 1045 H=1045:00 +CPU= 20.132
  15:17:58 STEP 1046 H=1046:00 +CPU= 22.543
  15:18:28 STEP 1047 H=1047:00 +CPU= 28.087
  15:18:47 STEP 1048 H=1048:00 +CPU= 18.607
  15:19:06 STEP 1049 H=1049:00 +CPU= 19.080
  15:21:32 STEP 1050 H=1050:00 +CPU= 33.607
  15:22:05 STEP 1051 H=1051:00 +CPU= 21.394
  15:22:25 STEP 1052 H=1052:00 +CPU= 20.047
  15:22:56 STEP 1053 H=1053:00 +CPU= 30.046
  15:23:16 STEP 1054 H=1054:00 +CPU= 20.104
  15:23:38 STEP 1055 H=1055:00 +CPU= 20.315
  15:24:42 STEP 1056 H=1056:00 +CPU= 36.012
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_43.zip
Uploading the intermediate file: upload_file_43.zip
  15:25:02 STEP 1057 H=1057:00 +CPU= 19.543
  15:25:22 STEP 1058 H=1058:00 +CPU= 19.990
  15:25:52 STEP 1059 H=1059:00 +CPU= 29.778
  15:26:12 STEP 1060 H=1060:00 +CPU= 19.830
  15:26:31 STEP 1061 H=1061:00 +CPU= 19.145
  15:27:01 STEP 1062 H=1062:00 +CPU= 30.212
  15:27:21 STEP 1063 H=1063:00 +CPU= 19.326
  15:27:41 STEP 1064 H=1064:00 +CPU= 19.759
  15:28:10 STEP 1065 H=1065:00 +CPU= 29.618
  15:28:31 STEP 1066 H=1066:00 +CPU= 19.336
  15:28:50 STEP 1067 H=1067:00 +CPU= 19.237
  15:29:24 STEP 1068 H=1068:00 +CPU= 33.645
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001068
  15:29:44 STEP 1069 H=1069:00 +CPU= 19.847
  15:30:04 STEP 1070 H=1070:00 +CPU= 19.834
  15:30:35 STEP 1071 H=1071:00 +CPU= 30.252
  15:30:55 STEP 1072 H=1072:00 +CPU= 20.230
  15:31:14 STEP 1073 H=1073:00 +CPU= 19.181
  15:31:45 STEP 1074 H=1074:00 +CPU= 30.727
  15:32:05 STEP 1075 H=1075:00 +CPU= 19.676
  15:32:24 STEP 1076 H=1076:00 +CPU= 19.679
  15:32:54 STEP 1077 H=1077:00 +CPU= 29.812
  15:33:15 STEP 1078 H=1078:00 +CPU= 20.152
  15:33:35 STEP 1079 H=1079:00 +CPU= 19.892
  15:34:16 STEP 1080 H=1080:00 +CPU= 35.807
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_44.zip
Uploading the intermediate file: upload_file_44.zip
  15:34:35 STEP 1081 H=1081:00 +CPU= 19.298
  15:34:55 STEP 1082 H=1082:00 +CPU= 19.839
  15:35:25 STEP 1083 H=1083:00 +CPU= 29.575
  15:35:45 STEP 1084 H=1084:00 +CPU= 19.868
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0460
(argv4) batchid: 974
(argv5) wuid: 12191104
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4cdb98d6d0e2c91b5ae49ecb04d148e4 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0460_2005050100_123_974_12191104.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0460_2005050100_123_974_12191104.zip
ic_ancil_file: ic_ancil_12191104
ifsdata_file: ifsdata_12191104
climate_data_file: clim_data_12191104
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3bc68b92bd78065a5e482a446699dde5 to: /var/lib/boinc-client/slots/8/ic_ancil_12191104.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12191104.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c4334ac8c39a104d1a78db0f0650bf76 to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12191104.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12191104.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12191104.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12191104.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12191104.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12191104.xml
-- Model is restarting --
Adjusting last_iter, 1084, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12191104.xml
last_cpu_time: 38101
upload_file_number: 45
last_iter: 1079
last_upload: 3888000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875
The child process has been launched with process id: 2675
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2675:2675] [20230205:153724:1675607844:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:40:00 STEP 1080 H=1080:00 +CPU= 36.956
  15:40:21 STEP 1081 H=1081:00 +CPU= 19.737
  15:40:40 STEP 1082 H=1082:00 +CPU= 19.877
  15:41:10 STEP 1083 H=1083:00 +CPU= 29.477
  15:41:30 STEP 1084 H=1084:00 +CPU= 19.807
  15:41:50 STEP 1085 H=1085:00 +CPU= 19.521
  15:42:19 STEP 1086 H=1086:00 +CPU= 29.807
  15:42:37 STEP 1087 H=1087:00 +CPU= 17.152
  15:42:54 STEP 1088 H=1088:00 +CPU= 17.706
  15:43:21 STEP 1089 H=1089:00 +CPU= 27.078
  15:43:40 STEP 1090 H=1090:00 +CPU= 18.461
  15:43:57 STEP 1091 H=1091:00 +CPU= 17.445
  15:52:23 STEP 1092 H=1092:00 +CPU= 37.804
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001092
  15:52:43 STEP 1093 H=1093:00 +CPU= 19.126
  15:53:04 STEP 1094 H=1094:00 +CPU= 20.113
  15:53:34 STEP 1095 H=1095:00 +CPU= 30.587
  15:53:54 STEP 1096 H=1096:00 +CPU= 19.877
  15:54:14 STEP 1097 H=1097:00 +CPU= 19.423
  15:54:45 STEP 1098 H=1098:00 +CPU= 30.416
  15:55:03 STEP 1099 H=1099:00 +CPU= 18.861
  15:55:23 STEP 1100 H=1100:00 +CPU= 19.489
  15:55:50 STEP 1101 H=1101:00 +CPU= 26.830
  15:56:08 STEP 1102 H=1102:00 +CPU= 17.671
  15:56:27 STEP 1103 H=1103:00 +CPU= 17.968
  15:57:55 STEP 1104 H=1104:00 +CPU= 36.062
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_45.zip
Uploading the intermediate file: upload_file_45.zip
  15:58:19 STEP 1105 H=1105:00 +CPU= 19.967
  15:58:39 STEP 1106 H=1106:00 +CPU= 20.103
  15:59:10 STEP 1107 H=1107:00 +CPU= 30.088
  15:59:30 STEP 1108 H=1108:00 +CPU= 20.228
  15:59:50 STEP 1109 H=1109:00 +CPU= 19.504
  16:00:21 STEP 1110 H=1110:00 +CPU= 30.672
  16:00:40 STEP 1111 H=1111:00 +CPU= 19.604
  16:01:01 STEP 1112 H=1112:00 +CPU= 20.130
  16:01:31 STEP 1113 H=1113:00 +CPU= 30.279
  16:01:51 STEP 1114 H=1114:00 +CPU= 20.188
  16:02:11 STEP 1115 H=1115:00 +CPU= 19.239
  16:02:45 STEP 1116 H=1116:00 +CPU= 33.424
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001116
  16:03:04 STEP 1117 H=1117:00 +CPU= 19.288
  16:03:24 STEP 1118 H=1118:00 +CPU= 20.001
  16:03:52 STEP 1119 H=1119:00 +CPU= 27.993
  16:04:10 STEP 1120 H=1120:00 +CPU= 17.688
  16:04:27 STEP 1121 H=1121:00 +CPU= 17.189
  16:04:58 STEP 1122 H=1122:00 +CPU= 28.100
  16:05:16 STEP 1123 H=1123:00 +CPU= 18.533
  16:05:37 STEP 1124 H=1124:00 +CPU= 20.237
  16:06:08 STEP 1125 H=1125:00 +CPU= 30.238
  16:06:28 STEP 1126 H=1126:00 +CPU= 20.145
  16:06:49 STEP 1127 H=1127:00 +CPU= 19.829
  16:07:42 STEP 1128 H=1128:00 +CPU= 33.743
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_46.zip
Uploading the intermediate file: upload_file_46.zip
  16:08:00 STEP 1129 H=1129:00 +CPU= 17.225
  16:08:17 STEP 1130 H=1130:00 +CPU= 17.688
  16:08:45 STEP 1131 H=1131:00 +CPU= 27.880
  16:09:05 STEP 1132 H=1132:00 +CPU= 18.051
  16:09:24 STEP 1133 H=1133:00 +CPU= 17.624
  16:10:20 STEP 1134 H=1134:00 +CPU= 31.861
  16:10:40 STEP 1135 H=1135:00 +CPU= 19.049
  16:10:59 STEP 1136 H=1136:00 +CPU= 19.760
  16:11:29 STEP 1137 H=1137:00 +CPU= 28.777
  16:11:46 STEP 1138 H=1138:00 +CPU= 17.712
  16:12:04 STEP 1139 H=1139:00 +CPU= 17.189
  16:12:36 STEP 1140 H=1140:00 +CPU= 30.699
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001140
  16:12:53 STEP 1141 H=1141:00 +CPU= 17.571
  16:13:15 STEP 1142 H=1142:00 +CPU= 18.091
  16:14:24 STEP 1143 H=1143:00 +CPU= 31.689
  16:14:45 STEP 1144 H=1144:00 +CPU= 20.178
  16:15:04 STEP 1145 H=1145:00 +CPU= 19.483
  16:15:35 STEP 1146 H=1146:00 +CPU= 30.244
  16:15:52 STEP 1147 H=1147:00 +CPU= 17.205
  16:16:10 STEP 1148 H=1148:00 +CPU= 17.675
  16:16:37 STEP 1149 H=1149:00 +CPU= 27.161
  16:16:56 STEP 1150 H=1150:00 +CPU= 18.148
  16:17:16 STEP 1151 H=1151:00 +CPU= 19.383
  16:18:34 STEP 1152 H=1152:00 +CPU= 36.251
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_47.zip
Uploading the intermediate file: upload_file_47.zip
  16:18:56 STEP 1153 H=1153:00 +CPU= 19.377
  16:19:16 STEP 1154 H=1154:00 +CPU= 20.174
  16:19:46 STEP 1155 H=1155:00 +CPU= 28.905
  16:20:04 STEP 1156 H=1156:00 +CPU= 17.746
  16:20:21 STEP 1157 H=1157:00 +CPU= 17.211
  16:20:50 STEP 1158 H=1158:00 +CPU= 28.114
  16:21:09 STEP 1159 H=1159:00 +CPU= 17.491
  16:21:28 STEP 1160 H=1160:00 +CPU= 18.110
  16:21:58 STEP 1161 H=1161:00 +CPU= 28.223
  16:22:44 STEP 1162 H=1162:00 +CPU= 22.148
  16:23:04 STEP 1163 H=1163:00 +CPU= 18.684
  16:23:51 STEP 1164 H=1164:00 +CPU= 32.842
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001164
  16:24:09 STEP 1165 H=1165:00 +CPU= 17.243
  16:24:26 STEP 1166 H=1166:00 +CPU= 17.763
  16:24:54 STEP 1167 H=1167:00 +CPU= 27.193
  16:25:13 STEP 1168 H=1168:00 +CPU= 18.443
  16:25:33 STEP 1169 H=1169:00 +CPU= 17.723
  16:26:04 STEP 1170 H=1170:00 +CPU= 28.268
  16:26:31 STEP 1171 H=1171:00 +CPU= 19.819
  16:26:51 STEP 1172 H=1172:00 +CPU= 19.530
  16:27:21 STEP 1173 H=1173:00 +CPU= 29.581
  16:27:41 STEP 1174 H=1174:00 +CPU= 19.769
  16:28:01 STEP 1175 H=1175:00 +CPU= 19.718
  16:29:01 STEP 1176 H=1176:00 +CPU= 36.221
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_48.zip
Uploading the intermediate file: upload_file_48.zip
  16:29:20 STEP 1177 H=1177:00 +CPU= 19.136
  16:29:40 STEP 1178 H=1178:00 +CPU= 19.841
  16:30:10 STEP 1179 H=1179:00 +CPU= 29.231
  16:30:27 STEP 1180 H=1180:00 +CPU= 17.755
  16:30:45 STEP 1181 H=1181:00 +CPU= 17.269
  16:31:15 STEP 1182 H=1182:00 +CPU= 28.167
  16:31:34 STEP 1183 H=1183:00 +CPU= 18.039
  16:31:54 STEP 1184 H=1184:00 +CPU= 19.795
  16:33:07 STEP 1185 H=1185:00 +CPU= 32.303
  16:33:31 STEP 1186 H=1186:00 +CPU= 18.460
  16:33:50 STEP 1187 H=1187:00 +CPU= 17.657
  16:36:15 STEP 1188 H=1188:00 +CPU= 35.698
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001188
  16:36:35 STEP 1189 H=1189:00 +CPU= 19.501
  16:36:55 STEP 1190 H=1190:00 +CPU= 20.003
  16:37:25 STEP 1191 H=1191:00 +CPU= 29.945
  16:37:46 STEP 1192 H=1192:00 +CPU= 20.154
  16:38:05 STEP 1193 H=1193:00 +CPU= 19.384
  16:38:35 STEP 1194 H=1194:00 +CPU= 30.332
  16:38:55 STEP 1195 H=1195:00 +CPU= 19.351
  16:39:15 STEP 1196 H=1196:00 +CPU= 19.818
  16:39:44 STEP 1197 H=1197:00 +CPU= 29.517
  16:40:04 STEP 1198 H=1198:00 +CPU= 19.917
  16:40:26 STEP 1199 H=1199:00 +CPU= 19.885
  16:41:29 STEP 1200 H=1200:00 +CPU= 36.117
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_49.zip
Uploading the intermediate file: upload_file_49.zip
  16:41:49 STEP 1201 H=1201:00 +CPU= 19.887
  16:42:11 STEP 1202 H=1202:00 +CPU= 19.934
  16:42:41 STEP 1203 H=1203:00 +CPU= 30.379
  16:43:02 STEP 1204 H=1204:00 +CPU= 20.272
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0460
(argv4) batchid: 974
(argv5) wuid: 12191104
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4cdb98d6d0e2c91b5ae49ecb04d148e4 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0460_2005050100_123_974_12191104.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0460_2005050100_123_974_12191104.zip
ic_ancil_file: ic_ancil_12191104
ifsdata_file: ifsdata_12191104
climate_data_file: clim_data_12191104
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3bc68b92bd78065a5e482a446699dde5 to: /var/lib/boinc-client/slots/8/ic_ancil_12191104.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12191104.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c4334ac8c39a104d1a78db0f0650bf76 to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12191104.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12191104.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12191104.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12191104.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12191104.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12191104.xml
-- Model is restarting --
Adjusting last_iter, 1205, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12191104.xml
last_cpu_time: 40978
upload_file_number: 50
last_iter: 1199
last_upload: 4320000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875
The child process has been launched with process id: 2793
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2793:2793] [20230205:164444:1675611884:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:49:06 STEP 1200 H=1200:00 +CPU= 38.428
  16:49:31 STEP 1201 H=1201:00 +CPU= 20.144
  16:49:52 STEP 1202 H=1202:00 +CPU= 20.147
  16:50:20 STEP 1203 H=1203:00 +CPU= 27.969
  16:50:38 STEP 1204 H=1204:00 +CPU= 17.760
  16:50:55 STEP 1205 H=1205:00 +CPU= 17.227
  16:51:23 STEP 1206 H=1206:00 +CPU= 28.115
  16:51:41 STEP 1207 H=1207:00 +CPU= 17.695
  16:51:59 STEP 1208 H=1208:00 +CPU= 17.979
  16:52:33 STEP 1209 H=1209:00 +CPU= 29.011
  16:52:53 STEP 1210 H=1210:00 +CPU= 19.729
  16:53:12 STEP 1211 H=1211:00 +CPU= 19.226
  16:53:53 STEP 1212 H=1212:00 +CPU= 32.443
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001212
  16:54:10 STEP 1213 H=1213:00 +CPU= 17.260
  16:54:28 STEP 1214 H=1214:00 +CPU= 17.754
  16:54:57 STEP 1215 H=1215:00 +CPU= 27.328
  16:55:18 STEP 1216 H=1216:00 +CPU= 18.640
  16:55:37 STEP 1217 H=1217:00 +CPU= 18.992
  16:56:42 STEP 1218 H=1218:00 +CPU= 31.871
  16:57:00 STEP 1219 H=1219:00 +CPU= 17.297
  16:57:20 STEP 1220 H=1220:00 +CPU= 18.427
  16:57:59 STEP 1221 H=1221:00 +CPU= 27.685
  16:58:19 STEP 1222 H=1222:00 +CPU= 18.478
  16:59:09 STEP 1223 H=1223:00 +CPU= 22.501
  17:00:04 STEP 1224 H=1224:00 +CPU= 33.955
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_50.zip
Uploading the intermediate file: upload_file_50.zip
  17:00:23 STEP 1225 H=1225:00 +CPU= 19.453
  17:00:43 STEP 1226 H=1226:00 +CPU= 19.551
  17:01:10 STEP 1227 H=1227:00 +CPU= 26.837
  17:01:28 STEP 1228 H=1228:00 +CPU= 17.710
  17:01:46 STEP 1229 H=1229:00 +CPU= 17.408
  17:02:14 STEP 1230 H=1230:00 +CPU= 28.239
  17:02:32 STEP 1231 H=1231:00 +CPU= 17.439
  17:02:50 STEP 1232 H=1232:00 +CPU= 18.329
  17:03:49 STEP 1233 H=1233:00 +CPU= 30.579
  17:04:09 STEP 1234 H=1234:00 +CPU= 19.808
  17:04:28 STEP 1235 H=1235:00 +CPU= 18.409
  17:05:00 STEP 1236 H=1236:00 +CPU= 29.967
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001236
  17:05:17 STEP 1237 H=1237:00 +CPU= 17.251
  17:05:36 STEP 1238 H=1238:00 +CPU= 18.424
  17:06:05 STEP 1239 H=1239:00 +CPU= 27.260
  17:06:24 STEP 1240 H=1240:00 +CPU= 18.241
  17:06:50 STEP 1241 H=1241:00 +CPU= 19.693
  17:07:29 STEP 1242 H=1242:00 +CPU= 30.499
  17:07:49 STEP 1243 H=1243:00 +CPU= 18.933
  17:08:09 STEP 1244 H=1244:00 +CPU= 20.204
  17:08:39 STEP 1245 H=1245:00 +CPU= 29.963
  17:08:59 STEP 1246 H=1246:00 +CPU= 19.714
  17:09:19 STEP 1247 H=1247:00 +CPU= 19.996
  17:10:08 STEP 1248 H=1248:00 +CPU= 34.765
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_51.zip
Uploading the intermediate file: upload_file_51.zip
  17:10:26 STEP 1249 H=1249:00 +CPU= 17.230
  17:10:44 STEP 1250 H=1250:00 +CPU= 17.778
  17:11:13 STEP 1251 H=1251:00 +CPU= 27.400
  17:11:33 STEP 1252 H=1252:00 +CPU= 18.327
  17:11:53 STEP 1253 H=1253:00 +CPU= 17.395
  17:12:22 STEP 1254 H=1254:00 +CPU= 28.191
  17:13:04 STEP 1255 H=1255:00 +CPU= 20.670
  17:13:24 STEP 1256 H=1256:00 +CPU= 18.922
  17:13:53 STEP 1257 H=1257:00 +CPU= 28.137
  17:14:13 STEP 1258 H=1258:00 +CPU= 19.918
  17:14:32 STEP 1259 H=1259:00 +CPU= 18.991
  17:15:02 STEP 1260 H=1260:00 +CPU= 29.944
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001260
  17:15:19 STEP 1261 H=1261:00 +CPU= 17.250
  17:15:38 STEP 1262 H=1262:00 +CPU= 18.056
  17:16:07 STEP 1263 H=1263:00 +CPU= 27.630
  17:16:25 STEP 1264 H=1264:00 +CPU= 17.950
  17:16:46 STEP 1265 H=1265:00 +CPU= 17.849
  17:17:51 STEP 1266 H=1266:00 +CPU= 32.186
  17:18:11 STEP 1267 H=1267:00 +CPU= 19.251
  17:18:55 STEP 1268 H=1268:00 +CPU= 20.228
  17:19:23 STEP 1269 H=1269:00 +CPU= 26.906
  17:19:43 STEP 1270 H=1270:00 +CPU= 18.006
  17:20:21 STEP 1271 H=1271:00 +CPU= 18.537
  17:23:11 STEP 1272 H=1272:00 +CPU= 36.047
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_52.zip
Uploading the intermediate file: upload_file_52.zip
  17:23:33 STEP 1273 H=1273:00 +CPU= 20.249
  17:23:53 STEP 1274 H=1274:00 +CPU= 20.381
  17:24:24 STEP 1275 H=1275:00 +CPU= 30.499
  17:24:44 STEP 1276 H=1276:00 +CPU= 20.425
  17:25:04 STEP 1277 H=1277:00 +CPU= 19.997
  17:25:35 STEP 1278 H=1278:00 +CPU= 30.788
  17:25:55 STEP 1279 H=1279:00 +CPU= 19.930
  17:26:16 STEP 1280 H=1280:00 +CPU= 20.347
  17:26:46 STEP 1281 H=1281:00 +CPU= 29.836
  17:27:06 STEP 1282 H=1282:00 +CPU= 20.252
  17:27:25 STEP 1283 H=1283:00 +CPU= 19.542
  17:27:59 STEP 1284 H=1284:00 +CPU= 32.206
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001284
  17:28:16 STEP 1285 H=1285:00 +CPU= 17.305
  17:28:34 STEP 1286 H=1286:00 +CPU= 17.726
  17:29:02 STEP 1287 H=1287:00 +CPU= 27.309
  17:29:20 STEP 1288 H=1288:00 +CPU= 18.150
  17:29:39 STEP 1289 H=1289:00 +CPU= 17.494
  17:30:10 STEP 1290 H=1290:00 +CPU= 30.043
  17:30:52 STEP 1291 H=1291:00 +CPU= 20.067
  17:31:12 STEP 1292 H=1292:00 +CPU= 19.509
  17:31:43 STEP 1293 H=1293:00 +CPU= 29.735
  17:32:03 STEP 1294 H=1294:00 +CPU= 20.148
  17:32:23 STEP 1295 H=1295:00 +CPU= 19.703
  17:33:19 STEP 1296 H=1296:00 +CPU= 36.259
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_53.zip
Uploading the intermediate file: upload_file_53.zip
  17:33:38 STEP 1297 H=1297:00 +CPU= 19.432
  17:33:58 STEP 1298 H=1298:00 +CPU= 20.128
  17:34:30 STEP 1299 H=1299:00 +CPU= 30.901
  17:34:50 STEP 1300 H=1300:00 +CPU= 19.961
  17:35:09 STEP 1301 H=1301:00 +CPU= 19.731
  17:35:40 STEP 1302 H=1302:00 +CPU= 30.445
  17:35:59 STEP 1303 H=1303:00 +CPU= 19.319
  17:36:20 STEP 1304 H=1304:00 +CPU= 20.218
  17:36:52 STEP 1305 H=1305:00 +CPU= 29.406
  17:37:10 STEP 1306 H=1306:00 +CPU= 17.722
  17:37:27 STEP 1307 H=1307:00 +CPU= 17.229
  17:37:59 STEP 1308 H=1308:00 +CPU= 30.760
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001308
  17:38:18 STEP 1309 H=1309:00 +CPU= 17.601
  17:38:39 STEP 1310 H=1310:00 +CPU= 18.078
  17:39:24 STEP 1311 H=1311:00 +CPU= 29.919
  17:39:44 STEP 1312 H=1312:00 +CPU= 19.739
  17:40:04 STEP 1313 H=1313:00 +CPU= 19.331
  17:40:35 STEP 1314 H=1314:00 +CPU= 30.618
  17:40:55 STEP 1315 H=1315:00 +CPU= 19.383
  17:41:15 STEP 1316 H=1316:00 +CPU= 20.075
  17:41:45 STEP 1317 H=1317:00 +CPU= 29.889
  17:42:05 STEP 1318 H=1318:00 +CPU= 19.820
  17:42:26 STEP 1319 H=1319:00 +CPU= 20.219
  17:43:10 STEP 1320 H=1320:00 +CPU= 34.238
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_54.zip
Uploading the intermediate file: upload_file_54.zip
  17:43:27 STEP 1321 H=1321:00 +CPU= 17.239
  17:43:45 STEP 1322 H=1322:00 +CPU= 17.671
  17:44:14 STEP 1323 H=1323:00 +CPU= 27.389
  17:44:35 STEP 1324 H=1324:00 +CPU= 18.736
  17:44:54 STEP 1325 H=1325:00 +CPU= 19.328
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0460
(argv4) batchid: 974
(argv5) wuid: 12191104
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4cdb98d6d0e2c91b5ae49ecb04d148e4 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0460_2005050100_123_974_12191104.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0460_2005050100_123_974_12191104.zip
ic_ancil_file: ic_ancil_12191104
ifsdata_file: ifsdata_12191104
climate_data_file: clim_data_12191104
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3bc68b92bd78065a5e482a446699dde5 to: /var/lib/boinc-client/slots/8/ic_ancil_12191104.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12191104.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c4334ac8c39a104d1a78db0f0650bf76 to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12191104.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12191104.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12191104.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12191104.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12191104.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12191104.xml
-- Model is restarting --
Adjusting last_iter, 1326, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12191104.xml
last_cpu_time: 43854
upload_file_number: 55
last_iter: 1319
last_upload: 4752000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875
The child process has been launched with process id: 2919
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:manuel-Lenovo-G50-80:1:1:2919:2919] [20230205:174650:1675615610:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:48:31 STEP 1320 H=1320:00 +CPU= 35.268
  17:48:52 STEP 1321 H=1321:00 +CPU= 19.644
  17:49:12 STEP 1322 H=1322:00 +CPU= 20.321
  17:49:43 STEP 1323 H=1323:00 +CPU= 29.505
  17:50:01 STEP 1324 H=1324:00 +CPU= 18.408
  17:50:19 STEP 1325 H=1325:00 +CPU= 17.464
  17:50:46 STEP 1326 H=1326:00 +CPU= 27.390
  17:51:04 STEP 1327 H=1327:00 +CPU= 17.675
  17:51:22 STEP 1328 H=1328:00 +CPU= 18.160
  17:51:56 STEP 1329 H=1329:00 +CPU= 30.155
  17:52:18 STEP 1330 H=1330:00 +CPU= 18.934
  17:52:35 STEP 1331 H=1331:00 +CPU= 17.199
  17:53:09 STEP 1332 H=1332:00 +CPU= 30.190
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001332
  17:53:29 STEP 1333 H=1333:00 +CPU= 18.135
  17:53:50 STEP 1334 H=1334:00 +CPU= 18.315
  17:54:22 STEP 1335 H=1335:00 +CPU= 27.568
  17:54:52 STEP 1336 H=1336:00 +CPU= 20.569
  17:55:21 STEP 1337 H=1337:00 +CPU= 18.751
  17:55:51 STEP 1338 H=1338:00 +CPU= 28.953
  17:56:10 STEP 1339 H=1339:00 +CPU= 19.350
  17:56:30 STEP 1340 H=1340:00 +CPU= 20.041
  17:57:00 STEP 1341 H=1341:00 +CPU= 29.756
  17:57:21 STEP 1342 H=1342:00 +CPU= 20.033
  17:57:42 STEP 1343 H=1343:00 +CPU= 20.517
  17:58:39 STEP 1344 H=1344:00 +CPU= 35.784
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191104/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_2005050100_123_974_12191104_1_r1853119875_55.zip
Uploading the intermediate file: upload_file_55.zip
  17:58:57 STEP 1345 H=1345:00 +CPU= 18.092
  17:59:15 STEP 1346 H=1346:00 +CPU= 17.821
  17:59:42 STEP 1347 H=1347:00 +CPU= 26.866
  18:00:01 STEP 1348 H=1348:00 +CPU= 18.242
  18:00:21 STEP 1349 H=1349:00 +CPU= 18.068
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:41:15 1506277 22294560 oifs_43r3_ps_0460_2005050100_123_974_12191104_1 1,296,000 9,269 0.0072


©2024 climateprediction.net