Name | oifs_43r3_ps_0561_1989050100_123_958_12175205_1 |
Workunit | 12175205 |
Created | 21 Dec 2022, 21:41:56 UTC |
Sent | 21 Dec 2022, 21:50:48 UTC |
Report deadline | 20 Jan 2023, 21:50:48 UTC |
Received | 23 Dec 2022, 11:17:26 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 1 (0x00000001) Unknown error code |
Computer ID | 1534826 |
Run time | 33 min 13 sec |
CPU time | 43 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 1.00 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,420.12 MB |
Peak swap size | 4,974.05 MB |
Peak disk usage | 1,206.94 MB |
Stderr | <core_client_version>7.18.1</core_client_version> <![CDATA[ <message> process exited with code 1 (0x1, -255)</message> <stderr_txt> 12e34a56fc2e5f9cf to: /root/slots/1/ifsdata/ifsdata_12175205.zip Unzipping the ifsdata_zip file: /root/slots/1/ifsdata/ifsdata_12175205.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/1/159l_2/clim_data_12175205.zip Unzipping the climate data zip file: /root/slots/1/159l_2/clim_data_12175205.zip Checking for progress XML file: /root/slots/1/progress_file_12175205.xml Creating progress file: /root/slots/1/progress_file_12175205.xml last_cpu_time: 0 upload_file_number: 0 last_iter: 0 last_upload: 0 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0561_1989050100_123_958_12175205_1_r1961723854 The child process has been launched with process id: 9145 Executing the command: /root/slots/1/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:9145:9145] [20221222:155555:1671720955:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:56:35 STEP 0 H= 0:00 +CPU= 28.305 Moving to projects directory: /root/slots/1/ICMGGhq0f+000000 Moving to projects directory: /root/slots/1/ICMSHhq0f+000000 Moving to projects directory: /root/slots/1/ICMUAhq0f+000000 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:56:58 STEP 1 H= 1:00 +CPU= 19.132 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:57:21 STEP 2 H= 2:00 +CPU= 17.417 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:57:55 STEP 3 H= 3:00 +CPU= 25.932 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:58:15 STEP 4 H= 4:00 +CPU= 16.390 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:58:36 STEP 5 H= 5:00 +CPU= 16.283 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:59:08 STEP 6 H= 6:00 +CPU= 26.215 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:59:28 STEP 7 H= 7:00 +CPU= 16.128 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:59:49 STEP 8 H= 8:00 +CPU= 16.334 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:00:20 STEP 9 H= 9:00 +CPU= 25.246 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:00:41 STEP 10 H= 10:00 +CPU= 16.405 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:01:01 STEP 11 H= 11:00 +CPU= 15.641 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:01:36 STEP 12 H= 12:00 +CPU= 27.203 Moving to projects directory: /root/slots/1/ICMGGhq0f+000012 Moving to projects directory: /root/slots/1/ICMSHhq0f+000012 Moving to projects directory: /root/slots/1/ICMUAhq0f+000012 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:01:55 STEP 13 H= 13:00 +CPU= 15.151 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:02:15 STEP 14 H= 14:00 +CPU= 15.560 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:02:46 STEP 15 H= 15:00 +CPU= 24.563 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:03:05 STEP 16 H= 16:00 +CPU= 15.734 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:03:24 STEP 17 H= 17:00 +CPU= 14.927 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:03:55 STEP 18 H= 18:00 +CPU= 24.169 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:04:14 STEP 19 H= 19:00 +CPU= 15.429 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:04:34 STEP 20 H= 20:00 +CPU= 15.476 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:05:06 STEP 21 H= 21:00 +CPU= 25.937 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:05:27 STEP 22 H= 22:00 +CPU= 16.753 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:05:48 STEP 23 H= 23:00 +CPU= 16.993 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:06:26 STEP 24 H= 24:00 +CPU= 30.017 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /root/slots/1/ICMGGhq0f+000024 Moving to projects directory: /root/slots/1/ICMSHhq0f+000024 Moving to projects directory: /root/slots/1/ICMUAhq0f+000024 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMGGhq0f+000000 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMSHhq0f+000000 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMUAhq0f+000000 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMGGhq0f+000012 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMSHhq0f+000012 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMUAhq0f+000012 Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0561_1989050100_123_958_12175205_1_r1961723854_0.zip Uploading the intermediate file: upload_file_0.zip 16:06:43 STEP 25 H= 25:00 +CPU= 15.005 16:07:00 STEP 26 H= 26:00 +CPU= 16.102 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:07:28 STEP 27 H= 27:00 +CPU= 23.818 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:07:48 STEP 28 H= 28:00 +CPU= 16.584 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:08:08 STEP 29 H= 29:00 +CPU= 16.005 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:08:43 STEP 30 H= 30:00 +CPU= 26.276 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:09:03 STEP 31 H= 31:00 +CPU= 16.264 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:09:24 STEP 32 H= 32:00 +CPU= 16.856 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:09:56 STEP 33 H= 33:00 +CPU= 25.677 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:10:16 STEP 34 H= 34:00 +CPU= 16.150 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:10:37 STEP 35 H= 35:00 +CPU= 16.188 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:11:10 STEP 36 H= 36:00 +CPU= 27.724 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /root/slots/1/ICMGGhq0f+000036 Moving to projects directory: /root/slots/1/ICMSHhq0f+000036 Moving to projects directory: /root/slots/1/ICMUAhq0f+000036 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:11:30 STEP 37 H= 37:00 +CPU= 15.704 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:11:51 STEP 38 H= 38:00 +CPU= 16.407 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:12:24 STEP 39 H= 39:00 +CPU= 24.905 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:12:44 STEP 40 H= 40:00 +CPU= 16.438 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:13:04 STEP 41 H= 41:00 +CPU= 15.342 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:13:36 STEP 42 H= 42:00 +CPU= 25.896 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:13:56 STEP 43 H= 43:00 +CPU= 15.743 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:14:16 STEP 44 H= 44:00 +CPU= 16.244 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:14:47 STEP 45 H= 45:00 +CPU= 25.165 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:15:07 STEP 46 H= 46:00 +CPU= 15.814 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:15:27 STEP 47 H= 47:00 +CPU= 15.620 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:16:01 STEP 48 H= 48:00 +CPU= 28.113 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /root/slots/1/ICMGGhq0f+000048 Moving to projects directory: /root/slots/1/ICMSHhq0f+000048 Moving to projects directory: /root/slots/1/ICMUAhq0f+000048 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMGGhq0f+000024 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMSHhq0f+000024 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMUAhq0f+000024 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMGGhq0f+000036 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMSHhq0f+000036 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMUAhq0f+000036 Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0561_1989050100_123_958_12175205_1_r1961723854_1.zip Uploading the intermediate file: upload_file_1.zip 16:16:19 STEP 49 H= 49:00 +CPU= 15.511 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:16:37 STEP 50 H= 50:00 +CPU= 15.364 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:17:08 STEP 51 H= 51:00 +CPU= 25.222 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:17:27 STEP 52 H= 52:00 +CPU= 15.297 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:17:48 STEP 53 H= 53:00 +CPU= 16.112 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:18:20 STEP 54 H= 54:00 +CPU= 25.765 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:18:39 STEP 55 H= 55:00 +CPU= 15.620 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:18:59 STEP 56 H= 56:00 +CPU= 16.115 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:19:30 STEP 57 H= 57:00 +CPU= 24.690 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:19:50 STEP 58 H= 58:00 +CPU= 15.687 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:20:09 STEP 59 H= 59:00 +CPU= 15.102 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:20:44 STEP 60 H= 60:00 +CPU= 26.732 Moving to projects directory: /root/slots/1/ICMGGhq0f+000060 Moving to projects directory: /root/slots/1/ICMSHhq0f+000060 Moving to projects directory: /root/slots/1/ICMUAhq0f+000060 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:21:05 STEP 61 H= 61:00 +CPU= 16.174 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:21:25 STEP 62 H= 62:00 +CPU= 16.438 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:21:57 STEP 63 H= 63:00 +CPU= 25.385 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:22:17 STEP 64 H= 64:00 +CPU= 16.658 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:22:38 STEP 65 H= 65:00 +CPU= 16.305 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:23:10 STEP 66 H= 66:00 +CPU= 25.718 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:23:31 STEP 67 H= 67:00 +CPU= 16.706 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:23:51 STEP 68 H= 68:00 +CPU= 16.366 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:24:24 STEP 69 H= 69:00 +CPU= 25.158 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:24:45 STEP 70 H= 70:00 +CPU= 16.430 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:25:05 STEP 71 H= 71:00 +CPU= 16.339 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:25:41 STEP 72 H= 72:00 +CPU= 29.236 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /root/slots/1/ICMGGhq0f+000072 Moving to projects directory: /root/slots/1/ICMSHhq0f+000072 Moving to projects directory: /root/slots/1/ICMUAhq0f+000072 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMGGhq0f+000048 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMSHhq0f+000048 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMUAhq0f+000048 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMGGhq0f+000060 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMSHhq0f+000060 Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12175205/ICMUAhq0f+000060 Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0561_1989050100_123_958_12175205_1_r1961723854_2.zip Uploading the intermediate file: upload_file_2.zip 16:25:58 STEP 73 H= 73:00 +CPU= 15.494 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:26:17 STEP 74 H= 74:00 +CPU= 16.127 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:26:48 STEP 75 H= 75:00 +CPU= 25.008 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:27:08 STEP 76 H= 76:00 +CPU= 16.340 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:27:28 STEP 77 H= 77:00 +CPU= 15.706 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:27:59 STEP 78 H= 78:00 +CPU= 25.193 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:28:19 STEP 79 H= 79:00 +CPU= 15.835 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:28:40 STEP 80 H= 80:00 +CPU= 16.138 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:29:10 STEP 81 H= 81:00 +CPU= 24.529 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:29:30 STEP 82 H= 82:00 +CPU= 15.843 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:29:50 STEP 83 H= 83:00 +CPU= 15.622 Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1989050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0561 (argv4) batchid: 958 (argv5) wuid: 12175205 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /root/slots/1 Project directory is: /root/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12175205 ..mkdir for temp folder for results failed Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /root/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_e9a232e86ebb6f54071a5e434d9c280d to: /root/slots/1/oifs_43r3_ps_0561_1989050100_123_958_12175205.zip Unzipping the namelist zip file: /root/slots/1/oifs_43r3_ps_0561_1989050100_123_958_12175205.zip ic_ancil_file: ic_ancil_12175205 ifsdata_file: ifsdata_12175205 climate_data_file: clim_data_12175205 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_eea45a3daeac360a197b6ddad99d7b14 to: /root/slots/1/ic_ancil_12175205.zip Unzipping the IC ancils zip file: /root/slots/1/ic_ancil_12175205.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c423325fd466fcc12e34a56fc2e5f9cf to: /root/slots/1/ifsdata/ifsdata_12175205.zip Unzipping the ifsdata_zip file: /root/slots/1/ifsdata/ifsdata_12175205.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/1/159l_2/clim_data_12175205.zip Unzipping the climate data zip file: /root/slots/1/159l_2/clim_data_12175205.zip Checking for progress XML file: /root/slots/1/progress_file_12175205.xml Opened progress file ok : /root/slots/1/progress_file_12175205.xml -- Model is restarting -- Adjusting last_iter, 84, to previous model restart step. Creating progress file: /root/slots/1/progress_file_12175205.xml last_cpu_time: 1633 upload_file_number: 3 last_iter: 71 last_upload: 259200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0561_1989050100_123_958_12175205_1_r1961723854 The child process has been launched with process id: 5910 Executing the command: /root/slots/1/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:5910:5910] [20221223:121125:1671793885:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:12:00 STEP 72 H= 72:00 +CPU= 24.774 Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1989050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0561 (argv4) batchid: 958 (argv5) wuid: 12175205 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /root/slots/1 Project directory is: /root/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12175205 ..mkdir for temp folder for results failed Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /root/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_e9a232e86ebb6f54071a5e434d9c280d to: /root/slots/1/oifs_43r3_ps_0561_1989050100_123_958_12175205.zip Unzipping the namelist zip file: /root/slots/1/oifs_43r3_ps_0561_1989050100_123_958_12175205.zip ic_ancil_file: ic_ancil_12175205 ifsdata_file: ifsdata_12175205 climate_data_file: clim_data_12175205 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_eea45a3daeac360a197b6ddad99d7b14 to: /root/slots/1/ic_ancil_12175205.zip Unzipping the IC ancils zip file: /root/slots/1/ic_ancil_12175205.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c423325fd466fcc12e34a56fc2e5f9cf to: /root/slots/1/ifsdata/ifsdata_12175205.zip Unzipping the ifsdata_zip file: /root/slots/1/ifsdata/ifsdata_12175205.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/1/159l_2/clim_data_12175205.zip Unzipping the climate data zip file: /root/slots/1/159l_2/clim_data_12175205.zip Checking for progress XML file: /root/slots/1/progress_file_12175205.xml Opened progress file ok : /root/slots/1/progress_file_12175205.xml -- Model is restarting -- Adjusting last_iter, 73, to previous model restart step. Creating progress file: /root/slots/1/progress_file_12175205.xml last_cpu_time: 1661 upload_file_number: 3 last_iter: 71 last_upload: 259200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0561_1989050100_123_958_12175205_1_r1961723854 The child process has been launched with process id: 6130 Executing the command: /root/slots/1/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:6130:6130] [20221223:121309:1671793989:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Quit request received from the BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1989050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0561 (argv4) batchid: 958 (argv5) wuid: 12175205 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /root/slots/1 Project directory is: /root/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12175205 ..mkdir for temp folder for results failed Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /root/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_e9a232e86ebb6f54071a5e434d9c280d to: /root/slots/1/oifs_43r3_ps_0561_1989050100_123_958_12175205.zip Unzipping the namelist zip file: /root/slots/1/oifs_43r3_ps_0561_1989050100_123_958_12175205.zip ic_ancil_file: ic_ancil_12175205 ifsdata_file: ifsdata_12175205 climate_data_file: clim_data_12175205 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_eea45a3daeac360a197b6ddad99d7b14 to: /root/slots/1/ic_ancil_12175205.zip Unzipping the IC ancils zip file: /root/slots/1/ic_ancil_12175205.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c423325fd466fcc12e34a56fc2e5f9cf to: /root/slots/1/ifsdata/ifsdata_12175205.zip Unzipping the ifsdata_zip file: /root/slots/1/ifsdata/ifsdata_12175205.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/1/159l_2/clim_data_12175205.zip Unzipping the climate data zip file: /root/slots/1/159l_2/clim_data_12175205.zip Checking for progress XML file: /root/slots/1/progress_file_12175205.xml Creating progress file: /root/slots/1/progress_file_12175205.xml last_cpu_time: 0 upload_file_number: 0 last_iter: 0 last_upload: 0 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0561_1989050100_123_958_12175205_1_r1961723854 The child process has been launched with process id: 6150 Executing the command: /root/slots/1/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:50-privat:1:1:6150:6150] [20221223:121417:1671794057:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:14:56 STEP 72 H= 72:00 +CPU= 28.716 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process The child process terminated with status: 0 ..Failed, model did not complete successfully </stderr_txt> ]]> |
No trickles! |
---|
©2024 climateprediction.net