climateprediction.net home page
Task 22295072

Task 22295072

Name oifs_43r3_ps_0023_2014050100_123_983_12199667_1
Workunit 12199667
Created 1 Jan 2023, 17:57:36 UTC
Sent 15 Jan 2023, 3:50:33 UTC
Report deadline 14 Feb 2023, 3:50:33 UTC
Received 15 Jan 2023, 7:20:14 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1538137
Run time 44 min 57 sec
CPU time 42 min 48 sec
Validate state Invalid
Credit 1,451.52
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,180.36 MB
Peak swap size 4,978.06 MB
Peak disk usage 2,031.77 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0023
(argv4) batchid: 983
(argv5) wuid: 12199667
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0aa6ed037377ae25b8044d6aa6dee5f0 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0023_2014050100_123_983_12199667.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0023_2014050100_123_983_12199667.zip
ic_ancil_file: ic_ancil_12199667
ifsdata_file: ifsdata_12199667
climate_data_file: clim_data_12199667
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e40cc7015937ba69e8d183d75f61ef90 to: /var/lib/boinc-client/slots/1/ic_ancil_12199667.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12199667.zip
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_79c9730cae5dbce6749779c34fc85ef8 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12199667.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12199667.zip
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12199667.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12199667.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12199667.xml
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12199667.xml
last_cpu_time: 0
upload_file_number: 0
last_iter: 0
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424
The child process has been launched with process id: 2016
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2016:2016] [20230114:220715:1673755635:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:07:28 STEP    0 H=   0:00 +CPU= 11.768
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000000
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000000
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000000
  22:07:36 STEP    1 H=   1:00 +CPU=  7.817
  22:07:43 STEP    2 H=   2:00 +CPU=  6.960
  22:07:54 STEP    3 H=   3:00 +CPU= 11.107
  22:08:01 STEP    4 H=   4:00 +CPU=  6.758
  22:08:08 STEP    5 H=   5:00 +CPU=  6.748
  22:08:18 STEP    6 H=   6:00 +CPU= 10.827
  22:08:25 STEP    7 H=   7:00 +CPU=  6.653
  22:08:32 STEP    8 H=   8:00 +CPU=  7.112
  22:08:43 STEP    9 H=   9:00 +CPU= 10.992
  22:08:50 STEP   10 H=  10:00 +CPU=  6.585
  22:08:57 STEP   11 H=  11:00 +CPU=  6.625
  22:09:09 STEP   12 H=  12:00 +CPU= 11.933
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000012
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000012
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000012
  22:09:16 STEP   13 H=  13:00 +CPU=  6.678
  22:09:26 STEP   14 H=  14:00 +CPU=  6.999
  22:09:37 STEP   15 H=  15:00 +CPU= 10.543
  22:09:44 STEP   16 H=  16:00 +CPU=  6.857
  22:09:51 STEP   17 H=  17:00 +CPU=  6.890
  22:10:01 STEP   18 H=  18:00 +CPU= 10.710
  22:10:08 STEP   19 H=  19:00 +CPU=  6.643
  22:10:15 STEP   20 H=  20:00 +CPU=  6.629
  22:10:25 STEP   21 H=  21:00 +CPU= 10.515
  22:10:32 STEP   22 H=  22:00 +CPU=  6.738
  22:10:39 STEP   23 H=  23:00 +CPU=  6.959
  22:10:52 STEP   24 H=  24:00 +CPU= 12.703
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000024
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000024
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000012
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000012
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000012
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424_0.zip
Uploading the intermediate file: upload_file_0.zip
  22:10:59 STEP   25 H=  25:00 +CPU=  6.719
  22:11:05 STEP   26 H=  26:00 +CPU=  6.728
  22:11:16 STEP   27 H=  27:00 +CPU= 10.916
  22:11:23 STEP   28 H=  28:00 +CPU=  6.733
  22:11:30 STEP   29 H=  29:00 +CPU=  6.746
  22:11:40 STEP   30 H=  30:00 +CPU= 10.640
  22:11:47 STEP   31 H=  31:00 +CPU=  6.637
  22:11:54 STEP   32 H=  32:00 +CPU=  6.736
  22:12:04 STEP   33 H=  33:00 +CPU= 10.640
  22:12:11 STEP   34 H=  34:00 +CPU=  6.496
  22:12:18 STEP   35 H=  35:00 +CPU=  6.931
  22:12:30 STEP   36 H=  36:00 +CPU= 11.818
  22:12:36 STEP   37 H=  37:00 +CPU=  6.598
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000036
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000036
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000036
  22:12:43 STEP   38 H=  38:00 +CPU=  7.002
  22:12:54 STEP   39 H=  39:00 +CPU= 10.320
  22:13:01 STEP   40 H=  40:00 +CPU=  7.363
  22:13:11 STEP   41 H=  41:00 +CPU=  6.809
  22:13:22 STEP   42 H=  42:00 +CPU= 10.658
  22:13:29 STEP   43 H=  43:00 +CPU=  6.598
  22:13:36 STEP   44 H=  44:00 +CPU=  6.752
  22:13:46 STEP   45 H=  45:00 +CPU= 10.499
  22:13:53 STEP   46 H=  46:00 +CPU=  6.933
  22:14:00 STEP   47 H=  47:00 +CPU=  6.784
  22:14:14 STEP   48 H=  48:00 +CPU= 12.504
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000048
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000048
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000036
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000036
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000036
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424_1.zip
Uploading the intermediate file: upload_file_1.zip
  22:14:20 STEP   49 H=  49:00 +CPU=  6.266
  22:14:27 STEP   50 H=  50:00 +CPU=  6.848
  22:14:38 STEP   51 H=  51:00 +CPU= 10.715
  22:14:45 STEP   52 H=  52:00 +CPU=  6.822
  22:14:52 STEP   53 H=  53:00 +CPU=  6.683
  22:15:02 STEP   54 H=  54:00 +CPU= 10.734
  22:15:09 STEP   55 H=  55:00 +CPU=  6.613
  22:15:16 STEP   56 H=  56:00 +CPU=  6.783
  22:15:26 STEP   57 H=  57:00 +CPU= 10.729
  22:15:33 STEP   58 H=  58:00 +CPU=  6.724
  22:15:39 STEP   59 H=  59:00 +CPU=  6.379
  22:15:52 STEP   60 H=  60:00 +CPU= 12.000
  22:15:58 STEP   61 H=  61:00 +CPU=  6.733
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000060
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000060
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000060
  22:16:05 STEP   62 H=  62:00 +CPU=  6.866
  22:16:16 STEP   63 H=  63:00 +CPU= 10.735
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0023
(argv4) batchid: 983
(argv5) wuid: 12199667
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0aa6ed037377ae25b8044d6aa6dee5f0 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0023_2014050100_123_983_12199667.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0023_2014050100_123_983_12199667.zip
ic_ancil_file: ic_ancil_12199667
ifsdata_file: ifsdata_12199667
climate_data_file: clim_data_12199667
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e40cc7015937ba69e8d183d75f61ef90 to: /var/lib/boinc-client/slots/1/ic_ancil_12199667.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12199667.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_79c9730cae5dbce6749779c34fc85ef8 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12199667.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12199667.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12199667.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12199667.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12199667.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12199667.xml
-- Model is restarting --
Adjusting last_iter, 64, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12199667.xml
last_cpu_time: 531
upload_file_number: 2
last_iter: 47
last_upload: 172800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424
The child process has been launched with process id: 2033
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2033:2033] [20230114:221722:1673756242:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:17:35 STEP   48 H=  48:00 +CPU= 11.820
  22:17:42 STEP   49 H=  49:00 +CPU=  6.462
  22:17:49 STEP   50 H=  50:00 +CPU=  6.953
  22:17:59 STEP   51 H=  51:00 +CPU= 10.187
  22:18:06 STEP   52 H=  52:00 +CPU=  6.894
  22:18:13 STEP   53 H=  53:00 +CPU=  6.602
  22:18:23 STEP   54 H=  54:00 +CPU= 10.695
  22:18:39 STEP   55 H=  55:00 +CPU=  6.921
  22:18:46 STEP   56 H=  56:00 +CPU=  6.497
  22:18:56 STEP   57 H=  57:00 +CPU= 10.570
  22:19:03 STEP   58 H=  58:00 +CPU=  6.614
  22:19:10 STEP   59 H=  59:00 +CPU=  6.571
  22:19:21 STEP   60 H=  60:00 +CPU= 11.447
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000060
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000060
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000060
  22:19:28 STEP   61 H=  61:00 +CPU=  6.703
  22:19:38 STEP   62 H=  62:00 +CPU=  6.697
  22:19:49 STEP   63 H=  63:00 +CPU= 10.914
  22:19:56 STEP   64 H=  64:00 +CPU=  6.627
  22:20:02 STEP   65 H=  65:00 +CPU=  6.445
  22:20:13 STEP   66 H=  66:00 +CPU= 10.947
  22:20:19 STEP   67 H=  67:00 +CPU=  6.279
  22:20:26 STEP   68 H=  68:00 +CPU=  6.744
  22:20:36 STEP   69 H=  69:00 +CPU= 10.479
  22:20:43 STEP   70 H=  70:00 +CPU=  6.988
  22:20:50 STEP   71 H=  71:00 +CPU=  6.634
  22:21:04 STEP   72 H=  72:00 +CPU= 12.432
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000072
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000072
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000060
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000060
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000060
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424_2.zip
Uploading the intermediate file: upload_file_2.zip
  22:21:11 STEP   73 H=  73:00 +CPU=  6.556
  22:21:18 STEP   74 H=  74:00 +CPU=  7.036
  22:21:28 STEP   75 H=  75:00 +CPU= 10.594
  22:21:35 STEP   76 H=  76:00 +CPU=  6.992
  22:21:42 STEP   77 H=  77:00 +CPU=  6.483
  22:21:52 STEP   78 H=  78:00 +CPU= 10.703
  22:21:59 STEP   79 H=  79:00 +CPU=  6.725
  22:22:06 STEP   80 H=  80:00 +CPU=  6.777
  22:22:17 STEP   81 H=  81:00 +CPU= 10.604
  22:22:25 STEP   82 H=  82:00 +CPU=  7.202
  22:22:31 STEP   83 H=  83:00 +CPU=  6.497
  22:22:45 STEP   84 H=  84:00 +CPU= 11.225
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000084
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000084
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000084
  22:22:52 STEP   85 H=  85:00 +CPU=  6.693
  22:22:58 STEP   86 H=  86:00 +CPU=  6.686
  22:23:09 STEP   87 H=  87:00 +CPU= 10.696
  22:23:16 STEP   88 H=  88:00 +CPU=  7.071
  22:23:23 STEP   89 H=  89:00 +CPU=  6.662
  22:23:34 STEP   90 H=  90:00 +CPU= 11.000
  22:23:40 STEP   91 H=  91:00 +CPU=  6.463
  22:23:47 STEP   92 H=  92:00 +CPU=  7.145
  22:23:58 STEP   93 H=  93:00 +CPU= 10.376
  22:24:05 STEP   94 H=  94:00 +CPU=  6.696
  22:24:11 STEP   95 H=  95:00 +CPU=  6.869
  22:24:25 STEP   96 H=  96:00 +CPU= 12.481
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000096
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000096
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000084
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000084
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000084
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424_3.zip
Uploading the intermediate file: upload_file_3.zip
  22:24:32 STEP   97 H=  97:00 +CPU=  6.345
  22:24:38 STEP   98 H=  98:00 +CPU=  6.510
  22:24:48 STEP   99 H=  99:00 +CPU= 10.071
  22:24:55 STEP  100 H= 100:00 +CPU=  6.642
  22:25:01 STEP  101 H= 101:00 +CPU=  6.511
  22:25:12 STEP  102 H= 102:00 +CPU= 10.562
  22:25:19 STEP  103 H= 103:00 +CPU=  6.678
  22:25:26 STEP  104 H= 104:00 +CPU=  7.021
  22:25:36 STEP  105 H= 105:00 +CPU= 10.630
  22:25:43 STEP  106 H= 106:00 +CPU=  6.814
  22:25:50 STEP  107 H= 107:00 +CPU=  6.631
  22:26:01 STEP  108 H= 108:00 +CPU= 11.570
  22:26:08 STEP  109 H= 109:00 +CPU=  6.749
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000108
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000108
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000108
  22:26:15 STEP  110 H= 110:00 +CPU=  6.761
  22:26:26 STEP  111 H= 111:00 +CPU= 10.790
  22:26:32 STEP  112 H= 112:00 +CPU=  6.738
  22:26:39 STEP  113 H= 113:00 +CPU=  6.923
  22:26:50 STEP  114 H= 114:00 +CPU= 10.609
  22:26:56 STEP  115 H= 115:00 +CPU=  6.408
  22:27:03 STEP  116 H= 116:00 +CPU=  6.925
  22:27:14 STEP  117 H= 117:00 +CPU= 10.833
  22:27:21 STEP  118 H= 118:00 +CPU=  6.764
  22:27:28 STEP  119 H= 119:00 +CPU=  6.809
  22:27:41 STEP  120 H= 120:00 +CPU= 12.302
  22:27:48 STEP  121 H= 121:00 +CPU=  6.529
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000120
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000120
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000108
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000108
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000108
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424_4.zip
Uploading the intermediate file: upload_file_4.zip
  22:27:55 STEP  122 H= 122:00 +CPU=  6.946
  22:28:06 STEP  123 H= 123:00 +CPU= 10.938
  22:28:13 STEP  124 H= 124:00 +CPU=  6.778
  22:28:19 STEP  125 H= 125:00 +CPU=  6.469
  22:28:30 STEP  126 H= 126:00 +CPU= 10.805
  22:28:37 STEP  127 H= 127:00 +CPU=  6.529
  22:28:43 STEP  128 H= 128:00 +CPU=  6.625
  22:28:54 STEP  129 H= 129:00 +CPU= 10.786
  22:29:01 STEP  130 H= 130:00 +CPU=  6.606
  22:29:07 STEP  131 H= 131:00 +CPU=  6.839
  22:29:19 STEP  132 H= 132:00 +CPU= 11.749
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000132
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000132
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000132
  22:29:26 STEP  133 H= 133:00 +CPU=  6.789
  22:29:33 STEP  134 H= 134:00 +CPU=  6.747
  22:29:45 STEP  135 H= 135:00 +CPU= 10.464
  22:29:51 STEP  136 H= 136:00 +CPU=  6.791
  22:29:58 STEP  137 H= 137:00 +CPU=  6.406
  22:30:09 STEP  138 H= 138:00 +CPU= 10.641
  22:30:15 STEP  139 H= 139:00 +CPU=  6.426
  22:30:22 STEP  140 H= 140:00 +CPU=  6.600
  22:30:33 STEP  141 H= 141:00 +CPU= 10.800
  22:30:39 STEP  142 H= 142:00 +CPU=  6.853
  22:30:46 STEP  143 H= 143:00 +CPU=  6.723
  22:30:59 STEP  144 H= 144:00 +CPU= 12.482
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000144
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000144
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000132
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000132
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000132
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424_5.zip
Uploading the intermediate file: upload_file_5.zip
  22:31:06 STEP  145 H= 145:00 +CPU=  6.857
  22:31:12 STEP  146 H= 146:00 +CPU=  6.655
  22:31:23 STEP  147 H= 147:00 +CPU= 10.707
  22:31:30 STEP  148 H= 148:00 +CPU=  6.818
  22:31:37 STEP  149 H= 149:00 +CPU=  6.722
  22:31:47 STEP  150 H= 150:00 +CPU= 10.605
  22:31:54 STEP  151 H= 151:00 +CPU=  6.895
  22:32:01 STEP  152 H= 152:00 +CPU=  6.744
  22:32:12 STEP  153 H= 153:00 +CPU= 10.748
  22:32:18 STEP  154 H= 154:00 +CPU=  6.875
  22:32:25 STEP  155 H= 155:00 +CPU=  6.552
  22:32:37 STEP  156 H= 156:00 +CPU= 11.781
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000156
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000156
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000156
  22:32:44 STEP  157 H= 157:00 +CPU=  6.812
  22:32:50 STEP  158 H= 158:00 +CPU=  6.604
  22:33:01 STEP  159 H= 159:00 +CPU= 10.707
  22:33:08 STEP  160 H= 160:00 +CPU=  6.756
  22:33:15 STEP  161 H= 161:00 +CPU=  6.648
  22:33:26 STEP  162 H= 162:00 +CPU= 11.189
  22:33:32 STEP  163 H= 163:00 +CPU=  6.612
  22:33:39 STEP  164 H= 164:00 +CPU=  6.814
  22:33:50 STEP  165 H= 165:00 +CPU= 10.494
  22:33:56 STEP  166 H= 166:00 +CPU=  6.754
  22:34:03 STEP  167 H= 167:00 +CPU=  6.645
  22:34:16 STEP  168 H= 168:00 +CPU= 11.991
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000168
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000168
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000156
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000156
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000156
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424_6.zip
Uploading the intermediate file: upload_file_6.zip
  22:34:23 STEP  169 H= 169:00 +CPU=  6.582
  22:34:30 STEP  170 H= 170:00 +CPU=  6.650
  22:34:40 STEP  171 H= 171:00 +CPU= 10.237
  22:34:47 STEP  172 H= 172:00 +CPU=  6.643
  22:34:53 STEP  173 H= 173:00 +CPU=  6.598
  22:35:04 STEP  174 H= 174:00 +CPU= 10.850
  22:35:11 STEP  175 H= 175:00 +CPU=  6.731
  22:35:17 STEP  176 H= 176:00 +CPU=  6.563
  22:35:28 STEP  177 H= 177:00 +CPU= 10.453
  22:35:35 STEP  178 H= 178:00 +CPU=  6.773
  22:35:41 STEP  179 H= 179:00 +CPU=  6.494
  22:35:52 STEP  180 H= 180:00 +CPU= 11.305
  22:35:59 STEP  181 H= 181:00 +CPU=  6.674
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000180
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000180
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000180
  22:36:06 STEP  182 H= 182:00 +CPU=  6.565
  22:36:16 STEP  183 H= 183:00 +CPU= 10.387
  22:36:23 STEP  184 H= 184:00 +CPU=  6.752
  22:36:30 STEP  185 H= 185:00 +CPU=  6.699
  22:36:40 STEP  186 H= 186:00 +CPU= 10.619
  22:36:47 STEP  187 H= 187:00 +CPU=  6.620
  22:36:53 STEP  188 H= 188:00 +CPU=  6.660
  22:37:04 STEP  189 H= 189:00 +CPU= 10.814
  22:37:11 STEP  190 H= 190:00 +CPU=  6.925
  22:37:18 STEP  191 H= 191:00 +CPU=  6.718
  22:37:31 STEP  192 H= 192:00 +CPU= 12.386
  22:37:38 STEP  193 H= 193:00 +CPU=  6.648
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000192
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000192
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000180
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000180
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000180
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424_7.zip
Uploading the intermediate file: upload_file_7.zip
  22:37:45 STEP  194 H= 194:00 +CPU=  6.502
  22:37:57 STEP  195 H= 195:00 +CPU= 10.738
  22:38:03 STEP  196 H= 196:00 +CPU=  6.944
  22:38:10 STEP  197 H= 197:00 +CPU=  6.520
  22:38:21 STEP  198 H= 198:00 +CPU= 10.517
  22:38:27 STEP  199 H= 199:00 +CPU=  6.778
  22:38:34 STEP  200 H= 200:00 +CPU=  6.872
  22:38:44 STEP  201 H= 201:00 +CPU= 10.161
  22:38:51 STEP  202 H= 202:00 +CPU=  6.996
  22:38:58 STEP  203 H= 203:00 +CPU=  6.389
  22:39:10 STEP  204 H= 204:00 +CPU= 11.715
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000204
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000204
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000204
  22:39:16 STEP  205 H= 205:00 +CPU=  6.465
  22:39:23 STEP  206 H= 206:00 +CPU=  6.877
  22:39:34 STEP  207 H= 207:00 +CPU= 10.723
  22:39:41 STEP  208 H= 208:00 +CPU=  6.941
  22:39:47 STEP  209 H= 209:00 +CPU=  6.696
  22:39:58 STEP  210 H= 210:00 +CPU= 10.613
  22:40:05 STEP  211 H= 211:00 +CPU=  6.835
  22:40:12 STEP  212 H= 212:00 +CPU=  6.782
  22:40:22 STEP  213 H= 213:00 +CPU= 10.364
  22:40:29 STEP  214 H= 214:00 +CPU=  7.004
  22:40:36 STEP  215 H= 215:00 +CPU=  6.607
  22:40:48 STEP  216 H= 216:00 +CPU= 12.211
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000216
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000216
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000204
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000204
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000204
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424_8.zip
Uploading the intermediate file: upload_file_8.zip
  22:40:55 STEP  217 H= 217:00 +CPU=  6.537
  22:41:01 STEP  218 H= 218:00 +CPU=  6.765
  22:41:12 STEP  219 H= 219:00 +CPU= 10.359
  22:41:19 STEP  220 H= 220:00 +CPU=  6.689
  22:41:25 STEP  221 H= 221:00 +CPU=  6.366
  22:41:36 STEP  222 H= 222:00 +CPU= 10.656
  22:41:42 STEP  223 H= 223:00 +CPU=  6.526
  22:41:49 STEP  224 H= 224:00 +CPU=  7.048
  22:42:00 STEP  225 H= 225:00 +CPU= 10.337
  22:42:06 STEP  226 H= 226:00 +CPU=  6.857
  22:42:13 STEP  227 H= 227:00 +CPU=  6.599
  22:42:25 STEP  228 H= 228:00 +CPU= 11.722
  22:42:32 STEP  229 H= 229:00 +CPU=  6.717
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000228
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000228
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000228
  22:42:38 STEP  230 H= 230:00 +CPU=  6.790
  22:42:49 STEP  231 H= 231:00 +CPU= 10.627
  22:42:56 STEP  232 H= 232:00 +CPU=  6.788
  22:43:03 STEP  233 H= 233:00 +CPU=  6.768
  22:43:13 STEP  234 H= 234:00 +CPU= 10.694
  22:43:20 STEP  235 H= 235:00 +CPU=  6.441
  22:43:26 STEP  236 H= 236:00 +CPU=  6.677
  22:43:37 STEP  237 H= 237:00 +CPU= 10.277
  22:43:43 STEP  238 H= 238:00 +CPU=  6.463
  22:43:50 STEP  239 H= 239:00 +CPU=  6.706
  22:44:03 STEP  240 H= 240:00 +CPU= 12.148
  22:44:09 STEP  241 H= 241:00 +CPU=  6.446
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000240
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000240
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000228
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000228
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000228
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424_9.zip
Uploading the intermediate file: upload_file_9.zip
  22:44:16 STEP  242 H= 242:00 +CPU=  6.931
  22:44:26 STEP  243 H= 243:00 +CPU= 10.415
  22:44:33 STEP  244 H= 244:00 +CPU=  6.983
  22:44:40 STEP  245 H= 245:00 +CPU=  6.389
  22:44:51 STEP  246 H= 246:00 +CPU= 10.792
  22:44:57 STEP  247 H= 247:00 +CPU=  6.660
  22:45:04 STEP  248 H= 248:00 +CPU=  6.892
  22:45:15 STEP  249 H= 249:00 +CPU= 10.390
  22:45:22 STEP  250 H= 250:00 +CPU=  7.321
  22:45:29 STEP  251 H= 251:00 +CPU=  6.457
  22:45:42 STEP  252 H= 252:00 +CPU= 11.911
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000252
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000252
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000252
  22:45:49 STEP  253 H= 253:00 +CPU=  6.684
  22:45:56 STEP  254 H= 254:00 +CPU=  6.826
  22:46:06 STEP  255 H= 255:00 +CPU= 10.666
  22:46:13 STEP  256 H= 256:00 +CPU=  7.182
  22:46:20 STEP  257 H= 257:00 +CPU=  6.518
  22:46:31 STEP  258 H= 258:00 +CPU= 10.633
  22:46:37 STEP  259 H= 259:00 +CPU=  6.492
  22:46:44 STEP  260 H= 260:00 +CPU=  6.586
  22:46:55 STEP  261 H= 261:00 +CPU= 10.773
  22:47:01 STEP  262 H= 262:00 +CPU=  6.629
  22:47:08 STEP  263 H= 263:00 +CPU=  6.900
  22:47:22 STEP  264 H= 264:00 +CPU= 12.673
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000264
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000264
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000252
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000252
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000252
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424_10.zip
Uploading the intermediate file: upload_file_10.zip
  22:47:29 STEP  265 H= 265:00 +CPU=  6.650
  22:47:35 STEP  266 H= 266:00 +CPU=  6.509
  22:47:46 STEP  267 H= 267:00 +CPU= 10.547
  22:47:53 STEP  268 H= 268:00 +CPU=  6.751
  22:48:00 STEP  269 H= 269:00 +CPU=  6.693
  22:48:10 STEP  270 H= 270:00 +CPU= 10.540
  22:48:17 STEP  271 H= 271:00 +CPU=  6.729
  22:48:24 STEP  272 H= 272:00 +CPU=  6.512
  22:48:34 STEP  273 H= 273:00 +CPU= 10.749
  22:48:41 STEP  274 H= 274:00 +CPU=  6.846
  22:48:48 STEP  275 H= 275:00 +CPU=  6.564
  22:49:00 STEP  276 H= 276:00 +CPU= 11.589
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000276
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000276
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000276
  22:49:06 STEP  277 H= 277:00 +CPU=  6.624
  22:49:13 STEP  278 H= 278:00 +CPU=  6.666
  22:49:23 STEP  279 H= 279:00 +CPU= 10.406
  22:49:30 STEP  280 H= 280:00 +CPU=  6.836
  22:49:37 STEP  281 H= 281:00 +CPU=  6.472
  22:49:48 STEP  282 H= 282:00 +CPU= 10.918
  22:49:54 STEP  283 H= 283:00 +CPU=  6.342
  22:50:01 STEP  284 H= 284:00 +CPU=  6.626
  22:50:11 STEP  285 H= 285:00 +CPU= 10.710
  22:50:18 STEP  286 H= 286:00 +CPU=  6.550
  22:50:24 STEP  287 H= 287:00 +CPU=  6.614
  22:50:37 STEP  288 H= 288:00 +CPU= 12.463
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000288
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000288
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMGGhq0f+000276
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMSHhq0f+000276
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199667/ICMUAhq0f+000276
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0023_2014050100_123_983_12199667_1_r1439408424_11.zip
Uploading the intermediate file: upload_file_11.zip
  22:50:44 STEP  289 H= 289:00 +CPU=  6.929
  22:50:51 STEP  290 H= 290:00 +CPU=  6.720
  22:51:02 STEP  291 H= 291:00 +CPU= 10.676
Uploading trickle at timestep: 1036800
  22:51:09 STEP  292 H= 292:00 +CPU=  6.596
  22:51:16 STEP  293 H= 293:00 +CPU=  6.839
  22:51:27 STEP  294 H= 294:00 +CPU= 10.912
  22:51:42 STEP  295 H= 295:00 +CPU=  7.666
  22:51:49 STEP  296 H= 296:00 +CPU=  6.795
  22:52:05 STEP  297 H= 297:00 +CPU= 10.689
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:20:12 1538137 22295072 oifs_43r3_ps_0023_2014050100_123_983_12199667_1 1,036,800 2,497 0.0024


©2024 climateprediction.net