climateprediction.net home page
Task 22281437

Task 22281437

Name oifs_43r3_ps_0867_2009050100_123_978_12195511_0
Workunit 12195511
Created 23 Dec 2022, 22:10:19 UTC
Sent 4 Jan 2023, 12:54:19 UTC
Report deadline 3 Feb 2023, 12:54:19 UTC
Received 11 Jan 2023, 13:46:13 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1536947
Run time 17 hours 13 min 36 sec
CPU time 16 hours 4 min 41 sec
Validate state Invalid
Credit 7,378.56
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,790.96 MB
Peak swap size 4,978.16 MB
Peak disk usage 7,735.20 MB
Stderr 
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
 BOINC client, suspending the child process
Resuming the child process
  16:31:01 STEP 1589 H=1589:00 +CPU= 19.730
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:31:37 STEP 1590 H=1590:00 +CPU= 34.426
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:32:02 STEP 1591 H=1591:00 +CPU= 20.619
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:32:26 STEP 1592 H=1592:00 +CPU= 21.535
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:33:05 STEP 1593 H=1593:00 +CPU= 35.318
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:33:28 STEP 1594 H=1594:00 +CPU= 20.911
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:33:51 STEP 1595 H=1595:00 +CPU= 20.790
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:35:03 STEP 1596 H=1596:00 +CPU= 39.180
Moving to projects directory: /root/slots/3/ICMGGhq0f+001596
Moving to projects directory: /root/slots/3/ICMSHhq0f+001596
Moving to projects directory: /root/slots/3/ICMUAhq0f+001596
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:35:27 STEP 1597 H=1597:00 +CPU= 21.072
  16:35:50 STEP 1598 H=1598:00 +CPU= 22.480
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:26 STEP 1599 H=1599:00 +CPU= 31.450
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:49 STEP 1600 H=1600:00 +CPU= 20.059
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:37:11 STEP 1601 H=1601:00 +CPU= 19.178
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:37:45 STEP 1602 H=1602:00 +CPU= 30.883
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:38:09 STEP 1603 H=1603:00 +CPU= 21.242
Suspend request received from the BOINC client, suspending the child process
Quit request received from the BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2009050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0867
(argv4) batchid: 978
(argv5) wuid: 12195511
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/3
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12195511
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9e67385a135d8d0dbbb2827b2e04e127 to: /root/slots/3/oifs_43r3_ps_0867_2009050100_123_978_12195511.zip
Unzipping the namelist zip file: /root/slots/3/oifs_43r3_ps_0867_2009050100_123_978_12195511.zip
ic_ancil_file: ic_ancil_12195511
ifsdata_file: ifsdata_12195511
climate_data_file: clim_data_12195511
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_bc4ad4a7299b896f48adb150ab81a734 to: /root/slots/3/ic_ancil_12195511.zip
Unzipping the IC ancils zip file: /root/slots/3/ic_ancil_12195511.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3f84a125fdf8c2af96bffb3434f4138e to: /root/slots/3/ifsdata/ifsdata_12195511.zip
Unzipping the ifsdata_zip file: /root/slots/3/ifsdata/ifsdata_12195511.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/3/159l_2/clim_data_12195511.zip
Unzipping the climate data zip file: /root/slots/3/159l_2/clim_data_12195511.zip
Checking for progress XML file: /root/slots/3/progress_file_12195511.xml
Opened progress file ok : /root/slots/3/progress_file_12195511.xml
-- Model is restarting --
Adjusting last_iter, 1604, to previous model restart step.
Creating progress file: /root/slots/3/progress_file_12195511.xml
last_cpu_time: 54084
upload_file_number: 66
last_iter: 1583
last_upload: 5702400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0867_2009050100_123_978_12195511_0_r1622059607
The child process has been launched with process id: 7668
Executing the command: /root/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7668:7668] [20230109:161343:1673277223:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:14:29 STEP 1584 H=1584:00 +CPU= 37.427
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:14:56 STEP 1585 H=1585:00 +CPU= 23.568
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:15:25 STEP 1586 H=1586:00 +CPU= 22.896
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:16:04 STEP 1587 H=1587:00 +CPU= 34.709
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:16:30 STEP 1588 H=1588:00 +CPU= 22.436
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:17:22 STEP 1589 H=1589:00 +CPU= 23.393
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:18:07 STEP 1590 H=1590:00 +CPU= 38.392
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:18:51 STEP 1591 H=1591:00 +CPU= 24.258
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:19:53 STEP 1592 H=1592:00 +CPU= 25.735
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:20:51 STEP 1593 H=1593:00 +CPU= 36.068
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:21:16 STEP 1594 H=1594:00 +CPU= 22.832
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:21:42 STEP 1595 H=1595:00 +CPU= 22.432
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:22:27 STEP 1596 H=1596:00 +CPU= 38.553
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/3/ICMGGhq0f+001596
Moving to projects directory: /root/slots/3/ICMSHhq0f+001596
Moving to projects directory: /root/slots/3/ICMUAhq0f+001596
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:23:06 STEP 1597 H=1597:00 +CPU= 22.656
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:23:51 STEP 1598 H=1598:00 +CPU= 24.772
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:24:34 STEP 1599 H=1599:00 +CPU= 36.445
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:25:17 STEP 1600 H=1600:00 +CPU= 24.653
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:25:58 STEP 1601 H=1601:00 +CPU= 25.508
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:27:01 STEP 1602 H=1602:00 +CPU= 40.878
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:27:54 STEP 1603 H=1603:00 +CPU= 25.622
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:28:38 STEP 1604 H=1604:00 +CPU= 26.587
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:30:01 STEP 1605 H=1605:00 +CPU= 40.591
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:30:42 STEP 1606 H=1606:00 +CPU= 26.254
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:31:15 STEP 1607 H=1607:00 +CPU= 26.297
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:32:44 STEP 1608 H=1608:00 +CPU= 45.805
Moving to projects directory: /root/slots/3/ICMGGhq0f+001608
Moving to projects directory: /root/slots/3/ICMSHhq0f+001608
Moving to projects directory: /root/slots/3/ICMUAhq0f+001608
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001584
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001584
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001584
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001596
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001596
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001596
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0867_2009050100_123_978_12195511_0_r1622059607_66.zip
Uploading the intermediate file: upload_file_66.zip
  16:33:07 STEP 1609 H=1609:00 +CPU= 21.539
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:33:37 STEP 1610 H=1610:00 +CPU= 24.304
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:34:14 STEP 1611 H=1611:00 +CPU= 31.151
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:34:36 STEP 1612 H=1612:00 +CPU= 19.608
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:34:59 STEP 1613 H=1613:00 +CPU= 19.615
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:35:36 STEP 1614 H=1614:00 +CPU= 31.443
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:35:59 STEP 1615 H=1615:00 +CPU= 19.264
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:22 STEP 1616 H=1616:00 +CPU= 19.926
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:59 STEP 1617 H=1617:00 +CPU= 31.760
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:37:22 STEP 1618 H=1618:00 +CPU= 19.780
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:37:40 STEP 1619 H=1619:00 +CPU= 16.164
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:38:13 STEP 1620 H=1620:00 +CPU= 28.833
Moving to projects directory: /root/slots/3/ICMGGhq0f+001620
Moving to projects directory: /root/slots/3/ICMSHhq0f+001620
Moving to projects directory: /root/slots/3/ICMUAhq0f+001620
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:38:31 STEP 1621 H=1621:00 +CPU= 16.135
  16:38:48 STEP 1622 H=1622:00 +CPU= 16.646
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:39:19 STEP 1623 H=1623:00 +CPU= 26.209
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:39:37 STEP 1624 H=1624:00 +CPU= 16.663
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:39:55 STEP 1625 H=1625:00 +CPU= 15.979
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:40:24 STEP 1626 H=1626:00 +CPU= 26.549
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:40:42 STEP 1627 H=1627:00 +CPU= 16.030
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:41:01 STEP 1628 H=1628:00 +CPU= 16.571
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:41:31 STEP 1629 H=1629:00 +CPU= 26.048
  16:41:48 STEP 1630 H=1630:00 +CPU= 16.487
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:42:06 STEP 1631 H=1631:00 +CPU= 16.395
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:42:41 STEP 1632 H=1632:00 +CPU= 29.906
Moving to projects directory: /root/slots/3/ICMGGhq0f+001632
Moving to projects directory: /root/slots/3/ICMSHhq0f+001632
Moving to projects directory: /root/slots/3/ICMUAhq0f+001632
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001608
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001608
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001608
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001620
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001620
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001620
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0867_2009050100_123_978_12195511_0_r1622059607_67.zip
Uploading the intermediate file: upload_file_67.zip
  16:42:57 STEP 1633 H=1633:00 +CPU= 15.941
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:43:15 STEP 1634 H=1634:00 +CPU= 16.648
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:43:43 STEP 1635 H=1635:00 +CPU= 25.955
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:44:02 STEP 1636 H=1636:00 +CPU= 16.656
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:44:20 STEP 1637 H=1637:00 +CPU= 15.973
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:44:51 STEP 1638 H=1638:00 +CPU= 26.581
  16:45:07 STEP 1639 H=1639:00 +CPU= 15.891
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:45:26 STEP 1640 H=1640:00 +CPU= 16.371
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:45:56 STEP 1641 H=1641:00 +CPU= 25.973
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:46:14 STEP 1642 H=1642:00 +CPU= 16.550
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:46:32 STEP 1643 H=1643:00 +CPU= 15.901
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:47:03 STEP 1644 H=1644:00 +CPU= 28.509
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/3/ICMGGhq0f+001644
Moving to projects directory: /root/slots/3/ICMSHhq0f+001644
Moving to projects directory: /root/slots/3/ICMUAhq0f+001644
  16:47:21 STEP 1645 H=1645:00 +CPU= 15.904
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:47:40 STEP 1646 H=1646:00 +CPU= 16.531
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:48:08 STEP 1647 H=1647:00 +CPU= 25.836
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:48:26 STEP 1648 H=1648:00 +CPU= 16.584
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:48:44 STEP 1649 H=1649:00 +CPU= 15.884
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:49:15 STEP 1650 H=1650:00 +CPU= 26.468
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:49:33 STEP 1651 H=1651:00 +CPU= 16.020
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:49:52 STEP 1652 H=1652:00 +CPU= 16.444
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:50:20 STEP 1653 H=1653:00 +CPU= 25.845
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:50:38 STEP 1654 H=1654:00 +CPU= 16.436
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:50:57 STEP 1655 H=1655:00 +CPU= 16.723
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:51:31 STEP 1656 H=1656:00 +CPU= 30.081
Moving to projects directory: /root/slots/3/ICMGGhq0f+001656
Moving to projects directory: /root/slots/3/ICMSHhq0f+001656
Moving to projects directory: /root/slots/3/ICMUAhq0f+001656
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001632
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001632
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001632
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001644
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001644
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001644
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0867_2009050100_123_978_12195511_0_r1622059607_68.zip
Uploading the intermediate file: upload_file_68.zip
  16:51:47 STEP 1657 H=1657:00 +CPU= 15.980
  16:52:04 STEP 1658 H=1658:00 +CPU= 16.479
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:52:34 STEP 1659 H=1659:00 +CPU= 25.795
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:52:53 STEP 1660 H=1660:00 +CPU= 16.456
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:53:11 STEP 1661 H=1661:00 +CPU= 16.225
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:53:40 STEP 1662 H=1662:00 +CPU= 26.916
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:53:58 STEP 1663 H=1663:00 +CPU= 15.860
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:54:17 STEP 1664 H=1664:00 +CPU= 16.976
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:54:45 STEP 1665 H=1665:00 +CPU= 26.103
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:55:04 STEP 1666 H=1666:00 +CPU= 16.434
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:55:22 STEP 1667 H=1667:00 +CPU= 15.914
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:55:55 STEP 1668 H=1668:00 +CPU= 28.553
Moving to projects directory: /root/slots/3/ICMGGhq0f+001668
Moving to projects directory: /root/slots/3/ICMSHhq0f+001668
Moving to projects directory: /root/slots/3/ICMUAhq0f+001668
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:56:13 STEP 1669 H=1669:00 +CPU= 16.034
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:56:31 STEP 1670 H=1670:00 +CPU= 16.534
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:57:00 STEP 1671 H=1671:00 +CPU= 26.135
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:57:18 STEP 1672 H=1672:00 +CPU= 16.496
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:57:36 STEP 1673 H=1673:00 +CPU= 15.891
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:58:05 STEP 1674 H=1674:00 +CPU= 26.709
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:58:23 STEP 1675 H=1675:00 +CPU= 15.994
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:58:42 STEP 1676 H=1676:00 +CPU= 16.407
  16:59:08 STEP 1677 H=1677:00 +CPU= 26.249
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:59:27 STEP 1678 H=1678:00 +CPU= 16.679
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:59:46 STEP 1679 H=1679:00 +CPU= 16.578
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:00:18 STEP 1680 H=1680:00 +CPU= 30.127
Moving to projects directory: /root/slots/3/ICMGGhq0f+001680
Moving to projects directory: /root/slots/3/ICMSHhq0f+001680
Moving to projects directory: /root/slots/3/ICMUAhq0f+001680
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001656
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001656
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001656
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001668
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001668
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001668
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0867_2009050100_123_978_12195511_0_r1622059607_69.zip
Uploading the intermediate file: upload_file_69.zip
  17:00:34 STEP 1681 H=1681:00 +CPU= 15.891
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:00:53 STEP 1682 H=1682:00 +CPU= 16.672
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:01:26 STEP 1683 H=1683:00 +CPU= 29.690
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:03:06 STEP 1684 H=1684:00 +CPU= 20.433
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:04:03 STEP 1685 H=1685:00 +CPU= 18.480
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:06:09 STEP 1686 H=1686:00 +CPU= 30.909
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:07:36 STEP 1687 H=1687:00 +CPU= 18.543
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:09:21 STEP 1688 H=1688:00 +CPU= 18.757
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:09:56 STEP 1689 H=1689:00 +CPU= 29.184
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:10:37 STEP 1690 H=1690:00 +CPU= 20.022
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:11:33 STEP 1691 H=1691:00 +CPU= 18.421
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:12:34 STEP 1692 H=1692:00 +CPU= 32.970
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/3/ICMGGhq0f+001692
Moving to projects directory: /root/slots/3/ICMSHhq0f+001692
Moving to projects directory: /root/slots/3/ICMUAhq0f+001692
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:13:33 STEP 1693 H=1693:00 +CPU= 18.650
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:13:55 STEP 1694 H=1694:00 +CPU= 18.863
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:15:03 STEP 1695 H=1695:00 +CPU= 30.394
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:15:34 STEP 1696 H=1696:00 +CPU= 18.985
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:15:54 STEP 1697 H=1697:00 +CPU= 17.933
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:16:39 STEP 1698 H=1698:00 +CPU= 29.962
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:17:00 STEP 1699 H=1699:00 +CPU= 18.171
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:17:21 STEP 1700 H=1700:00 +CPU= 18.461
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:17:56 STEP 1701 H=1701:00 +CPU= 29.422
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:18:17 STEP 1702 H=1702:00 +CPU= 18.761
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:18:38 STEP 1703 H=1703:00 +CPU= 18.434
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:19:18 STEP 1704 H=1704:00 +CPU= 33.892
Moving to projects directory: /root/slots/3/ICMGGhq0f+001704
Moving to projects directory: /root/slots/3/ICMSHhq0f+001704
Moving to projects directory: /root/slots/3/ICMUAhq0f+001704
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001680
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001680
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001680
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001692
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001692
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001692
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0867_2009050100_123_978_12195511_0_r1622059607_70.zip
Uploading the intermediate file: upload_file_70.zip
  17:19:37 STEP 1705 H=1705:00 +CPU= 17.886
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:19:58 STEP 1706 H=1706:00 +CPU= 18.341
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:20:32 STEP 1707 H=1707:00 +CPU= 29.437
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:20:54 STEP 1708 H=1708:00 +CPU= 18.875
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:21:15 STEP 1709 H=1709:00 +CPU= 17.959
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:21:51 STEP 1710 H=1710:00 +CPU= 30.220
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:22:12 STEP 1711 H=1711:00 +CPU= 18.202
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:22:33 STEP 1712 H=1712:00 +CPU= 18.643
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:23:06 STEP 1713 H=1713:00 +CPU= 29.772
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:23:38 STEP 1714 H=1714:00 +CPU= 18.723
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:23:58 STEP 1715 H=1715:00 +CPU= 17.880
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:24:36 STEP 1716 H=1716:00 +CPU= 32.456
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/3/ICMGGhq0f+001716
Moving to projects directory: /root/slots/3/ICMSHhq0f+001716
Moving to projects directory: /root/slots/3/ICMUAhq0f+001716
  17:24:57 STEP 1717 H=1717:00 +CPU= 17.997
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:25:16 STEP 1718 H=1718:00 +CPU= 16.421
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:25:46 STEP 1719 H=1719:00 +CPU= 26.187
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:26:05 STEP 1720 H=1720:00 +CPU= 16.607
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:26:23 STEP 1721 H=1721:00 +CPU= 15.827
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:26:52 STEP 1722 H=1722:00 +CPU= 26.833
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:27:10 STEP 1723 H=1723:00 +CPU= 16.130
  17:27:27 STEP 1724 H=1724:00 +CPU= 16.335
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:27:55 STEP 1725 H=1725:00 +CPU= 26.054
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:28:14 STEP 1726 H=1726:00 +CPU= 16.527
  17:28:30 STEP 1727 H=1727:00 +CPU= 16.695
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2009050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0867
(argv4) batchid: 978
(argv5) wuid: 12195511
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/3
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12195511
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9e67385a135d8d0dbbb2827b2e04e127 to: /root/slots/3/oifs_43r3_ps_0867_2009050100_123_978_12195511.zip
Unzipping the namelist zip file: /root/slots/3/oifs_43r3_ps_0867_2009050100_123_978_12195511.zip
ic_ancil_file: ic_ancil_12195511
ifsdata_file: ifsdata_12195511
climate_data_file: clim_data_12195511
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_bc4ad4a7299b896f48adb150ab81a734 to: /root/slots/3/ic_ancil_12195511.zip
Unzipping the IC ancils zip file: /root/slots/3/ic_ancil_12195511.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3f84a125fdf8c2af96bffb3434f4138e to: /root/slots/3/ifsdata/ifsdata_12195511.zip
Unzipping the ifsdata_zip file: /root/slots/3/ifsdata/ifsdata_12195511.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/3/159l_2/clim_data_12195511.zip
Unzipping the climate data zip file: /root/slots/3/159l_2/clim_data_12195511.zip
Checking for progress XML file: /root/slots/3/progress_file_12195511.xml
Opened progress file ok : /root/slots/3/progress_file_12195511.xml
-- Model is restarting --
Adjusting last_iter, 1728, to previous model restart step.
Creating progress file: /root/slots/3/progress_file_12195511.xml
last_cpu_time: 57353
upload_file_number: 71
last_iter: 1727
last_upload: 6134400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0867_2009050100_123_978_12195511_0_r1622059607
The child process has been launched with process id: 8107
Executing the command: /root/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:8107:8107] [20230111:143335:1673444015:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:34:11 STEP 1728 H=1728:00 +CPU= 28.746
Moving to projects directory: /root/slots/3/ICMGGhq0f+001728
Moving to projects directory: /root/slots/3/ICMSHhq0f+001728
Moving to projects directory: /root/slots/3/ICMUAhq0f+001728
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001704
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001704
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001704
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001716
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001716
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001716
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0867_2009050100_123_978_12195511_0_r1622059607_71.zip
Uploading the intermediate file: upload_file_71.zip
  14:34:27 STEP 1729 H=1729:00 +CPU= 15.818
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:34:45 STEP 1730 H=1730:00 +CPU= 16.259
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:35:13 STEP 1731 H=1731:00 +CPU= 25.553
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:35:31 STEP 1732 H=1732:00 +CPU= 16.337
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:35:49 STEP 1733 H=1733:00 +CPU= 15.741
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:36:19 STEP 1734 H=1734:00 +CPU= 27.562
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:36:38 STEP 1735 H=1735:00 +CPU= 16.736
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:36:57 STEP 1736 H=1736:00 +CPU= 16.480
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:37:27 STEP 1737 H=1737:00 +CPU= 25.878
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:37:45 STEP 1738 H=1738:00 +CPU= 16.386
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:38:03 STEP 1739 H=1739:00 +CPU= 16.164
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:38:34 STEP 1740 H=1740:00 +CPU= 28.752
Moving to projects directory: /root/slots/3/ICMGGhq0f+001740
Moving to projects directory: /root/slots/3/ICMSHhq0f+001740
Moving to projects directory: /root/slots/3/ICMUAhq0f+001740
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:38:52 STEP 1741 H=1741:00 +CPU= 16.114
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:39:11 STEP 1742 H=1742:00 +CPU= 16.477
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:39:39 STEP 1743 H=1743:00 +CPU= 25.777
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:39:57 STEP 1744 H=1744:00 +CPU= 16.496
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:40:15 STEP 1745 H=1745:00 +CPU= 16.053
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:40:46 STEP 1746 H=1746:00 +CPU= 26.234
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:41:04 STEP 1747 H=1747:00 +CPU= 15.916
  14:41:20 STEP 1748 H=1748:00 +CPU= 16.430
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:41:50 STEP 1749 H=1749:00 +CPU= 25.795
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:42:08 STEP 1750 H=1750:00 +CPU= 16.320
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:42:27 STEP 1751 H=1751:00 +CPU= 16.264
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:42:58 STEP 1752 H=1752:00 +CPU= 29.573
Moving to projects directory: /root/slots/3/ICMGGhq0f+001752
Moving to projects directory: /root/slots/3/ICMSHhq0f+001752
Moving to projects directory: /root/slots/3/ICMUAhq0f+001752
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001728
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001728
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001728
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMGGhq0f+001740
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMSHhq0f+001740
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12195511/ICMUAhq0f+001740
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0867_2009050100_123_978_12195511_0_r1622059607_72.zip
Uploading the intermediate file: upload_file_72.zip
  14:43:14 STEP 1753 H=1753:00 +CPU= 15.111
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:12:49 1536947 22281437 oifs_43r3_ps_0867_2009050100_123_978_12195511_0 5,270,400 50,344 0.0096
09 May 2023 16:11:49 1536947 22281437 oifs_43r3_ps_0867_2009050100_123_978_12195511_0 4,147,200 41,351 0.0100
09 May 2023 16:09:21 1536947 22281437 oifs_43r3_ps_0867_2009050100_123_978_12195511_0 2,678,400 27,959 0.0104
09 May 2023 16:09:13 1536947 22281437 oifs_43r3_ps_0867_2009050100_123_978_12195511_0 1,814,400 18,968 0.0105


©2024 climateprediction.net