climateprediction.net home page
Task 22315826

Task 22315826

Name oifs_43r3_00sa_2019110100_123_993_12214456_0
Workunit 12214456
Created 24 Feb 2023, 15:04:22 UTC
Sent 24 Feb 2023, 16:03:52 UTC
Report deadline 25 Apr 2023, 16:03:52 UTC
Received 5 Mar 2023, 18:14:58 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537223
Run time 19 hours 14 min 38 sec
CPU time 19 hours 14 min 38 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 5.75 GFLOPS
Application version OpenIFS 43r3 v1.21
x86_64-pc-linux-gnu
Peak working set size 4,423.72 MB
Peak swap size 4,983.85 MB
Peak disk usage 1,540.19 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
3:49 STEP 2543 H=2543:00 +CPU= 14.164
  22:14:15 STEP 2544 H=2544:00 +CPU= 25.404
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002544
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002544
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_105.zip
Uploading the intermediate file: upload_file_105.zip
  22:14:28 STEP 2545 H=2545:00 +CPU= 13.816
  22:14:43 STEP 2546 H=2546:00 +CPU= 14.199
  22:15:05 STEP 2547 H=2547:00 +CPU= 21.953
  22:15:19 STEP 2548 H=2548:00 +CPU= 14.175
  22:15:33 STEP 2549 H=2549:00 +CPU= 13.839
  22:15:55 STEP 2550 H=2550:00 +CPU= 22.368
  22:16:09 STEP 2551 H=2551:00 +CPU= 13.985
  22:16:23 STEP 2552 H=2552:00 +CPU= 14.321
  22:16:45 STEP 2553 H=2553:00 +CPU= 22.101
  22:17:00 STEP 2554 H=2554:00 +CPU= 14.346
  22:17:14 STEP 2555 H=2555:00 +CPU= 13.931
  22:17:38 STEP 2556 H=2556:00 +CPU= 23.982
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002556
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002556
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002556
  22:17:52 STEP 2557 H=2557:00 +CPU= 13.890
  22:18:06 STEP 2558 H=2558:00 +CPU= 14.310
  22:18:28 STEP 2559 H=2559:00 +CPU= 21.986
  22:18:42 STEP 2560 H=2560:00 +CPU= 14.318
  22:18:56 STEP 2561 H=2561:00 +CPU= 13.901
  22:19:19 STEP 2562 H=2562:00 +CPU= 22.476
  22:19:33 STEP 2563 H=2563:00 +CPU= 14.015
  22:19:47 STEP 2564 H=2564:00 +CPU= 14.217
  22:20:09 STEP 2565 H=2565:00 +CPU= 21.956
  22:20:23 STEP 2566 H=2566:00 +CPU= 14.215
  22:20:37 STEP 2567 H=2567:00 +CPU= 14.250
  22:21:03 STEP 2568 H=2568:00 +CPU= 25.506
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002568
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002568
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_106.zip
Uploading the intermediate file: upload_file_106.zip
  22:21:17 STEP 2569 H=2569:00 +CPU= 13.935
  22:21:31 STEP 2570 H=2570:00 +CPU= 14.370
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3539MB (heap), 4416MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x8102d0 (old at (nil))
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3843] 3539 MB (maxheap), 4416 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :            CPG_DRV 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :             >OMP-CPG                  (1025) 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :              CPG 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :               CPG_DYN 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :                LACDYN 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :                 LASSIE 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :                  SITNU 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :                   VERINT 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222132:1677878492:11825.840] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :                    VERINT_DGEMM_2 
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222133:1677878493:11826.840] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:AMD5700GMint:1:1:1964:1964] [20230303:222133:1677878493:11826.840] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8440 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE89FB  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000810F01  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC8950  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008E4C13  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000A3A0AD  verint_                   168  verint.F90
oifs_43r3_model.e  0000000000C50008  sitnu_                    133  sitnu.F90
oifs_43r3_model.e  0000000000E070FD  lassie_                   140  lassie.F90
oifs_43r3_model.e  0000000000E00799  lacdyn_                   301  lacdyn.F90
oifs_43r3_model.e  0000000000DBE4F2  cpg_dyn_                  290  cpg_dyn.F90
oifs_43r3_model.e  0000000000D58CDE  cpg_                      777  cpg.F90
oifs_43r3_model.e  0000000000D557D8  cpg_drv_                  351  cpg_drv.F90
oifs_43r3_model.e  0000000000C8D578  gp_model_                 419  gp_model.F90
oifs_43r3_model.e  000000000138E0F2  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  000000000139C38E  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  00000000004D61C4  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004339DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  000000000041D979  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  000000000041C8F2  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000407BD8  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040734D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9C50  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 00sa
(argv4) batchid: 993
(argv5) wuid: 12214456
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_eb2e76420391c0967d6eb4f4aa973eab to: /var/lib/boinc-client/slots/4/oifs_43r3_00sa_2019110100_123_993_12214456.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_00sa_2019110100_123_993_12214456.zip
ic_ancil_file: ic_ancil_12214456
ifsdata_file: ifsdata_12214456
climate_data_file: clim_data_12214456
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_dec701a483f2d997c6243a0d01080bd4 to: /var/lib/boinc-client/slots/4/ic_ancil_12214456.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12214456.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8714b05c30b83c0c7d609ba135ddca6c to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12214456.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12214456.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12214456.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12214456.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12214456.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12214456.xml
-- Model is restarting --
Adjusting last_iter, 2568, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12214456.xml
last_cpu_time: 62813
upload_file_number: 106
last_iter: 2569
last_upload: 9158400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906
The child process has been launched with process id: 1938
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:AMD5700GMint:1:1:1938:1938] [20230304:201202:1677957122:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002568
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_106.zip
Uploading the intermediate file: upload_file_106.zip
  20:12:26 STEP 2568 H=2568:00 +CPU= 24.790
  20:12:40 STEP 2569 H=2569:00 +CPU= 13.937
  20:12:54 STEP 2570 H=2570:00 +CPU= 14.185
  20:13:16 STEP 2571 H=2571:00 +CPU= 21.768
  20:13:30 STEP 2572 H=2572:00 +CPU= 14.050
  20:13:44 STEP 2573 H=2573:00 +CPU= 13.697
  20:14:06 STEP 2574 H=2574:00 +CPU= 22.130
  20:14:20 STEP 2575 H=2575:00 +CPU= 13.712
  20:14:34 STEP 2576 H=2576:00 +CPU= 14.026
  20:14:56 STEP 2577 H=2577:00 +CPU= 21.709
  20:15:10 STEP 2578 H=2578:00 +CPU= 14.100
  20:15:23 STEP 2579 H=2579:00 +CPU= 13.749
  20:15:47 STEP 2580 H=2580:00 +CPU= 23.612
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002580
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002580
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002580
  20:16:01 STEP 2581 H=2581:00 +CPU= 13.803
  20:16:15 STEP 2582 H=2582:00 +CPU= 14.182
  20:16:37 STEP 2583 H=2583:00 +CPU= 21.769
  20:16:51 STEP 2584 H=2584:00 +CPU= 14.180
  20:17:05 STEP 2585 H=2585:00 +CPU= 13.931
  20:17:27 STEP 2586 H=2586:00 +CPU= 22.191
  20:17:41 STEP 2587 H=2587:00 +CPU= 13.752
  20:17:55 STEP 2588 H=2588:00 +CPU= 14.132
  20:18:17 STEP 2589 H=2589:00 +CPU= 21.814
  20:18:31 STEP 2590 H=2590:00 +CPU= 14.151
  20:18:45 STEP 2591 H=2591:00 +CPU= 14.175
  20:19:10 STEP 2592 H=2592:00 +CPU= 25.180
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002592
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002592
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002592
  20:19:24 STEP 2593 H=2593:00 +CPU= 13.797
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002592
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_107.zip
Uploading the intermediate file: upload_file_107.zip
  20:19:38 STEP 2594 H=2594:00 +CPU= 14.179
  20:20:00 STEP 2595 H=2595:00 +CPU= 21.832
  20:20:14 STEP 2596 H=2596:00 +CPU= 14.107
  20:20:28 STEP 2597 H=2597:00 +CPU= 13.839
  20:20:51 STEP 2598 H=2598:00 +CPU= 22.335
  20:21:04 STEP 2599 H=2599:00 +CPU= 13.919
  20:21:19 STEP 2600 H=2600:00 +CPU= 14.206
  20:21:41 STEP 2601 H=2601:00 +CPU= 21.964
  20:21:55 STEP 2602 H=2602:00 +CPU= 14.165
  20:22:09 STEP 2603 H=2603:00 +CPU= 13.837
  20:22:33 STEP 2604 H=2604:00 +CPU= 23.942
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002604
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002604
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002604
  20:22:46 STEP 2605 H=2605:00 +CPU= 13.827
  20:23:01 STEP 2606 H=2606:00 +CPU= 14.198
  20:23:23 STEP 2607 H=2607:00 +CPU= 21.909
  20:23:37 STEP 2608 H=2608:00 +CPU= 14.204
  20:23:51 STEP 2609 H=2609:00 +CPU= 13.785
  20:24:13 STEP 2610 H=2610:00 +CPU= 22.268
  20:24:27 STEP 2611 H=2611:00 +CPU= 13.811
  20:24:41 STEP 2612 H=2612:00 +CPU= 14.144
  20:25:03 STEP 2613 H=2613:00 +CPU= 21.890
  20:25:17 STEP 2614 H=2614:00 +CPU= 14.135
  20:25:31 STEP 2615 H=2615:00 +CPU= 14.145
  20:25:56 STEP 2616 H=2616:00 +CPU= 25.406
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002616
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002616
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002616
  20:26:10 STEP 2617 H=2617:00 +CPU= 13.725
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002616
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_108.zip
Uploading the intermediate file: upload_file_108.zip
  20:26:24 STEP 2618 H=2618:00 +CPU= 14.096
  20:26:46 STEP 2619 H=2619:00 +CPU= 21.790
  20:27:00 STEP 2620 H=2620:00 +CPU= 14.202
  20:27:14 STEP 2621 H=2621:00 +CPU= 13.831
  20:27:36 STEP 2622 H=2622:00 +CPU= 22.310
  20:27:50 STEP 2623 H=2623:00 +CPU= 13.809
  20:28:04 STEP 2624 H=2624:00 +CPU= 14.147
  20:28:26 STEP 2625 H=2625:00 +CPU= 21.878
  20:28:40 STEP 2626 H=2626:00 +CPU= 14.172
  20:28:54 STEP 2627 H=2627:00 +CPU= 13.769
  20:29:18 STEP 2628 H=2628:00 +CPU= 23.799
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002628
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002628
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002628
  20:29:32 STEP 2629 H=2629:00 +CPU= 13.750
  20:29:46 STEP 2630 H=2630:00 +CPU= 14.092
  20:30:08 STEP 2631 H=2631:00 +CPU= 21.818
  20:30:22 STEP 2632 H=2632:00 +CPU= 14.153
  20:30:36 STEP 2633 H=2633:00 +CPU= 13.760
  20:30:58 STEP 2634 H=2634:00 +CPU= 22.205
  20:31:12 STEP 2635 H=2635:00 +CPU= 13.796
  20:31:26 STEP 2636 H=2636:00 +CPU= 14.108
  20:31:47 STEP 2637 H=2637:00 +CPU= 21.752
  20:32:02 STEP 2638 H=2638:00 +CPU= 14.134
  20:32:16 STEP 2639 H=2639:00 +CPU= 14.222
  20:32:41 STEP 2640 H=2640:00 +CPU= 25.242
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002640
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002640
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002640
  20:32:55 STEP 2641 H=2641:00 +CPU= 13.781
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002640
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_109.zip
Uploading the intermediate file: upload_file_109.zip
  20:33:09 STEP 2642 H=2642:00 +CPU= 14.166
  20:33:31 STEP 2643 H=2643:00 +CPU= 21.839
  20:33:45 STEP 2644 H=2644:00 +CPU= 14.255
  20:33:59 STEP 2645 H=2645:00 +CPU= 13.869
  20:34:21 STEP 2646 H=2646:00 +CPU= 22.253
  20:34:35 STEP 2647 H=2647:00 +CPU= 13.870
  20:34:50 STEP 2648 H=2648:00 +CPU= 14.295
  20:35:11 STEP 2649 H=2649:00 +CPU= 21.888
  20:35:26 STEP 2650 H=2650:00 +CPU= 14.202
  20:35:39 STEP 2651 H=2651:00 +CPU= 13.841
  20:36:03 STEP 2652 H=2652:00 +CPU= 23.802
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002652
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002652
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002652
  20:36:17 STEP 2653 H=2653:00 +CPU= 13.836
  20:36:31 STEP 2654 H=2654:00 +CPU= 14.179
  20:36:53 STEP 2655 H=2655:00 +CPU= 21.897
  20:37:07 STEP 2656 H=2656:00 +CPU= 14.178
  20:37:21 STEP 2657 H=2657:00 +CPU= 13.866
  20:37:44 STEP 2658 H=2658:00 +CPU= 22.266
  20:37:57 STEP 2659 H=2659:00 +CPU= 13.832
  20:38:12 STEP 2660 H=2660:00 +CPU= 14.169
  20:38:33 STEP 2661 H=2661:00 +CPU= 21.858
  20:38:48 STEP 2662 H=2662:00 +CPU= 14.166
  20:39:02 STEP 2663 H=2663:00 +CPU= 14.193
  20:39:27 STEP 2664 H=2664:00 +CPU= 25.431
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002664
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002664
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002664
  20:39:41 STEP 2665 H=2665:00 +CPU= 13.819
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002664
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_110.zip
Uploading the intermediate file: upload_file_110.zip
  20:39:55 STEP 2666 H=2666:00 +CPU= 14.204
  20:40:17 STEP 2667 H=2667:00 +CPU= 21.897
  20:40:31 STEP 2668 H=2668:00 +CPU= 14.228
  20:40:45 STEP 2669 H=2669:00 +CPU= 13.930
  20:41:08 STEP 2670 H=2670:00 +CPU= 22.425
  20:41:22 STEP 2671 H=2671:00 +CPU= 13.827
  20:41:36 STEP 2672 H=2672:00 +CPU= 14.193
  20:41:58 STEP 2673 H=2673:00 +CPU= 21.862
  20:42:12 STEP 2674 H=2674:00 +CPU= 14.060
  20:42:25 STEP 2675 H=2675:00 +CPU= 13.711
  20:42:49 STEP 2676 H=2676:00 +CPU= 23.696
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002676
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002676
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002676
  20:43:03 STEP 2677 H=2677:00 +CPU= 13.705
  20:43:17 STEP 2678 H=2678:00 +CPU= 14.030
  20:43:39 STEP 2679 H=2679:00 +CPU= 21.700
  20:43:53 STEP 2680 H=2680:00 +CPU= 14.059
  20:44:06 STEP 2681 H=2681:00 +CPU= 13.695
  20:44:28 STEP 2682 H=2682:00 +CPU= 22.125
  20:44:42 STEP 2683 H=2683:00 +CPU= 13.697
  20:44:56 STEP 2684 H=2684:00 +CPU= 14.011
  20:45:18 STEP 2685 H=2685:00 +CPU= 21.653
  20:45:32 STEP 2686 H=2686:00 +CPU= 14.070
  20:45:46 STEP 2687 H=2687:00 +CPU= 14.103
  20:46:11 STEP 2688 H=2688:00 +CPU= 25.192
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002688
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002688
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002688
  20:46:25 STEP 2689 H=2689:00 +CPU= 13.681
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002688
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_111.zip
Uploading the intermediate file: upload_file_111.zip
  20:46:39 STEP 2690 H=2690:00 +CPU= 14.008
  20:47:01 STEP 2691 H=2691:00 +CPU= 21.704
  20:47:15 STEP 2692 H=2692:00 +CPU= 14.080
  20:47:28 STEP 2693 H=2693:00 +CPU= 13.746
  20:47:51 STEP 2694 H=2694:00 +CPU= 22.196
  20:48:04 STEP 2695 H=2695:00 +CPU= 13.791
  20:48:19 STEP 2696 H=2696:00 +CPU= 14.127
  20:48:40 STEP 2697 H=2697:00 +CPU= 21.769
  20:48:54 STEP 2698 H=2698:00 +CPU= 14.075
  20:49:08 STEP 2699 H=2699:00 +CPU= 13.681
  20:49:32 STEP 2700 H=2700:00 +CPU= 23.743
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002700
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002700
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002700
  20:49:46 STEP 2701 H=2701:00 +CPU= 13.709
  20:50:00 STEP 2702 H=2702:00 +CPU= 14.135
  20:50:22 STEP 2703 H=2703:00 +CPU= 21.785
  20:50:36 STEP 2704 H=2704:00 +CPU= 14.157
  20:50:49 STEP 2705 H=2705:00 +CPU= 13.718
  20:51:12 STEP 2706 H=2706:00 +CPU= 22.143
  20:51:25 STEP 2707 H=2707:00 +CPU= 13.759
  20:51:39 STEP 2708 H=2708:00 +CPU= 14.092
  20:52:01 STEP 2709 H=2709:00 +CPU= 21.718
  20:52:15 STEP 2710 H=2710:00 +CPU= 14.081
  20:52:29 STEP 2711 H=2711:00 +CPU= 14.156
  20:52:55 STEP 2712 H=2712:00 +CPU= 25.378
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002712
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002712
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002712
  20:53:09 STEP 2713 H=2713:00 +CPU= 13.716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002712
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_112.zip
Uploading the intermediate file: upload_file_112.zip
  20:53:23 STEP 2714 H=2714:00 +CPU= 14.021
  20:53:44 STEP 2715 H=2715:00 +CPU= 21.814
  20:53:59 STEP 2716 H=2716:00 +CPU= 14.159
  20:54:12 STEP 2717 H=2717:00 +CPU= 13.819
  20:54:35 STEP 2718 H=2718:00 +CPU= 22.278
  20:54:48 STEP 2719 H=2719:00 +CPU= 13.762
  20:55:03 STEP 2720 H=2720:00 +CPU= 14.140
  20:55:24 STEP 2721 H=2721:00 +CPU= 21.837
  20:55:39 STEP 2722 H=2722:00 +CPU= 14.147
  20:55:52 STEP 2723 H=2723:00 +CPU= 13.770
  20:56:16 STEP 2724 H=2724:00 +CPU= 23.725
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002724
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002724
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002724
  20:56:30 STEP 2725 H=2725:00 +CPU= 13.718
  20:56:44 STEP 2726 H=2726:00 +CPU= 14.030
  20:57:06 STEP 2727 H=2727:00 +CPU= 21.739
  20:57:20 STEP 2728 H=2728:00 +CPU= 14.086
  20:57:33 STEP 2729 H=2729:00 +CPU= 13.697
  20:57:56 STEP 2730 H=2730:00 +CPU= 22.197
  20:58:09 STEP 2731 H=2731:00 +CPU= 13.667
  20:58:23 STEP 2732 H=2732:00 +CPU= 14.026
  20:58:45 STEP 2733 H=2733:00 +CPU= 21.698
  20:58:59 STEP 2734 H=2734:00 +CPU= 14.040
  20:59:13 STEP 2735 H=2735:00 +CPU= 14.099
  20:59:38 STEP 2736 H=2736:00 +CPU= 25.271
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002736
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002736
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002736
  20:59:52 STEP 2737 H=2737:00 +CPU= 13.672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002736
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_113.zip
Uploading the intermediate file: upload_file_113.zip
  21:00:06 STEP 2738 H=2738:00 +CPU= 14.064
  21:00:28 STEP 2739 H=2739:00 +CPU= 21.741
  21:00:42 STEP 2740 H=2740:00 +CPU= 14.200
  21:00:56 STEP 2741 H=2741:00 +CPU= 13.758
  21:01:18 STEP 2742 H=2742:00 +CPU= 22.287
  21:01:32 STEP 2743 H=2743:00 +CPU= 13.683
  21:01:46 STEP 2744 H=2744:00 +CPU= 14.108
  21:02:08 STEP 2745 H=2745:00 +CPU= 21.725
  21:02:22 STEP 2746 H=2746:00 +CPU= 14.065
  21:02:36 STEP 2747 H=2747:00 +CPU= 13.749
  21:02:59 STEP 2748 H=2748:00 +CPU= 23.734
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002748
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002748
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002748
  21:03:13 STEP 2749 H=2749:00 +CPU= 13.723
  21:03:27 STEP 2750 H=2750:00 +CPU= 14.095
  21:03:49 STEP 2751 H=2751:00 +CPU= 21.780
  21:04:03 STEP 2752 H=2752:00 +CPU= 14.166
  21:04:17 STEP 2753 H=2753:00 +CPU= 13.697
  21:04:39 STEP 2754 H=2754:00 +CPU= 22.151
  21:04:53 STEP 2755 H=2755:00 +CPU= 13.755
  21:05:07 STEP 2756 H=2756:00 +CPU= 14.060
  21:05:28 STEP 2757 H=2757:00 +CPU= 21.727
  21:05:43 STEP 2758 H=2758:00 +CPU= 14.060
  21:05:57 STEP 2759 H=2759:00 +CPU= 14.132
  21:06:22 STEP 2760 H=2760:00 +CPU= 25.326
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002760
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002760
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002760
  21:06:36 STEP 2761 H=2761:00 +CPU= 13.705
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002760
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_114.zip
Uploading the intermediate file: upload_file_114.zip
  21:06:50 STEP 2762 H=2762:00 +CPU= 14.063
  21:07:12 STEP 2763 H=2763:00 +CPU= 21.772
  21:07:26 STEP 2764 H=2764:00 +CPU= 14.169
  21:07:40 STEP 2765 H=2765:00 +CPU= 13.840
  21:08:02 STEP 2766 H=2766:00 +CPU= 22.253
  21:08:16 STEP 2767 H=2767:00 +CPU= 13.835
  21:08:30 STEP 2768 H=2768:00 +CPU= 14.145
  21:08:52 STEP 2769 H=2769:00 +CPU= 21.782
  21:09:06 STEP 2770 H=2770:00 +CPU= 14.100
  21:09:20 STEP 2771 H=2771:00 +CPU= 13.763
  21:09:43 STEP 2772 H=2772:00 +CPU= 23.678
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002772
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002772
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002772
  21:09:57 STEP 2773 H=2773:00 +CPU= 13.837
  21:10:11 STEP 2774 H=2774:00 +CPU= 14.101
  21:10:33 STEP 2775 H=2775:00 +CPU= 21.808
  21:10:47 STEP 2776 H=2776:00 +CPU= 14.011
  21:11:01 STEP 2777 H=2777:00 +CPU= 13.742
  21:11:23 STEP 2778 H=2778:00 +CPU= 22.144
  21:11:37 STEP 2779 H=2779:00 +CPU= 13.643
  21:11:51 STEP 2780 H=2780:00 +CPU= 14.043
  21:12:12 STEP 2781 H=2781:00 +CPU= 21.710
  21:12:26 STEP 2782 H=2782:00 +CPU= 14.021
  21:12:40 STEP 2783 H=2783:00 +CPU= 14.044
  21:13:06 STEP 2784 H=2784:00 +CPU= 25.193
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002784
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002784
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002784
  21:13:19 STEP 2785 H=2785:00 +CPU= 13.669
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002784
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_115.zip
Uploading the intermediate file: upload_file_115.zip
  21:13:33 STEP 2786 H=2786:00 +CPU= 14.007
Uploading trickle at timestep: 10026000
  21:13:55 STEP 2787 H=2787:00 +CPU= 21.703
  21:14:09 STEP 2788 H=2788:00 +CPU= 14.014
  21:14:23 STEP 2789 H=2789:00 +CPU= 13.733
  21:14:45 STEP 2790 H=2790:00 +CPU= 22.257
  21:14:59 STEP 2791 H=2791:00 +CPU= 13.746
  21:15:13 STEP 2792 H=2792:00 +CPU= 14.056
  21:15:35 STEP 2793 H=2793:00 +CPU= 21.752
  21:15:49 STEP 2794 H=2794:00 +CPU= 14.131
  21:16:02 STEP 2795 H=2795:00 +CPU= 13.721
  21:16:26 STEP 2796 H=2796:00 +CPU= 23.752
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002796
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002796
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002796
  21:16:40 STEP 2797 H=2797:00 +CPU= 13.758
  21:16:54 STEP 2798 H=2798:00 +CPU= 14.127
  21:17:16 STEP 2799 H=2799:00 +CPU= 21.855
  21:17:30 STEP 2800 H=2800:00 +CPU= 14.157
  21:17:44 STEP 2801 H=2801:00 +CPU= 13.719
  21:18:06 STEP 2802 H=2802:00 +CPU= 22.180
  21:18:20 STEP 2803 H=2803:00 +CPU= 13.784
  21:18:34 STEP 2804 H=2804:00 +CPU= 14.093
  21:18:56 STEP 2805 H=2805:00 +CPU= 21.887
  21:19:10 STEP 2806 H=2806:00 +CPU= 14.147
  21:19:24 STEP 2807 H=2807:00 +CPU= 14.177
  21:19:49 STEP 2808 H=2808:00 +CPU= 25.295
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002808
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002808
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002808
  21:20:03 STEP 2809 H=2809:00 +CPU= 13.713
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002808
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_116.zip
Uploading the intermediate file: upload_file_116.zip
  21:20:17 STEP 2810 H=2810:00 +CPU= 14.098
  21:20:39 STEP 2811 H=2811:00 +CPU= 21.917
  21:20:53 STEP 2812 H=2812:00 +CPU= 14.148
  21:21:07 STEP 2813 H=2813:00 +CPU= 13.740
  21:21:29 STEP 2814 H=2814:00 +CPU= 22.296
  21:21:43 STEP 2815 H=2815:00 +CPU= 13.772
  21:21:57 STEP 2816 H=2816:00 +CPU= 14.151
  21:22:19 STEP 2817 H=2817:00 +CPU= 21.843
  21:22:33 STEP 2818 H=2818:00 +CPU= 14.121
  21:22:47 STEP 2819 H=2819:00 +CPU= 13.777
  21:23:11 STEP 2820 H=2820:00 +CPU= 23.771
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002820
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002820
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002820
  21:23:25 STEP 2821 H=2821:00 +CPU= 13.753
  21:23:39 STEP 2822 H=2822:00 +CPU= 14.146
  21:24:00 STEP 2823 H=2823:00 +CPU= 21.785
  21:24:15 STEP 2824 H=2824:00 +CPU= 14.130
  21:24:28 STEP 2825 H=2825:00 +CPU= 13.765
  21:24:51 STEP 2826 H=2826:00 +CPU= 22.250
  21:25:04 STEP 2827 H=2827:00 +CPU= 13.737
  21:25:18 STEP 2828 H=2828:00 +CPU= 14.040
  21:25:40 STEP 2829 H=2829:00 +CPU= 21.828
  21:25:54 STEP 2830 H=2830:00 +CPU= 14.088
  21:26:09 STEP 2831 H=2831:00 +CPU= 14.110
  21:26:34 STEP 2832 H=2832:00 +CPU= 25.319
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002832
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002832
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002832
  21:26:48 STEP 2833 H=2833:00 +CPU= 13.700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002832
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_117.zip
Uploading the intermediate file: upload_file_117.zip
  21:27:02 STEP 2834 H=2834:00 +CPU= 14.091
  21:27:23 STEP 2835 H=2835:00 +CPU= 21.766
  21:27:38 STEP 2836 H=2836:00 +CPU= 14.140
  21:27:51 STEP 2837 H=2837:00 +CPU= 13.700
  21:28:14 STEP 2838 H=2838:00 +CPU= 22.266
  21:28:27 STEP 2839 H=2839:00 +CPU= 13.707
  21:28:41 STEP 2840 H=2840:00 +CPU= 14.067
  21:29:03 STEP 2841 H=2841:00 +CPU= 21.772
  21:29:17 STEP 2842 H=2842:00 +CPU= 14.004
  21:29:31 STEP 2843 H=2843:00 +CPU= 13.678
  21:29:54 STEP 2844 H=2844:00 +CPU= 23.683
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002844
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002844
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002844
  21:30:08 STEP 2845 H=2845:00 +CPU= 13.658
  21:30:22 STEP 2846 H=2846:00 +CPU= 14.011
  21:30:44 STEP 2847 H=2847:00 +CPU= 21.727
  21:30:58 STEP 2848 H=2848:00 +CPU= 14.003
  21:31:12 STEP 2849 H=2849:00 +CPU= 13.680
  21:31:34 STEP 2850 H=2850:00 +CPU= 22.138
  21:31:47 STEP 2851 H=2851:00 +CPU= 13.673
  21:32:01 STEP 2852 H=2852:00 +CPU= 14.064
  21:32:23 STEP 2853 H=2853:00 +CPU= 21.720
  21:32:37 STEP 2854 H=2854:00 +CPU= 14.019
  21:32:51 STEP 2855 H=2855:00 +CPU= 14.101
  21:33:17 STEP 2856 H=2856:00 +CPU= 25.291
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002856
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002856
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002856
  21:33:30 STEP 2857 H=2857:00 +CPU= 13.701
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002856
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_118.zip
Uploading the intermediate file: upload_file_118.zip
  21:33:44 STEP 2858 H=2858:00 +CPU= 14.063
  21:34:06 STEP 2859 H=2859:00 +CPU= 21.849
  21:34:20 STEP 2860 H=2860:00 +CPU= 14.178
  21:34:34 STEP 2861 H=2861:00 +CPU= 13.830
  21:34:57 STEP 2862 H=2862:00 +CPU= 22.244
  21:35:10 STEP 2863 H=2863:00 +CPU= 13.723
  21:35:24 STEP 2864 H=2864:00 +CPU= 14.095
  21:35:46 STEP 2865 H=2865:00 +CPU= 21.776
  21:36:00 STEP 2866 H=2866:00 +CPU= 14.046
  21:36:14 STEP 2867 H=2867:00 +CPU= 13.776
  21:36:38 STEP 2868 H=2868:00 +CPU= 23.809
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002868
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002868
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002868
  21:36:52 STEP 2869 H=2869:00 +CPU= 13.739
  21:37:06 STEP 2870 H=2870:00 +CPU= 14.167
  21:37:28 STEP 2871 H=2871:00 +CPU= 21.850
  21:37:42 STEP 2872 H=2872:00 +CPU= 14.106
  21:37:55 STEP 2873 H=2873:00 +CPU= 13.726
  21:38:18 STEP 2874 H=2874:00 +CPU= 22.246
  21:38:31 STEP 2875 H=2875:00 +CPU= 13.742
  21:38:45 STEP 2876 H=2876:00 +CPU= 14.055
  21:39:07 STEP 2877 H=2877:00 +CPU= 21.834
  21:39:21 STEP 2878 H=2878:00 +CPU= 14.041
  21:39:35 STEP 2879 H=2879:00 +CPU= 14.057
  21:40:01 STEP 2880 H=2880:00 +CPU= 25.300
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002880
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002880
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002880
  21:40:14 STEP 2881 H=2881:00 +CPU= 13.452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002880
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_119.zip
Uploading the intermediate file: upload_file_119.zip
  21:40:28 STEP 2882 H=2882:00 +CPU= 13.787
  21:40:49 STEP 2883 H=2883:00 +CPU= 21.352
  21:41:03 STEP 2884 H=2884:00 +CPU= 13.790
  21:41:17 STEP 2885 H=2885:00 +CPU= 13.456
  21:41:39 STEP 2886 H=2886:00 +CPU= 21.900
  21:41:52 STEP 2887 H=2887:00 +CPU= 13.528
  21:42:06 STEP 2888 H=2888:00 +CPU= 13.882
  21:42:27 STEP 2889 H=2889:00 +CPU= 21.485
  21:42:41 STEP 2890 H=2890:00 +CPU= 13.860
  21:42:55 STEP 2891 H=2891:00 +CPU= 13.472
  21:43:18 STEP 2892 H=2892:00 +CPU= 23.316
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002892
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002892
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002892
  21:43:32 STEP 2893 H=2893:00 +CPU= 13.476
  21:43:45 STEP 2894 H=2894:00 +CPU= 13.864
  21:44:07 STEP 2895 H=2895:00 +CPU= 21.433
  21:44:21 STEP 2896 H=2896:00 +CPU= 13.871
  21:44:34 STEP 2897 H=2897:00 +CPU= 13.431
  21:44:56 STEP 2898 H=2898:00 +CPU= 21.786
  21:45:09 STEP 2899 H=2899:00 +CPU= 13.430
  21:45:23 STEP 2900 H=2900:00 +CPU= 13.819
  21:45:45 STEP 2901 H=2901:00 +CPU= 21.300
  21:45:58 STEP 2902 H=2902:00 +CPU= 13.722
  21:46:12 STEP 2903 H=2903:00 +CPU= 13.708
  21:46:37 STEP 2904 H=2904:00 +CPU= 24.722
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002904
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002904
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002904
  21:46:50 STEP 2905 H=2905:00 +CPU= 13.353
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002904
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_120.zip
Uploading the intermediate file: upload_file_120.zip
  21:47:04 STEP 2906 H=2906:00 +CPU= 13.666
  21:47:25 STEP 2907 H=2907:00 +CPU= 21.383
  21:47:39 STEP 2908 H=2908:00 +CPU= 13.806
  21:47:52 STEP 2909 H=2909:00 +CPU= 13.429
  21:48:14 STEP 2910 H=2910:00 +CPU= 21.802
  21:48:27 STEP 2911 H=2911:00 +CPU= 13.355
  21:48:41 STEP 2912 H=2912:00 +CPU= 13.775
  21:49:03 STEP 2913 H=2913:00 +CPU= 21.383
  21:49:17 STEP 2914 H=2914:00 +CPU= 13.845
  21:49:30 STEP 2915 H=2915:00 +CPU= 13.445
  21:49:53 STEP 2916 H=2916:00 +CPU= 23.376
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002916
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002916
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002916
  21:50:07 STEP 2917 H=2917:00 +CPU= 13.422
  21:50:21 STEP 2918 H=2918:00 +CPU= 13.821
  21:50:42 STEP 2919 H=2919:00 +CPU= 21.351
  21:50:56 STEP 2920 H=2920:00 +CPU= 13.812
  21:51:09 STEP 2921 H=2921:00 +CPU= 13.422
  21:51:31 STEP 2922 H=2922:00 +CPU= 21.796
  21:51:44 STEP 2923 H=2923:00 +CPU= 13.383
  21:51:58 STEP 2924 H=2924:00 +CPU= 13.775
  21:52:20 STEP 2925 H=2925:00 +CPU= 21.352
  21:52:33 STEP 2926 H=2926:00 +CPU= 13.747
  21:52:47 STEP 2927 H=2927:00 +CPU= 13.767
  21:53:12 STEP 2928 H=2928:00 +CPU= 24.864
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002928
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002928
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002928
  21:53:25 STEP 2929 H=2929:00 +CPU= 13.450
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002928
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_121.zip
Uploading the intermediate file: upload_file_121.zip
  21:53:39 STEP 2930 H=2930:00 +CPU= 13.750
  21:54:01 STEP 2931 H=2931:00 +CPU= 21.436
  21:54:14 STEP 2932 H=2932:00 +CPU= 13.808
  21:54:28 STEP 2933 H=2933:00 +CPU= 13.549
  21:54:50 STEP 2934 H=2934:00 +CPU= 21.919
  21:55:03 STEP 2935 H=2935:00 +CPU= 13.445
  21:55:17 STEP 2936 H=2936:00 +CPU= 13.790
  21:55:39 STEP 2937 H=2937:00 +CPU= 21.468
  21:55:52 STEP 2938 H=2938:00 +CPU= 13.831
  21:56:06 STEP 2939 H=2939:00 +CPU= 13.415
  21:56:29 STEP 2940 H=2940:00 +CPU= 23.378
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002940
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002940
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002940
  21:56:43 STEP 2941 H=2941:00 +CPU= 13.465
  21:56:56 STEP 2942 H=2942:00 +CPU= 13.787
  21:57:18 STEP 2943 H=2943:00 +CPU= 21.442
  21:57:32 STEP 2944 H=2944:00 +CPU= 13.841
  21:57:45 STEP 2945 H=2945:00 +CPU= 13.447
  21:58:07 STEP 2946 H=2946:00 +CPU= 21.776
  21:58:20 STEP 2947 H=2947:00 +CPU= 13.406
  21:58:34 STEP 2948 H=2948:00 +CPU= 13.790
  21:58:56 STEP 2949 H=2949:00 +CPU= 21.384
  21:59:09 STEP 2950 H=2950:00 +CPU= 13.766
  21:59:23 STEP 2951 H=2951:00 +CPU= 13.784
  21:59:47 STEP 2952 H=2952:00 +CPU= 24.207
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 4.865283731608E-01 1.865286143845E-01 6.362598939121E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.260318722649E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.48989191816039E-02 -1.72994898449055E+00  1.89761310057317E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.19925486346338E-02 -1.00000000000000E+00  1.00000000000000E+00
  MAXGPFV : MAX. VALUE =    1.07372992596507     
  MAXGPFV : MAX. VALUE =   0.734836887391928     
 NSTEP =  2952 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGhnw4+002952 MODE=a
 IO-STREAM CLOSED - ICMGGhnw4+002952
 MPI-TASK:   1 -    362741050 BYTES IN      8 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0068( 4.4%)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302231500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302233000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302234500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200303000000  !!!!!!!!!!!! 
 21:59:47 STEP 2952 H=2952:00 +CPU= 24.207
 END CNT3
 NSTEP =  2952 CNT0   000000000  21.224  21.224  21.232   1.000               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           35/   804992476 WRITTEN         5968/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    147    0       35       786125     5968 ************    18.2     1.0    17.2    -0.7    25.4
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 6465830.3 6465830.3    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 6465816.1 6465816.1    100.00      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  417    1297.7    1297.7      8.37      0.00
   9 SPCM     - SPECTRAL COMP.                     384     161.5     161.5      0.96      0.00
  10 SCAN2M - PHYSICS                              385    7846.4    7846.4     46.72      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 32     755.2     755.2      0.37      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 17      10.3      10.3      0.00      0.00
  13 SCAN2M - RADIATION CALC.                      129    7933.4    7933.4     15.83      0.00
  17 GRIDFPOS IN CNT4                               32      28.6      28.6      0.01      0.00
  27 RESTART FILES - WRITING                        16    1510.4    1510.4      0.37      0.00
  28 RESTART FILES - READING                         1    -292.2       0.0      0.00      0.00
  29 SU4FPOS IN CNT4                                32       0.0       0.0      0.00      0.00
  30 DYNFPOS IN CNT4                                32    1344.3    1344.3      0.67      0.00
  31 POSDDH IN STEPO                                 2       0.5       0.5      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    385    2076.7    2076.7     12.37      0.00
  38 WAM      - TOTAL COST OF WAVE MODEL           385    1633.1    1633.1      9.72      0.00
  39 SU0YOMB                                         1     217.2     217.2      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    385       6.4       6.4      0.04      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              384      83.4      83.4      0.50      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              384      16.3      16.3      0.10      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 17       0.1       0.1      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     1318      80.1      80.1      1.63      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       800      81.0      81.0      1.00      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        800      32.6      32.6      0.40      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      1318      26.4      26.4      0.54      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            2      12.4      12.4      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           1318      11.7      11.7      0.24      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            800       9.6       9.6      0.12      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           1318      11.1      11.1      0.23      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            800       8.8       8.8      0.11      0.00
 400 GSTATS                                      77590       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 74251       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     6465.8 CPU TIME    6465.6 VECTOR TIME     6465.6


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     1643.6
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhnw4+002952
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhnw4+002952
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhnw4+002952
Adding to the zip: /var/lib/boinc-client/slots/4/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/4/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMSHhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMUAhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214456/ICMGGhnw4+002940
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00sa_2019110100_123_993_12214456_0_r1227949906_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
22:03:09 (1876): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:55:23 1537223 22315826 oifs_43r3_00sa_2019110100_123_993_12214456_0 10,623,600 69,278 0.0065
09 May 2023 16:55:20 1537223 22315826 oifs_43r3_00sa_2019110100_123_993_12214456_0 10,026,000 66,493 0.0066
09 May 2023 16:55:15 1537223 22315826 oifs_43r3_00sa_2019110100_123_993_12214456_0 8,467,200 59,173 0.0070
09 May 2023 16:55:15 1537223 22315826 oifs_43r3_00sa_2019110100_123_993_12214456_0 7,603,200 55,122 0.0072
09 May 2023 16:54:52 1537223 22315826 oifs_43r3_00sa_2019110100_123_993_12214456_0 6,134,400 48,106 0.0078
09 May 2023 16:54:52 1537223 22315826 oifs_43r3_00sa_2019110100_123_993_12214456_0 5,270,400 44,065 0.0084
09 May 2023 16:54:51 1537223 22315826 oifs_43r3_00sa_2019110100_123_993_12214456_0 4,406,400 40,013 0.0091
09 May 2023 16:54:50 1537223 22315826 oifs_43r3_00sa_2019110100_123_993_12214456_0 2,937,600 32,895 0.0112
09 May 2023 16:53:43 1537223 22315826 oifs_43r3_00sa_2019110100_123_993_12214456_0 1,814,400 23,533 0.0130
09 May 2023 16:53:33 1537223 22315826 oifs_43r3_00sa_2019110100_123_993_12214456_0 864,000 12,631 0.0146


©2024 climateprediction.net