climateprediction.net home page
Task 22261240

Task 22261240

Name oifs_43r3_ps_0843_1990050100_123_959_12176487_0
Workunit 12176487
Created 21 Dec 2022, 20:14:47 UTC
Sent 21 Dec 2022, 20:53:43 UTC
Report deadline 20 Jan 2023, 20:53:43 UTC
Received 25 Dec 2022, 13:07:46 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1537224
Run time 18 hours 44 min 20 sec
CPU time 16 hours 29 min 45 sec
Validate state Invalid
Credit 10,039.68
Device peak FLOPS 5.24 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.70 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,402.34 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
/ICMUAhq0f+001692
  12:36:26 STEP 1693 H=1693:00 +CPU= 21.725
  12:36:50 STEP 1694 H=1694:00 +CPU= 22.727
  12:37:29 STEP 1695 H=1695:00 +CPU= 36.024
  12:37:52 STEP 1696 H=1696:00 +CPU= 22.258
  12:38:15 STEP 1697 H=1697:00 +CPU= 21.664
  12:38:53 STEP 1698 H=1698:00 +CPU= 35.173
  12:39:16 STEP 1699 H=1699:00 +CPU= 21.997
  12:39:40 STEP 1700 H=1700:00 +CPU= 22.335
  12:40:17 STEP 1701 H=1701:00 +CPU= 33.803
  12:40:39 STEP 1702 H=1702:00 +CPU= 22.367
  12:41:03 STEP 1703 H=1703:00 +CPU= 22.191
  12:41:47 STEP 1704 H=1704:00 +CPU= 40.415
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_70.zip
Uploading the intermediate file: upload_file_70.zip
  12:42:11 STEP 1705 H=1705:00 +CPU= 21.871
  12:42:34 STEP 1706 H=1706:00 +CPU= 22.361
  12:43:11 STEP 1707 H=1707:00 +CPU= 34.676
  12:43:35 STEP 1708 H=1708:00 +CPU= 22.833
  12:43:58 STEP 1709 H=1709:00 +CPU= 21.346
  12:44:34 STEP 1710 H=1710:00 +CPU= 35.127
  12:44:57 STEP 1711 H=1711:00 +CPU= 22.262
  12:45:21 STEP 1712 H=1712:00 +CPU= 23.045
  12:45:58 STEP 1713 H=1713:00 +CPU= 35.021
  12:46:22 STEP 1714 H=1714:00 +CPU= 22.564
  12:46:45 STEP 1715 H=1715:00 +CPU= 21.790
  12:47:24 STEP 1716 H=1716:00 +CPU= 37.525
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001716
  12:47:47 STEP 1717 H=1717:00 +CPU= 21.951
  12:48:11 STEP 1718 H=1718:00 +CPU= 22.157
  12:48:47 STEP 1719 H=1719:00 +CPU= 33.641
  12:49:09 STEP 1720 H=1720:00 +CPU= 21.576
  12:49:32 STEP 1721 H=1721:00 +CPU= 21.350
  12:50:09 STEP 1722 H=1722:00 +CPU= 35.261
  12:50:34 STEP 1723 H=1723:00 +CPU= 22.599
  12:50:57 STEP 1724 H=1724:00 +CPU= 22.029
  12:51:33 STEP 1725 H=1725:00 +CPU= 34.908
  12:51:57 STEP 1726 H=1726:00 +CPU= 22.859
  12:52:21 STEP 1727 H=1727:00 +CPU= 22.298
  12:53:05 STEP 1728 H=1728:00 +CPU= 40.067
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_71.zip
Uploading the intermediate file: upload_file_71.zip
  12:53:29 STEP 1729 H=1729:00 +CPU= 22.262
  12:53:53 STEP 1730 H=1730:00 +CPU= 22.866
  12:54:29 STEP 1731 H=1731:00 +CPU= 33.733
  12:54:53 STEP 1732 H=1732:00 +CPU= 23.259
  12:55:16 STEP 1733 H=1733:00 +CPU= 21.534
  12:55:52 STEP 1734 H=1734:00 +CPU= 34.629
  12:56:16 STEP 1735 H=1735:00 +CPU= 22.074
  12:56:38 STEP 1736 H=1736:00 +CPU= 22.098
  12:57:13 STEP 1737 H=1737:00 +CPU= 33.515
  12:57:36 STEP 1738 H=1738:00 +CPU= 21.785
  12:57:58 STEP 1739 H=1739:00 +CPU= 21.412
  12:58:38 STEP 1740 H=1740:00 +CPU= 37.492
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001740
  12:59:01 STEP 1741 H=1741:00 +CPU= 21.500
  12:59:24 STEP 1742 H=1742:00 +CPU= 22.255
  13:00:00 STEP 1743 H=1743:00 +CPU= 33.895
  13:00:23 STEP 1744 H=1744:00 +CPU= 22.169
  13:00:47 STEP 1745 H=1745:00 +CPU= 21.573
  13:01:23 STEP 1746 H=1746:00 +CPU= 34.442
  13:01:46 STEP 1747 H=1747:00 +CPU= 21.877
  13:02:10 STEP 1748 H=1748:00 +CPU= 22.298
  13:02:46 STEP 1749 H=1749:00 +CPU= 33.673
  13:03:09 STEP 1750 H=1750:00 +CPU= 22.184
  13:03:33 STEP 1751 H=1751:00 +CPU= 22.780
  13:04:20 STEP 1752 H=1752:00 +CPU= 42.586
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_72.zip
Uploading the intermediate file: upload_file_72.zip
  13:04:43 STEP 1753 H=1753:00 +CPU= 22.045
  13:05:07 STEP 1754 H=1754:00 +CPU= 21.829
  13:05:42 STEP 1755 H=1755:00 +CPU= 33.332
  13:06:05 STEP 1756 H=1756:00 +CPU= 22.296
  13:06:29 STEP 1757 H=1757:00 +CPU= 22.078
  13:07:08 STEP 1758 H=1758:00 +CPU= 36.166
  13:07:31 STEP 1759 H=1759:00 +CPU= 21.766
  13:07:54 STEP 1760 H=1760:00 +CPU= 21.779
  13:08:30 STEP 1761 H=1761:00 +CPU= 33.793
  13:08:53 STEP 1762 H=1762:00 +CPU= 22.223
  13:09:16 STEP 1763 H=1763:00 +CPU= 21.515
  13:09:56 STEP 1764 H=1764:00 +CPU= 37.700
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001764
  13:10:18 STEP 1765 H=1765:00 +CPU= 21.333
  13:10:42 STEP 1766 H=1766:00 +CPU= 22.659
  13:11:18 STEP 1767 H=1767:00 +CPU= 34.232
  13:11:43 STEP 1768 H=1768:00 +CPU= 23.022
  13:12:06 STEP 1769 H=1769:00 +CPU= 21.836
  13:12:42 STEP 1770 H=1770:00 +CPU= 34.887
  13:13:05 STEP 1771 H=1771:00 +CPU= 21.628
  13:13:29 STEP 1772 H=1772:00 +CPU= 22.436
  13:14:05 STEP 1773 H=1773:00 +CPU= 33.754
  13:14:29 STEP 1774 H=1774:00 +CPU= 22.238
  13:14:56 STEP 1775 H=1775:00 +CPU= 25.049
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.086] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4749MB (heap), 4403MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.091] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.091] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.091] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.091] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.091] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4749 MB (maxheap), 4403 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.091] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.091] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     1        (1004) 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               LAPINEA 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131524:1671884124:5744.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                LARMES 
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131525:1671884125:5745.093] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:egonmint:1:1:2395:2395] [20221224:131525:1671884125:5745.093] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D53D6B  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000DE9AAD  larmes_                   544  larmes.F90
oifs_43r3_model.e  0000000000DE50FD  lapinea_                  270  lapinea.F90
oifs_43r3_model.e  0000000000D7A4EC  call_sl_                  300  call_sl.F90
oifs_43r3_model.e  0000000000D784FD  call_sl_heap_              94  call_sl_heap.F90
oifs_43r3_model.e  0000000000BF9958  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1990050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0843
(argv4) batchid: 959
(argv5) wuid: 12176487
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/16
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2b9e84a9768f191023e0278ee1f2f58b to: /var/lib/boinc-client/slots/16/oifs_43r3_ps_0843_1990050100_123_959_12176487.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/16/oifs_43r3_ps_0843_1990050100_123_959_12176487.zip
ic_ancil_file: ic_ancil_12176487
ifsdata_file: ifsdata_12176487
climate_data_file: clim_data_12176487
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_60d5056e355fd5605ededd51a4059c66 to: /var/lib/boinc-client/slots/16/ic_ancil_12176487.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/16/ic_ancil_12176487.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_89c8c55ba71b76cc9db66fc66811f289 to: /var/lib/boinc-client/slots/16/ifsdata/ifsdata_12176487.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/16/ifsdata/ifsdata_12176487.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/16/159l_2/clim_data_12176487.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/16/159l_2/clim_data_12176487.zip
Checking for progress XML file: /var/lib/boinc-client/slots/16/progress_file_12176487.xml
Opened progress file ok : /var/lib/boinc-client/slots/16/progress_file_12176487.xml
-- Model is restarting --
Adjusting last_iter, 1776, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/16/progress_file_12176487.xml
last_cpu_time: 48463
upload_file_number: 73
last_iter: 1775
last_upload: 6307200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688
The child process has been launched with process id: 2438
Executing the command: /var/lib/boinc-client/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:egonmint:1:1:2438:2438] [20221225:102735:1671960455:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:28:11 STEP 1776 H=1776:00 +CPU= 33.840
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_73.zip
Uploading the intermediate file: upload_file_73.zip
  10:28:36 STEP 1777 H=1777:00 +CPU= 21.760
  10:28:58 STEP 1778 H=1778:00 +CPU= 22.557
  10:29:33 STEP 1779 H=1779:00 +CPU= 34.379
  10:29:56 STEP 1780 H=1780:00 +CPU= 22.520
  10:30:18 STEP 1781 H=1781:00 +CPU= 21.586
  10:30:53 STEP 1782 H=1782:00 +CPU= 34.728
  10:31:15 STEP 1783 H=1783:00 +CPU= 21.832
  10:31:39 STEP 1784 H=1784:00 +CPU= 23.193
  10:32:13 STEP 1785 H=1785:00 +CPU= 34.476
  10:32:37 STEP 1786 H=1786:00 +CPU= 22.473
  10:33:00 STEP 1787 H=1787:00 +CPU= 21.986
  10:33:39 STEP 1788 H=1788:00 +CPU= 38.075
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001788
  10:34:01 STEP 1789 H=1789:00 +CPU= 22.229
  10:34:24 STEP 1790 H=1790:00 +CPU= 22.799
  10:34:59 STEP 1791 H=1791:00 +CPU= 34.241
  10:35:23 STEP 1792 H=1792:00 +CPU= 22.931
  10:35:45 STEP 1793 H=1793:00 +CPU= 21.995
  10:36:22 STEP 1794 H=1794:00 +CPU= 35.326
  10:36:45 STEP 1795 H=1795:00 +CPU= 21.906
  10:37:08 STEP 1796 H=1796:00 +CPU= 22.665
  10:37:43 STEP 1797 H=1797:00 +CPU= 34.313
  10:38:07 STEP 1798 H=1798:00 +CPU= 22.734
  10:38:30 STEP 1799 H=1799:00 +CPU= 22.932
  10:39:11 STEP 1800 H=1800:00 +CPU= 40.278
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_74.zip
Uploading the intermediate file: upload_file_74.zip
  10:39:33 STEP 1801 H=1801:00 +CPU= 21.688
  10:39:57 STEP 1802 H=1802:00 +CPU= 22.887
  10:40:33 STEP 1803 H=1803:00 +CPU= 34.797
  10:40:56 STEP 1804 H=1804:00 +CPU= 22.939
  10:41:18 STEP 1805 H=1805:00 +CPU= 21.460
  10:41:52 STEP 1806 H=1806:00 +CPU= 34.270
  10:42:15 STEP 1807 H=1807:00 +CPU= 22.799
  10:42:39 STEP 1808 H=1808:00 +CPU= 23.060
  10:43:14 STEP 1809 H=1809:00 +CPU= 34.631
  10:43:37 STEP 1810 H=1810:00 +CPU= 22.747
  10:44:00 STEP 1811 H=1811:00 +CPU= 22.058
  10:44:39 STEP 1812 H=1812:00 +CPU= 38.469
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001812
  10:45:01 STEP 1813 H=1813:00 +CPU= 21.780
  10:45:24 STEP 1814 H=1814:00 +CPU= 22.274
  10:45:58 STEP 1815 H=1815:00 +CPU= 33.466
  10:46:21 STEP 1816 H=1816:00 +CPU= 21.742
  10:46:43 STEP 1817 H=1817:00 +CPU= 21.402
  10:47:19 STEP 1818 H=1818:00 +CPU= 35.638
  10:47:42 STEP 1819 H=1819:00 +CPU= 22.348
  10:48:05 STEP 1820 H=1820:00 +CPU= 23.192
  10:48:39 STEP 1821 H=1821:00 +CPU= 33.630
  10:49:03 STEP 1822 H=1822:00 +CPU= 23.060
  10:49:26 STEP 1823 H=1823:00 +CPU= 22.715
  10:50:09 STEP 1824 H=1824:00 +CPU= 41.254
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_75.zip
Uploading the intermediate file: upload_file_75.zip
  10:50:33 STEP 1825 H=1825:00 +CPU= 22.447
  10:50:57 STEP 1826 H=1826:00 +CPU= 23.790
  10:51:34 STEP 1827 H=1827:00 +CPU= 36.127
  10:51:57 STEP 1828 H=1828:00 +CPU= 23.008
  10:52:20 STEP 1829 H=1829:00 +CPU= 22.249
  10:52:55 STEP 1830 H=1830:00 +CPU= 34.627
  10:53:18 STEP 1831 H=1831:00 +CPU= 21.872
  10:53:41 STEP 1832 H=1832:00 +CPU= 22.384
  10:54:16 STEP 1833 H=1833:00 +CPU= 34.346
  10:54:39 STEP 1834 H=1834:00 +CPU= 22.810
  10:55:01 STEP 1835 H=1835:00 +CPU= 21.996
  10:55:41 STEP 1836 H=1836:00 +CPU= 38.655
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001836
  10:56:03 STEP 1837 H=1837:00 +CPU= 22.143
  10:56:26 STEP 1838 H=1838:00 +CPU= 22.433
  10:57:01 STEP 1839 H=1839:00 +CPU= 34.251
  10:57:24 STEP 1840 H=1840:00 +CPU= 22.133
  10:57:45 STEP 1841 H=1841:00 +CPU= 20.739
  10:58:20 STEP 1842 H=1842:00 +CPU= 34.193
  10:58:41 STEP 1843 H=1843:00 +CPU= 21.538
  10:59:04 STEP 1844 H=1844:00 +CPU= 22.320
  10:59:39 STEP 1845 H=1845:00 +CPU= 34.512
  11:00:03 STEP 1846 H=1846:00 +CPU= 24.087
  11:00:27 STEP 1847 H=1847:00 +CPU= 23.635
  11:01:11 STEP 1848 H=1848:00 +CPU= 42.855
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_76.zip
Uploading the intermediate file: upload_file_76.zip
  11:01:35 STEP 1849 H=1849:00 +CPU= 22.549
  11:01:59 STEP 1850 H=1850:00 +CPU= 23.656
  11:02:35 STEP 1851 H=1851:00 +CPU= 35.119
  11:02:58 STEP 1852 H=1852:00 +CPU= 22.868
  11:03:19 STEP 1853 H=1853:00 +CPU= 21.706
  11:03:55 STEP 1854 H=1854:00 +CPU= 35.105
  11:04:17 STEP 1855 H=1855:00 +CPU= 21.972
  11:04:40 STEP 1856 H=1856:00 +CPU= 22.536
  11:05:15 STEP 1857 H=1857:00 +CPU= 34.480
  11:05:38 STEP 1858 H=1858:00 +CPU= 22.493
  11:06:00 STEP 1859 H=1859:00 +CPU= 22.264
  11:06:40 STEP 1860 H=1860:00 +CPU= 39.000
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001860
  11:07:02 STEP 1861 H=1861:00 +CPU= 21.835
  11:07:25 STEP 1862 H=1862:00 +CPU= 22.163
  11:07:59 STEP 1863 H=1863:00 +CPU= 33.668
  11:08:21 STEP 1864 H=1864:00 +CPU= 22.037
  11:08:43 STEP 1865 H=1865:00 +CPU= 22.015
  11:09:18 STEP 1866 H=1866:00 +CPU= 34.801
  11:09:41 STEP 1867 H=1867:00 +CPU= 22.403
  11:10:04 STEP 1868 H=1868:00 +CPU= 22.671
  11:10:39 STEP 1869 H=1869:00 +CPU= 33.624
  11:11:01 STEP 1870 H=1870:00 +CPU= 21.728
  11:11:24 STEP 1871 H=1871:00 +CPU= 21.979
  11:12:05 STEP 1872 H=1872:00 +CPU= 39.831
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_77.zip
Uploading the intermediate file: upload_file_77.zip
  11:12:27 STEP 1873 H=1873:00 +CPU= 21.922
  11:12:50 STEP 1874 H=1874:00 +CPU= 22.180
  11:13:24 STEP 1875 H=1875:00 +CPU= 33.331
  11:13:45 STEP 1876 H=1876:00 +CPU= 21.429
  11:14:07 STEP 1877 H=1877:00 +CPU= 21.401
  11:14:42 STEP 1878 H=1878:00 +CPU= 34.039
  11:15:04 STEP 1879 H=1879:00 +CPU= 21.455
  11:15:26 STEP 1880 H=1880:00 +CPU= 22.204
  11:16:00 STEP 1881 H=1881:00 +CPU= 33.942
  11:16:22 STEP 1882 H=1882:00 +CPU= 21.995
  11:16:44 STEP 1883 H=1883:00 +CPU= 21.187
  11:17:21 STEP 1884 H=1884:00 +CPU= 36.819
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001884
  11:17:43 STEP 1885 H=1885:00 +CPU= 21.070
  11:18:04 STEP 1886 H=1886:00 +CPU= 21.656
  11:18:38 STEP 1887 H=1887:00 +CPU= 33.458
  11:19:00 STEP 1888 H=1888:00 +CPU= 21.222
  11:19:22 STEP 1889 H=1889:00 +CPU= 21.748
  11:19:57 STEP 1890 H=1890:00 +CPU= 34.679
  11:20:19 STEP 1891 H=1891:00 +CPU= 21.748
  11:20:43 STEP 1892 H=1892:00 +CPU= 23.467
  11:21:18 STEP 1893 H=1893:00 +CPU= 34.879
  11:21:40 STEP 1894 H=1894:00 +CPU= 22.469
  11:22:03 STEP 1895 H=1895:00 +CPU= 22.550
  11:22:46 STEP 1896 H=1896:00 +CPU= 41.528
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_78.zip
Uploading the intermediate file: upload_file_78.zip
  11:23:10 STEP 1897 H=1897:00 +CPU= 21.758
  11:23:33 STEP 1898 H=1898:00 +CPU= 22.179
  11:24:07 STEP 1899 H=1899:00 +CPU= 33.097
  11:24:29 STEP 1900 H=1900:00 +CPU= 21.751
  11:24:51 STEP 1901 H=1901:00 +CPU= 21.315
  11:25:27 STEP 1902 H=1902:00 +CPU= 34.840
  11:25:49 STEP 1903 H=1903:00 +CPU= 21.625
  11:26:11 STEP 1904 H=1904:00 +CPU= 22.003
  11:26:46 STEP 1905 H=1905:00 +CPU= 33.674
  11:27:09 STEP 1906 H=1906:00 +CPU= 21.879
  11:27:30 STEP 1907 H=1907:00 +CPU= 21.333
  11:28:08 STEP 1908 H=1908:00 +CPU= 38.000
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001908
  11:28:30 STEP 1909 H=1909:00 +CPU= 21.468
  11:28:53 STEP 1910 H=1910:00 +CPU= 22.323
  11:29:27 STEP 1911 H=1911:00 +CPU= 34.030
  11:29:50 STEP 1912 H=1912:00 +CPU= 22.222
  11:30:11 STEP 1913 H=1913:00 +CPU= 21.789
  11:30:47 STEP 1914 H=1914:00 +CPU= 35.150
  11:31:10 STEP 1915 H=1915:00 +CPU= 22.456
  11:31:33 STEP 1916 H=1916:00 +CPU= 22.898
  11:32:07 STEP 1917 H=1917:00 +CPU= 33.887
  11:32:30 STEP 1918 H=1918:00 +CPU= 22.355
  11:32:53 STEP 1919 H=1919:00 +CPU= 22.635
  11:33:35 STEP 1920 H=1920:00 +CPU= 40.715
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_79.zip
Uploading the intermediate file: upload_file_79.zip
  11:33:59 STEP 1921 H=1921:00 +CPU= 22.325
  11:34:21 STEP 1922 H=1922:00 +CPU= 22.634
  11:34:55 STEP 1923 H=1923:00 +CPU= 33.337
  11:35:18 STEP 1924 H=1924:00 +CPU= 22.181
  11:35:39 STEP 1925 H=1925:00 +CPU= 21.497
  11:36:15 STEP 1926 H=1926:00 +CPU= 35.432
  11:36:37 STEP 1927 H=1927:00 +CPU= 22.070
  11:37:00 STEP 1928 H=1928:00 +CPU= 22.519
  11:37:34 STEP 1929 H=1929:00 +CPU= 33.645
  11:37:57 STEP 1930 H=1930:00 +CPU= 22.614
  11:38:19 STEP 1931 H=1931:00 +CPU= 21.675
  11:38:59 STEP 1932 H=1932:00 +CPU= 39.753
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001932
  11:39:21 STEP 1933 H=1933:00 +CPU= 22.408
  11:39:44 STEP 1934 H=1934:00 +CPU= 23.051
  11:40:20 STEP 1935 H=1935:00 +CPU= 34.949
  11:40:43 STEP 1936 H=1936:00 +CPU= 22.663
  11:41:05 STEP 1937 H=1937:00 +CPU= 22.337
  11:41:40 STEP 1938 H=1938:00 +CPU= 34.598
  11:42:02 STEP 1939 H=1939:00 +CPU= 21.726
  11:42:25 STEP 1940 H=1940:00 +CPU= 22.905
  11:43:00 STEP 1941 H=1941:00 +CPU= 34.469
  11:43:23 STEP 1942 H=1942:00 +CPU= 22.767
  11:43:47 STEP 1943 H=1943:00 +CPU= 23.172
  11:44:29 STEP 1944 H=1944:00 +CPU= 41.416
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_80.zip
Uploading the intermediate file: upload_file_80.zip
  11:44:51 STEP 1945 H=1945:00 +CPU= 21.460
  11:45:13 STEP 1946 H=1946:00 +CPU= 21.808
  11:45:46 STEP 1947 H=1947:00 +CPU= 32.891
  11:46:09 STEP 1948 H=1948:00 +CPU= 22.031
  11:46:30 STEP 1949 H=1949:00 +CPU= 21.618
  11:47:05 STEP 1950 H=1950:00 +CPU= 34.097
  11:47:27 STEP 1951 H=1951:00 +CPU= 21.475
  11:47:49 STEP 1952 H=1952:00 +CPU= 22.409
  11:48:25 STEP 1953 H=1953:00 +CPU= 35.138
  11:48:49 STEP 1954 H=1954:00 +CPU= 23.757
  11:49:13 STEP 1955 H=1955:00 +CPU= 23.760
  11:49:52 STEP 1956 H=1956:00 +CPU= 38.664
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001956
  11:50:15 STEP 1957 H=1957:00 +CPU= 22.031
  11:50:39 STEP 1958 H=1958:00 +CPU= 23.217
  11:51:14 STEP 1959 H=1959:00 +CPU= 35.181
  11:51:38 STEP 1960 H=1960:00 +CPU= 23.345
  11:52:00 STEP 1961 H=1961:00 +CPU= 21.827
  11:52:35 STEP 1962 H=1962:00 +CPU= 34.378
  11:52:57 STEP 1963 H=1963:00 +CPU= 21.513
  11:53:20 STEP 1964 H=1964:00 +CPU= 22.522
  11:53:53 STEP 1965 H=1965:00 +CPU= 33.560
  11:54:16 STEP 1966 H=1966:00 +CPU= 22.282
  11:54:38 STEP 1967 H=1967:00 +CPU= 21.901
  11:55:19 STEP 1968 H=1968:00 +CPU= 40.181
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_81.zip
Uploading the intermediate file: upload_file_81.zip
  11:55:42 STEP 1969 H=1969:00 +CPU= 21.472
  11:56:05 STEP 1970 H=1970:00 +CPU= 22.435
  11:56:38 STEP 1971 H=1971:00 +CPU= 33.209
  11:57:02 STEP 1972 H=1972:00 +CPU= 22.967
  11:57:24 STEP 1973 H=1973:00 +CPU= 21.832
  11:58:00 STEP 1974 H=1974:00 +CPU= 35.347
  11:58:22 STEP 1975 H=1975:00 +CPU= 22.135
  11:58:45 STEP 1976 H=1976:00 +CPU= 22.748
  11:59:20 STEP 1977 H=1977:00 +CPU= 34.580
  11:59:43 STEP 1978 H=1978:00 +CPU= 22.350
  12:00:04 STEP 1979 H=1979:00 +CPU= 21.270
  12:00:43 STEP 1980 H=1980:00 +CPU= 38.288
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001980
  12:01:05 STEP 1981 H=1981:00 +CPU= 21.538
  12:01:27 STEP 1982 H=1982:00 +CPU= 22.310
  12:02:02 STEP 1983 H=1983:00 +CPU= 33.681
  12:02:24 STEP 1984 H=1984:00 +CPU= 22.351
  12:02:47 STEP 1985 H=1985:00 +CPU= 21.606
  12:03:22 STEP 1986 H=1986:00 +CPU= 34.864
  12:03:44 STEP 1987 H=1987:00 +CPU= 21.737
  12:04:06 STEP 1988 H=1988:00 +CPU= 22.246
  12:04:40 STEP 1989 H=1989:00 +CPU= 33.569
  12:05:02 STEP 1990 H=1990:00 +CPU= 22.352
  12:05:26 STEP 1991 H=1991:00 +CPU= 22.326
  12:06:08 STEP 1992 H=1992:00 +CPU= 40.786
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_82.zip
Uploading the intermediate file: upload_file_82.zip
  12:06:30 STEP 1993 H=1993:00 +CPU= 21.909
Uploading trickle at timestep: 7171200
  12:06:53 STEP 1994 H=1994:00 +CPU= 22.504
  12:07:28 STEP 1995 H=1995:00 +CPU= 34.208
  12:07:50 STEP 1996 H=1996:00 +CPU= 22.301
  12:08:13 STEP 1997 H=1997:00 +CPU= 22.055
  12:08:48 STEP 1998 H=1998:00 +CPU= 34.805
  12:09:09 STEP 1999 H=1999:00 +CPU= 21.803
  12:09:32 STEP 2000 H=2000:00 +CPU= 22.207
  12:10:06 STEP 2001 H=2001:00 +CPU= 34.034
  12:10:29 STEP 2002 H=2002:00 +CPU= 22.262
  12:10:51 STEP 2003 H=2003:00 +CPU= 22.002
  12:11:31 STEP 2004 H=2004:00 +CPU= 38.951
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002004
  12:11:53 STEP 2005 H=2005:00 +CPU= 21.509
  12:12:16 STEP 2006 H=2006:00 +CPU= 22.698
  12:12:50 STEP 2007 H=2007:00 +CPU= 33.571
  12:13:12 STEP 2008 H=2008:00 +CPU= 21.838
  12:13:35 STEP 2009 H=2009:00 +CPU= 22.223
  12:14:11 STEP 2010 H=2010:00 +CPU= 35.480
  12:14:34 STEP 2011 H=2011:00 +CPU= 23.731
  12:14:58 STEP 2012 H=2012:00 +CPU= 23.422
  12:15:34 STEP 2013 H=2013:00 +CPU= 35.578
  12:15:57 STEP 2014 H=2014:00 +CPU= 22.605
  12:16:20 STEP 2015 H=2015:00 +CPU= 22.575
  12:17:02 STEP 2016 H=2016:00 +CPU= 40.619
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_83.zip
Uploading the intermediate file: upload_file_83.zip
  12:17:25 STEP 2017 H=2017:00 +CPU= 21.573
  12:17:47 STEP 2018 H=2018:00 +CPU= 22.303
  12:18:22 STEP 2019 H=2019:00 +CPU= 34.051
  12:18:44 STEP 2020 H=2020:00 +CPU= 22.529
  12:19:07 STEP 2021 H=2021:00 +CPU= 21.966
  12:19:42 STEP 2022 H=2022:00 +CPU= 34.510
  12:20:03 STEP 2023 H=2023:00 +CPU= 21.018
  12:20:26 STEP 2024 H=2024:00 +CPU= 22.462
  12:21:00 STEP 2025 H=2025:00 +CPU= 34.556
  12:21:23 STEP 2026 H=2026:00 +CPU= 22.476
  12:21:46 STEP 2027 H=2027:00 +CPU= 22.373
  12:22:25 STEP 2028 H=2028:00 +CPU= 38.900
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002028
  12:22:47 STEP 2029 H=2029:00 +CPU= 22.007
  12:23:10 STEP 2030 H=2030:00 +CPU= 22.714
  12:23:45 STEP 2031 H=2031:00 +CPU= 34.770
  12:24:08 STEP 2032 H=2032:00 +CPU= 22.220
  12:24:29 STEP 2033 H=2033:00 +CPU= 21.019
  12:25:04 STEP 2034 H=2034:00 +CPU= 34.759
  12:25:26 STEP 2035 H=2035:00 +CPU= 21.841
  12:25:50 STEP 2036 H=2036:00 +CPU= 23.420
  12:26:26 STEP 2037 H=2037:00 +CPU= 36.156
  12:26:50 STEP 2038 H=2038:00 +CPU= 23.595
  12:27:15 STEP 2039 H=2039:00 +CPU= 24.205
  12:27:58 STEP 2040 H=2040:00 +CPU= 41.722
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_84.zip
Uploading the intermediate file: upload_file_84.zip
  12:28:22 STEP 2041 H=2041:00 +CPU= 22.329
  12:28:45 STEP 2042 H=2042:00 +CPU= 23.169
  12:29:21 STEP 2043 H=2043:00 +CPU= 35.548
  12:29:45 STEP 2044 H=2044:00 +CPU= 23.313
  12:30:08 STEP 2045 H=2045:00 +CPU= 23.002
  12:30:47 STEP 2046 H=2046:00 +CPU= 37.310
  12:31:11 STEP 2047 H=2047:00 +CPU= 23.563
  12:31:36 STEP 2048 H=2048:00 +CPU= 25.570
  12:32:15 STEP 2049 H=2049:00 +CPU= 37.347
  12:32:37 STEP 2050 H=2050:00 +CPU= 22.347
  12:33:00 STEP 2051 H=2051:00 +CPU= 22.198
  12:33:39 STEP 2052 H=2052:00 +CPU= 38.273
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002052
  12:34:01 STEP 2053 H=2053:00 +CPU= 22.213
  12:34:24 STEP 2054 H=2054:00 +CPU= 22.214
  12:34:59 STEP 2055 H=2055:00 +CPU= 35.409
  12:35:23 STEP 2056 H=2056:00 +CPU= 22.964
  12:35:45 STEP 2057 H=2057:00 +CPU= 21.941
  12:36:21 STEP 2058 H=2058:00 +CPU= 35.355
  12:36:43 STEP 2059 H=2059:00 +CPU= 21.916
  12:37:06 STEP 2060 H=2060:00 +CPU= 22.731
  12:37:41 STEP 2061 H=2061:00 +CPU= 34.444
  12:38:09 STEP 2062 H=2062:00 +CPU= 22.738
  12:38:32 STEP 2063 H=2063:00 +CPU= 22.996
  12:39:14 STEP 2064 H=2064:00 +CPU= 41.046
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_85.zip
Uploading the intermediate file: upload_file_85.zip
  12:39:36 STEP 2065 H=2065:00 +CPU= 22.037
  12:39:59 STEP 2066 H=2066:00 +CPU= 22.546
  12:40:34 STEP 2067 H=2067:00 +CPU= 34.203
  12:40:57 STEP 2068 H=2068:00 +CPU= 23.263
  12:41:20 STEP 2069 H=2069:00 +CPU= 22.219
  12:41:57 STEP 2070 H=2070:00 +CPU= 36.523
  12:42:19 STEP 2071 H=2071:00 +CPU= 21.737
  12:42:42 STEP 2072 H=2072:00 +CPU= 22.972
  12:43:18 STEP 2073 H=2073:00 +CPU= 34.443
  12:43:40 STEP 2074 H=2074:00 +CPU= 22.197
  12:44:02 STEP 2075 H=2075:00 +CPU= 21.792
  12:44:41 STEP 2076 H=2076:00 +CPU= 37.971
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002076
  12:45:04 STEP 2077 H=2077:00 +CPU= 22.233
  12:45:27 STEP 2078 H=2078:00 +CPU= 23.116
  12:46:01 STEP 2079 H=2079:00 +CPU= 33.440
  12:46:24 STEP 2080 H=2080:00 +CPU= 22.796
  12:46:46 STEP 2081 H=2081:00 +CPU= 21.818
  12:47:22 STEP 2082 H=2082:00 +CPU= 35.230
  12:47:43 STEP 2083 H=2083:00 +CPU= 21.009
  12:48:05 STEP 2084 H=2084:00 +CPU= 21.704
  12:48:38 STEP 2085 H=2085:00 +CPU= 33.005
  12:49:01 STEP 2086 H=2086:00 +CPU= 22.442
  12:49:25 STEP 2087 H=2087:00 +CPU= 23.033
  12:50:05 STEP 2088 H=2088:00 +CPU= 39.100
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_86.zip
Uploading the intermediate file: upload_file_86.zip
  12:50:29 STEP 2089 H=2089:00 +CPU= 21.729
  12:50:51 STEP 2090 H=2090:00 +CPU= 22.152
  12:51:25 STEP 2091 H=2091:00 +CPU= 33.520
  12:51:48 STEP 2092 H=2092:00 +CPU= 22.039
  12:52:10 STEP 2093 H=2093:00 +CPU= 21.706
  12:52:45 STEP 2094 H=2094:00 +CPU= 34.629
  12:53:07 STEP 2095 H=2095:00 +CPU= 21.305
  12:53:29 STEP 2096 H=2096:00 +CPU= 22.178
  12:54:04 STEP 2097 H=2097:00 +CPU= 33.900
  12:54:26 STEP 2098 H=2098:00 +CPU= 22.327
  12:54:49 STEP 2099 H=2099:00 +CPU= 22.197
  12:55:28 STEP 2100 H=2100:00 +CPU= 38.622
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002100
  12:55:50 STEP 2101 H=2101:00 +CPU= 21.726
  12:56:13 STEP 2102 H=2102:00 +CPU= 22.374
  12:56:47 STEP 2103 H=2103:00 +CPU= 34.245
  12:57:10 STEP 2104 H=2104:00 +CPU= 22.418
  12:57:32 STEP 2105 H=2105:00 +CPU= 21.431
  12:58:06 STEP 2106 H=2106:00 +CPU= 34.128
  12:58:28 STEP 2107 H=2107:00 +CPU= 21.531
  12:58:51 STEP 2108 H=2108:00 +CPU= 22.328
  12:59:25 STEP 2109 H=2109:00 +CPU= 34.370
  12:59:48 STEP 2110 H=2110:00 +CPU= 22.609
  13:00:11 STEP 2111 H=2111:00 +CPU= 22.499
  13:00:52 STEP 2112 H=2112:00 +CPU= 39.900
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_87.zip
Uploading the intermediate file: upload_file_87.zip
  13:01:16 STEP 2113 H=2113:00 +CPU= 22.324
  13:01:40 STEP 2114 H=2114:00 +CPU= 23.732
  13:02:16 STEP 2115 H=2115:00 +CPU= 35.971
  13:02:41 STEP 2116 H=2116:00 +CPU= 24.241
  13:03:04 STEP 2117 H=2117:00 +CPU= 22.938
  13:03:42 STEP 2118 H=2118:00 +CPU= 36.610
  13:04:04 STEP 2119 H=2119:00 +CPU= 21.741
  13:04:27 STEP 2120 H=2120:00 +CPU= 23.083
  13:05:03 STEP 2121 H=2121:00 +CPU= 35.487
  13:05:27 STEP 2122 H=2122:00 +CPU= 22.961
  13:05:49 STEP 2123 H=2123:00 +CPU= 21.620
  13:06:28 STEP 2124 H=2124:00 +CPU= 38.392
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002124
  13:06:50 STEP 2125 H=2125:00 +CPU= 21.568
  13:07:12 STEP 2126 H=2126:00 +CPU= 22.065
  13:07:46 STEP 2127 H=2127:00 +CPU= 33.628
  13:08:09 STEP 2128 H=2128:00 +CPU= 21.676
  13:08:31 STEP 2129 H=2129:00 +CPU= 21.228
  13:09:05 STEP 2130 H=2130:00 +CPU= 33.927
  13:09:27 STEP 2131 H=2131:00 +CPU= 21.226
  13:09:49 STEP 2132 H=2132:00 +CPU= 22.106
  13:10:23 STEP 2133 H=2133:00 +CPU= 33.730
  13:10:47 STEP 2134 H=2134:00 +CPU= 22.973
  13:11:10 STEP 2135 H=2135:00 +CPU= 22.808
  13:11:52 STEP 2136 H=2136:00 +CPU= 41.062
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_88.zip
Uploading the intermediate file: upload_file_88.zip
  13:12:15 STEP 2137 H=2137:00 +CPU= 21.710
  13:12:38 STEP 2138 H=2138:00 +CPU= 22.636
  13:13:13 STEP 2139 H=2139:00 +CPU= 34.163
  13:13:35 STEP 2140 H=2140:00 +CPU= 22.614
  13:13:58 STEP 2141 H=2141:00 +CPU= 21.859
  13:14:34 STEP 2142 H=2142:00 +CPU= 35.321
  13:14:56 STEP 2143 H=2143:00 +CPU= 21.474
  13:15:19 STEP 2144 H=2144:00 +CPU= 22.405
  13:15:53 STEP 2145 H=2145:00 +CPU= 34.107
  13:16:16 STEP 2146 H=2146:00 +CPU= 22.741
  13:16:38 STEP 2147 H=2147:00 +CPU= 21.961
  13:17:17 STEP 2148 H=2148:00 +CPU= 38.002
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002148
  13:17:38 STEP 2149 H=2149:00 +CPU= 21.540
  13:18:01 STEP 2150 H=2150:00 +CPU= 22.022
  13:18:35 STEP 2151 H=2151:00 +CPU= 33.682
  13:18:57 STEP 2152 H=2152:00 +CPU= 22.183
  13:19:19 STEP 2153 H=2153:00 +CPU= 21.562
  13:19:54 STEP 2154 H=2154:00 +CPU= 34.125
  13:20:16 STEP 2155 H=2155:00 +CPU= 21.754
  13:20:38 STEP 2156 H=2156:00 +CPU= 22.137
  13:21:12 STEP 2157 H=2157:00 +CPU= 32.652
  13:21:34 STEP 2158 H=2158:00 +CPU= 22.520
  13:21:57 STEP 2159 H=2159:00 +CPU= 22.215
  13:22:38 STEP 2160 H=2160:00 +CPU= 40.069
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176487/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1990050100_123_959_12176487_0_r801152688_89.zip
Uploading the intermediate file: upload_file_89.zip
  13:23:02 STEP 2161 H=2161:00 +CPU= 23.201
  13:23:30 STEP 2162 H=2162:00 +CPU= 27.001
  13:24:07 STEP 2163 H=2163:00 +CPU= 37.034
  13:24:29 STEP 2164 H=2164:00 +CPU= 22.366
  13:24:51 STEP 2165 H=2165:00 +CPU= 21.573
  13:25:26 STEP 2166 H=2166:00 +CPU= 34.496
  13:25:49 STEP 2167 H=2167:00 +CPU= 21.714
  13:26:12 STEP 2168 H=2168:00 +CPU= 22.249
  13:26:46 STEP 2169 H=2169:00 +CPU= 33.746
  13:27:08 STEP 2170 H=2170:00 +CPU= 22.351
  13:27:30 STEP 2171 H=2171:00 +CPU= 21.438
  13:28:09 STEP 2172 H=2172:00 +CPU= 37.690
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+002172
  13:28:31 STEP 2173 H=2173:00 +CPU= 21.696
  13:28:53 STEP 2174 H=2174:00 +CPU= 21.861
  13:29:27 STEP 2175 H=2175:00 +CPU= 33.648
  13:29:50 STEP 2176 H=2176:00 +CPU= 22.524
  13:30:11 STEP 2177 H=2177:00 +CPU= 21.552
  13:30:46 STEP 2178 H=2178:00 +CPU= 34.020
  13:31:08 STEP 2179 H=2179:00 +CPU= 21.933
  13:31:30 STEP 2180 H=2180:00 +CPU= 22.176
  13:32:04 STEP 2181 H=2181:00 +CPU= 33.195
  13:32:27 STEP 2182 H=2182:00 +CPU= 22.473
  13:32:50 STEP 2183 H=2183:00 +CPU= 22.132
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:54:51 1537224 22261240 oifs_43r3_ps_0843_1990050100_123_959_12176487_0 7,171,200 54,272 0.0076
09 May 2023 15:50:54 1537224 22261240 oifs_43r3_ps_0843_1990050100_123_959_12176487_0 5,184,000 39,226 0.0076
09 May 2023 15:48:42 1537224 22261240 oifs_43r3_ps_0843_1990050100_123_959_12176487_0 3,024,000 23,029 0.0076
09 May 2023 15:47:58 1537224 22261240 oifs_43r3_ps_0843_1990050100_123_959_12176487_0 1,814,400 13,763 0.0076
09 May 2023 15:47:39 1537224 22261240 oifs_43r3_ps_0843_1990050100_123_959_12176487_0 950,400 7,112 0.0075


©2024 climateprediction.net