climateprediction.net home page
Task 22274187

Task 22274187

Name oifs_43r3_ps_0046_2003050100_123_972_12188690_0
Workunit 12188690
Created 23 Dec 2022, 4:19:50 UTC
Sent 10 Jan 2023, 6:30:13 UTC
Report deadline 9 Feb 2023, 6:30:13 UTC
Received 11 Jan 2023, 5:11:33 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1463165
Run time 12 hours 42 min 59 sec
CPU time 12 hours 41 min 10 sec
Validate state Invalid
Credit 6,168.96
Device peak FLOPS 4.74 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,792.20 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,971.24 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
EP 1510 H=1510:00 +CPU= 19.532
  14:14:11 STEP 1511 H=1511:00 +CPU= 19.464
  14:14:46 STEP 1512 H=1512:00 +CPU= 35.537
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_62.zip
Uploading the intermediate file: upload_file_62.zip
  14:15:05 STEP 1513 H=1513:00 +CPU= 18.966
  14:15:25 STEP 1514 H=1514:00 +CPU= 19.528
  14:15:55 STEP 1515 H=1515:00 +CPU= 30.568
  14:16:15 STEP 1516 H=1516:00 +CPU= 19.549
  14:16:34 STEP 1517 H=1517:00 +CPU= 18.951
  14:17:05 STEP 1518 H=1518:00 +CPU= 31.297
  14:17:24 STEP 1519 H=1519:00 +CPU= 18.941
  14:17:44 STEP 1520 H=1520:00 +CPU= 19.579
  14:18:14 STEP 1521 H=1521:00 +CPU= 30.615
  14:18:34 STEP 1522 H=1522:00 +CPU= 19.570
  14:18:53 STEP 1523 H=1523:00 +CPU= 18.952
  14:19:27 STEP 1524 H=1524:00 +CPU= 33.843
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001524
  14:19:46 STEP 1525 H=1525:00 +CPU= 18.965
  14:20:05 STEP 1526 H=1526:00 +CPU= 19.577
  14:20:36 STEP 1527 H=1527:00 +CPU= 30.577
  14:20:55 STEP 1528 H=1528:00 +CPU= 19.538
  14:21:14 STEP 1529 H=1529:00 +CPU= 18.917
  14:21:45 STEP 1530 H=1530:00 +CPU= 31.251
  14:22:04 STEP 1531 H=1531:00 +CPU= 18.911
  14:22:24 STEP 1532 H=1532:00 +CPU= 19.521
  14:22:54 STEP 1533 H=1533:00 +CPU= 30.543
  14:23:14 STEP 1534 H=1534:00 +CPU= 19.509
  14:23:33 STEP 1535 H=1535:00 +CPU= 19.441
  14:24:09 STEP 1536 H=1536:00 +CPU= 35.512
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_63.zip
Uploading the intermediate file: upload_file_63.zip
  14:24:28 STEP 1537 H=1537:00 +CPU= 18.980
  14:24:48 STEP 1538 H=1538:00 +CPU= 19.551
  14:25:18 STEP 1539 H=1539:00 +CPU= 30.574
  14:25:38 STEP 1540 H=1540:00 +CPU= 19.574
  14:25:57 STEP 1541 H=1541:00 +CPU= 18.968
  14:26:28 STEP 1542 H=1542:00 +CPU= 31.319
  14:26:47 STEP 1543 H=1543:00 +CPU= 18.965
  14:27:07 STEP 1544 H=1544:00 +CPU= 19.589
  14:27:37 STEP 1545 H=1545:00 +CPU= 30.614
  14:27:57 STEP 1546 H=1546:00 +CPU= 19.581
  14:28:16 STEP 1547 H=1547:00 +CPU= 18.964
  14:28:50 STEP 1548 H=1548:00 +CPU= 33.797
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001548
  14:29:09 STEP 1549 H=1549:00 +CPU= 18.988
  14:29:28 STEP 1550 H=1550:00 +CPU= 19.617
  14:29:59 STEP 1551 H=1551:00 +CPU= 30.620
  14:30:18 STEP 1552 H=1552:00 +CPU= 19.578
  14:30:37 STEP 1553 H=1553:00 +CPU= 18.945
  14:31:09 STEP 1554 H=1554:00 +CPU= 31.311
  14:31:28 STEP 1555 H=1555:00 +CPU= 18.949
  14:31:47 STEP 1556 H=1556:00 +CPU= 19.556
  14:32:18 STEP 1557 H=1557:00 +CPU= 30.586
  14:32:37 STEP 1558 H=1558:00 +CPU= 19.533
  14:32:57 STEP 1559 H=1559:00 +CPU= 19.470
  14:33:32 STEP 1560 H=1560:00 +CPU= 35.549
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_64.zip
Uploading the intermediate file: upload_file_64.zip
  14:33:51 STEP 1561 H=1561:00 +CPU= 18.993
  14:34:11 STEP 1562 H=1562:00 +CPU= 19.548
  14:34:41 STEP 1563 H=1563:00 +CPU= 30.565
  14:35:01 STEP 1564 H=1564:00 +CPU= 19.558
  14:35:20 STEP 1565 H=1565:00 +CPU= 18.971
  14:35:51 STEP 1566 H=1566:00 +CPU= 31.293
  14:36:10 STEP 1567 H=1567:00 +CPU= 18.957
  14:36:30 STEP 1568 H=1568:00 +CPU= 19.601
  14:37:00 STEP 1569 H=1569:00 +CPU= 30.625
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0046
(argv4) batchid: 972
(argv5) wuid: 12188690
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/2
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_729736f1ea4f53dbf2c4f1102340f758 to: /var/lib/boinc/slots/2/oifs_43r3_ps_0046_2003050100_123_972_12188690.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_0046_2003050100_123_972_12188690.zip
ic_ancil_file: ic_ancil_12188690
ifsdata_file: ifsdata_12188690
climate_data_file: clim_data_12188690
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_179027e2adace701c19cf6e45361391a to: /var/lib/boinc/slots/2/ic_ancil_12188690.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/2/ic_ancil_12188690.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_7b2e728072908e2a4b13b5bd836fff2f to: /var/lib/boinc/slots/2/ifsdata/ifsdata_12188690.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/2/ifsdata/ifsdata_12188690.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/2/159l_2/clim_data_12188690.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/2/159l_2/clim_data_12188690.zip
Checking for progress XML file: /var/lib/boinc/slots/2/progress_file_12188690.xml
Opened progress file ok : /var/lib/boinc/slots/2/progress_file_12188690.xml
-- Model is restarting --
Adjusting last_iter, 1569, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/2/progress_file_12188690.xml
last_cpu_time: 37830
upload_file_number: 65
last_iter: 1559
last_upload: 5616000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200
The child process has been launched with process id: 15314
Executing the command: /var/lib/boinc/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:15314:15314] [20230110:162842:1673393322:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:29:18 STEP 1560 H=1560:00 +CPU= 33.737
  16:29:37 STEP 1561 H=1561:00 +CPU= 18.901
  16:29:56 STEP 1562 H=1562:00 +CPU= 19.486
  16:30:27 STEP 1563 H=1563:00 +CPU= 30.540
  16:30:46 STEP 1564 H=1564:00 +CPU= 19.503
  16:31:05 STEP 1565 H=1565:00 +CPU= 18.903
  16:31:36 STEP 1566 H=1566:00 +CPU= 31.283
  16:31:55 STEP 1567 H=1567:00 +CPU= 18.935
  16:32:15 STEP 1568 H=1568:00 +CPU= 19.550
  16:32:45 STEP 1569 H=1569:00 +CPU= 30.598
  16:33:05 STEP 1570 H=1570:00 +CPU= 19.545
  16:33:24 STEP 1571 H=1571:00 +CPU= 18.934
  16:33:58 STEP 1572 H=1572:00 +CPU= 33.799
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001572
  16:34:17 STEP 1573 H=1573:00 +CPU= 18.927
  16:34:36 STEP 1574 H=1574:00 +CPU= 19.547
  16:35:07 STEP 1575 H=1575:00 +CPU= 30.571
  16:35:26 STEP 1576 H=1576:00 +CPU= 19.516
  16:35:45 STEP 1577 H=1577:00 +CPU= 18.908
  16:36:16 STEP 1578 H=1578:00 +CPU= 31.267
  16:36:35 STEP 1579 H=1579:00 +CPU= 18.902
  16:36:55 STEP 1580 H=1580:00 +CPU= 19.505
  16:37:25 STEP 1581 H=1581:00 +CPU= 30.551
  16:37:45 STEP 1582 H=1582:00 +CPU= 19.493
  16:38:04 STEP 1583 H=1583:00 +CPU= 19.408
  16:38:40 STEP 1584 H=1584:00 +CPU= 35.904
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_65.zip
Uploading the intermediate file: upload_file_65.zip
  16:38:59 STEP 1585 H=1585:00 +CPU= 19.163
  16:39:19 STEP 1586 H=1586:00 +CPU= 19.712
  16:39:50 STEP 1587 H=1587:00 +CPU= 30.958
  16:40:10 STEP 1588 H=1588:00 +CPU= 19.690
  16:40:29 STEP 1589 H=1589:00 +CPU= 19.079
  16:41:00 STEP 1590 H=1590:00 +CPU= 31.678
  16:41:20 STEP 1591 H=1591:00 +CPU= 19.132
  16:41:39 STEP 1592 H=1592:00 +CPU= 19.744
  16:42:10 STEP 1593 H=1593:00 +CPU= 30.989
  16:42:30 STEP 1594 H=1594:00 +CPU= 19.715
  16:42:49 STEP 1595 H=1595:00 +CPU= 19.102
  16:43:23 STEP 1596 H=1596:00 +CPU= 34.192
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001596
  16:43:42 STEP 1597 H=1597:00 +CPU= 19.082
  16:44:02 STEP 1598 H=1598:00 +CPU= 19.709
  16:44:33 STEP 1599 H=1599:00 +CPU= 30.886
  16:44:53 STEP 1600 H=1600:00 +CPU= 19.692
  16:45:12 STEP 1601 H=1601:00 +CPU= 19.058
  16:45:43 STEP 1602 H=1602:00 +CPU= 31.613
  16:46:02 STEP 1603 H=1603:00 +CPU= 19.046
  16:46:22 STEP 1604 H=1604:00 +CPU= 19.670
  16:46:53 STEP 1605 H=1605:00 +CPU= 30.910
  16:47:13 STEP 1606 H=1606:00 +CPU= 19.657
  16:47:32 STEP 1607 H=1607:00 +CPU= 19.591
  16:48:08 STEP 1608 H=1608:00 +CPU= 35.549
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_66.zip
Uploading the intermediate file: upload_file_66.zip
  16:48:27 STEP 1609 H=1609:00 +CPU= 18.981
  16:48:46 STEP 1610 H=1610:00 +CPU= 19.563
  16:49:17 STEP 1611 H=1611:00 +CPU= 30.614
  16:49:37 STEP 1612 H=1612:00 +CPU= 19.546
  16:49:56 STEP 1613 H=1613:00 +CPU= 18.948
  16:50:27 STEP 1614 H=1614:00 +CPU= 31.314
  16:50:46 STEP 1615 H=1615:00 +CPU= 18.974
  16:51:05 STEP 1616 H=1616:00 +CPU= 19.590
  16:51:36 STEP 1617 H=1617:00 +CPU= 30.605
  16:51:56 STEP 1618 H=1618:00 +CPU= 19.588
  16:52:15 STEP 1619 H=1619:00 +CPU= 18.970
  16:52:48 STEP 1620 H=1620:00 +CPU= 33.861
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001620
  16:53:07 STEP 1621 H=1621:00 +CPU= 18.983
  16:53:27 STEP 1622 H=1622:00 +CPU= 19.592
  16:53:58 STEP 1623 H=1623:00 +CPU= 30.620
  16:54:17 STEP 1624 H=1624:00 +CPU= 19.574
  16:54:36 STEP 1625 H=1625:00 +CPU= 18.937
  16:55:08 STEP 1626 H=1626:00 +CPU= 31.330
  16:55:26 STEP 1627 H=1627:00 +CPU= 18.931
  16:55:46 STEP 1628 H=1628:00 +CPU= 19.531
  16:56:17 STEP 1629 H=1629:00 +CPU= 30.553
  16:56:36 STEP 1630 H=1630:00 +CPU= 19.502
  16:56:56 STEP 1631 H=1631:00 +CPU= 19.432
  16:57:31 STEP 1632 H=1632:00 +CPU= 35.832
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_67.zip
Uploading the intermediate file: upload_file_67.zip
  16:57:50 STEP 1633 H=1633:00 +CPU= 19.121
  16:58:10 STEP 1634 H=1634:00 +CPU= 19.642
  16:58:41 STEP 1635 H=1635:00 +CPU= 30.867
  16:59:01 STEP 1636 H=1636:00 +CPU= 19.665
  16:59:20 STEP 1637 H=1637:00 +CPU= 19.047
  16:59:51 STEP 1638 H=1638:00 +CPU= 31.545
  17:00:10 STEP 1639 H=1639:00 +CPU= 19.013
  17:00:30 STEP 1640 H=1640:00 +CPU= 19.602
  17:01:01 STEP 1641 H=1641:00 +CPU= 30.662
  17:01:20 STEP 1642 H=1642:00 +CPU= 19.591
  17:01:39 STEP 1643 H=1643:00 +CPU= 18.974
  17:02:13 STEP 1644 H=1644:00 +CPU= 33.838
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001644
  17:02:32 STEP 1645 H=1645:00 +CPU= 18.991
  17:02:52 STEP 1646 H=1646:00 +CPU= 19.602
  17:03:22 STEP 1647 H=1647:00 +CPU= 30.619
  17:03:42 STEP 1648 H=1648:00 +CPU= 19.572
  17:04:01 STEP 1649 H=1649:00 +CPU= 18.946
  17:04:32 STEP 1650 H=1650:00 +CPU= 31.300
  17:04:51 STEP 1651 H=1651:00 +CPU= 18.936
  17:05:10 STEP 1652 H=1652:00 +CPU= 19.550
  17:05:41 STEP 1653 H=1653:00 +CPU= 30.614
  17:06:01 STEP 1654 H=1654:00 +CPU= 19.534
  17:06:20 STEP 1655 H=1655:00 +CPU= 19.468
  17:06:56 STEP 1656 H=1656:00 +CPU= 35.569
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_68.zip
Uploading the intermediate file: upload_file_68.zip
  17:07:15 STEP 1657 H=1657:00 +CPU= 19.036
  17:07:34 STEP 1658 H=1658:00 +CPU= 19.610
  17:08:05 STEP 1659 H=1659:00 +CPU= 30.604
  17:08:25 STEP 1660 H=1660:00 +CPU= 19.557
  17:08:43 STEP 1661 H=1661:00 +CPU= 18.943
  17:09:15 STEP 1662 H=1662:00 +CPU= 31.321
  17:09:34 STEP 1663 H=1663:00 +CPU= 18.961
  17:09:53 STEP 1664 H=1664:00 +CPU= 19.590
  17:10:24 STEP 1665 H=1665:00 +CPU= 30.638
  17:10:44 STEP 1666 H=1666:00 +CPU= 19.590
  17:11:03 STEP 1667 H=1667:00 +CPU= 18.969
  17:11:36 STEP 1668 H=1668:00 +CPU= 33.809
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001668
  17:11:55 STEP 1669 H=1669:00 +CPU= 18.929
  17:12:15 STEP 1670 H=1670:00 +CPU= 19.552
  17:12:45 STEP 1671 H=1671:00 +CPU= 30.525
  17:13:05 STEP 1672 H=1672:00 +CPU= 19.508
  17:13:24 STEP 1673 H=1673:00 +CPU= 18.884
  17:13:55 STEP 1674 H=1674:00 +CPU= 31.230
  17:14:14 STEP 1675 H=1675:00 +CPU= 18.877
  17:14:33 STEP 1676 H=1676:00 +CPU= 19.492
  17:15:04 STEP 1677 H=1677:00 +CPU= 30.546
  17:15:23 STEP 1678 H=1678:00 +CPU= 19.500
  17:15:43 STEP 1679 H=1679:00 +CPU= 19.422
  17:16:18 STEP 1680 H=1680:00 +CPU= 35.584
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_69.zip
Uploading the intermediate file: upload_file_69.zip
  17:16:37 STEP 1681 H=1681:00 +CPU= 19.020
  17:16:57 STEP 1682 H=1682:00 +CPU= 19.579
  17:17:28 STEP 1683 H=1683:00 +CPU= 30.578
  17:17:47 STEP 1684 H=1684:00 +CPU= 19.540
  17:18:06 STEP 1685 H=1685:00 +CPU= 18.942
  17:18:37 STEP 1686 H=1686:00 +CPU= 31.324
  17:18:56 STEP 1687 H=1687:00 +CPU= 18.940
  17:19:16 STEP 1688 H=1688:00 +CPU= 19.567
  17:19:47 STEP 1689 H=1689:00 +CPU= 30.599
  17:20:06 STEP 1690 H=1690:00 +CPU= 19.577
  17:20:25 STEP 1691 H=1691:00 +CPU= 18.948
  17:20:59 STEP 1692 H=1692:00 +CPU= 33.803
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001692
  17:21:18 STEP 1693 H=1693:00 +CPU= 18.949
  17:21:37 STEP 1694 H=1694:00 +CPU= 19.571
  17:22:08 STEP 1695 H=1695:00 +CPU= 30.610
  17:22:28 STEP 1696 H=1696:00 +CPU= 19.549
  17:22:47 STEP 1697 H=1697:00 +CPU= 18.947
  17:23:18 STEP 1698 H=1698:00 +CPU= 31.297
  17:23:37 STEP 1699 H=1699:00 +CPU= 18.916
  17:23:56 STEP 1700 H=1700:00 +CPU= 19.526
  17:24:27 STEP 1701 H=1701:00 +CPU= 30.545
  17:24:46 STEP 1702 H=1702:00 +CPU= 19.509
  17:25:06 STEP 1703 H=1703:00 +CPU= 19.442
  17:25:41 STEP 1704 H=1704:00 +CPU= 35.629
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_70.zip
Uploading the intermediate file: upload_file_70.zip
  17:26:00 STEP 1705 H=1705:00 +CPU= 18.937
  17:26:20 STEP 1706 H=1706:00 +CPU= 19.548
  17:26:51 STEP 1707 H=1707:00 +CPU= 30.607
  17:27:10 STEP 1708 H=1708:00 +CPU= 19.531
  17:27:29 STEP 1709 H=1709:00 +CPU= 18.923
  17:28:00 STEP 1710 H=1710:00 +CPU= 31.296
  17:28:19 STEP 1711 H=1711:00 +CPU= 18.953
  17:28:39 STEP 1712 H=1712:00 +CPU= 19.567
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0046
(argv4) batchid: 972
(argv5) wuid: 12188690
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/2
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_729736f1ea4f53dbf2c4f1102340f758 to: /var/lib/boinc/slots/2/oifs_43r3_ps_0046_2003050100_123_972_12188690.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_0046_2003050100_123_972_12188690.zip
ic_ancil_file: ic_ancil_12188690
ifsdata_file: ifsdata_12188690
climate_data_file: clim_data_12188690
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_179027e2adace701c19cf6e45361391a to: /var/lib/boinc/slots/2/ic_ancil_12188690.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/2/ic_ancil_12188690.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_7b2e728072908e2a4b13b5bd836fff2f to: /var/lib/boinc/slots/2/ifsdata/ifsdata_12188690.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/2/ifsdata/ifsdata_12188690.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/2/159l_2/clim_data_12188690.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/2/159l_2/clim_data_12188690.zip
Checking for progress XML file: /var/lib/boinc/slots/2/progress_file_12188690.xml
Opened progress file ok : /var/lib/boinc/slots/2/progress_file_12188690.xml
-- Model is restarting --
Adjusting last_iter, 1713, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/2/progress_file_12188690.xml
last_cpu_time: 41446
upload_file_number: 71
last_iter: 1703
last_upload: 6134400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200
The child process has been launched with process id: 15925
Executing the command: /var/lib/boinc/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:15925:15925] [20230110:190721:1673402841:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:07:56 STEP 1704 H=1704:00 +CPU= 33.910
  19:08:15 STEP 1705 H=1705:00 +CPU= 18.901
  19:08:35 STEP 1706 H=1706:00 +CPU= 19.489
  19:09:05 STEP 1707 H=1707:00 +CPU= 30.595
  19:09:25 STEP 1708 H=1708:00 +CPU= 19.510
  19:09:44 STEP 1709 H=1709:00 +CPU= 18.905
  19:10:15 STEP 1710 H=1710:00 +CPU= 31.324
  19:10:34 STEP 1711 H=1711:00 +CPU= 18.922
  19:10:54 STEP 1712 H=1712:00 +CPU= 19.542
  19:11:24 STEP 1713 H=1713:00 +CPU= 30.606
  19:11:44 STEP 1714 H=1714:00 +CPU= 19.552
  19:12:03 STEP 1715 H=1715:00 +CPU= 18.917
  19:12:37 STEP 1716 H=1716:00 +CPU= 33.770
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001716
  19:12:56 STEP 1717 H=1717:00 +CPU= 18.955
  19:13:15 STEP 1718 H=1718:00 +CPU= 19.569
  19:13:46 STEP 1719 H=1719:00 +CPU= 30.607
  19:14:05 STEP 1720 H=1720:00 +CPU= 19.547
  19:14:24 STEP 1721 H=1721:00 +CPU= 18.928
  19:14:56 STEP 1722 H=1722:00 +CPU= 31.318
  19:15:14 STEP 1723 H=1723:00 +CPU= 18.937
  19:15:34 STEP 1724 H=1724:00 +CPU= 19.539
  19:16:05 STEP 1725 H=1725:00 +CPU= 30.597
  19:16:24 STEP 1726 H=1726:00 +CPU= 19.522
  19:16:44 STEP 1727 H=1727:00 +CPU= 19.424
  19:17:20 STEP 1728 H=1728:00 +CPU= 35.945
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_71.zip
Uploading the intermediate file: upload_file_71.zip
  19:17:39 STEP 1729 H=1729:00 +CPU= 19.120
  19:17:58 STEP 1730 H=1730:00 +CPU= 19.637
  19:18:29 STEP 1731 H=1731:00 +CPU= 30.706
  19:18:49 STEP 1732 H=1732:00 +CPU= 19.622
  19:19:08 STEP 1733 H=1733:00 +CPU= 18.997
  19:19:39 STEP 1734 H=1734:00 +CPU= 31.417
  19:19:58 STEP 1735 H=1735:00 +CPU= 19.017
  19:20:18 STEP 1736 H=1736:00 +CPU= 19.642
  19:20:48 STEP 1737 H=1737:00 +CPU= 30.707
  19:21:08 STEP 1738 H=1738:00 +CPU= 19.608
  19:21:27 STEP 1739 H=1739:00 +CPU= 18.997
  19:22:01 STEP 1740 H=1740:00 +CPU= 33.886
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001740
  19:22:20 STEP 1741 H=1741:00 +CPU= 18.997
  19:22:40 STEP 1742 H=1742:00 +CPU= 19.609
  19:23:10 STEP 1743 H=1743:00 +CPU= 30.677
  19:23:30 STEP 1744 H=1744:00 +CPU= 19.591
  19:23:49 STEP 1745 H=1745:00 +CPU= 18.970
  19:24:20 STEP 1746 H=1746:00 +CPU= 31.387
  19:24:39 STEP 1747 H=1747:00 +CPU= 18.989
  19:24:59 STEP 1748 H=1748:00 +CPU= 19.565
  19:25:29 STEP 1749 H=1749:00 +CPU= 30.656
  19:25:49 STEP 1750 H=1750:00 +CPU= 19.574
  19:26:08 STEP 1751 H=1751:00 +CPU= 19.481
  19:26:44 STEP 1752 H=1752:00 +CPU= 35.575
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_72.zip
Uploading the intermediate file: upload_file_72.zip
  19:27:03 STEP 1753 H=1753:00 +CPU= 18.962
  19:27:22 STEP 1754 H=1754:00 +CPU= 19.514
  19:27:53 STEP 1755 H=1755:00 +CPU= 30.589
  19:28:13 STEP 1756 H=1756:00 +CPU= 19.522
  19:28:32 STEP 1757 H=1757:00 +CPU= 18.916
  19:29:03 STEP 1758 H=1758:00 +CPU= 31.344
  19:29:22 STEP 1759 H=1759:00 +CPU= 18.928
  19:29:41 STEP 1760 H=1760:00 +CPU= 19.548
  19:30:12 STEP 1761 H=1761:00 +CPU= 30.637
  19:30:32 STEP 1762 H=1762:00 +CPU= 19.566
  19:30:51 STEP 1763 H=1763:00 +CPU= 18.951
  19:31:24 STEP 1764 H=1764:00 +CPU= 33.831
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001764
  19:31:43 STEP 1765 H=1765:00 +CPU= 18.940
  19:32:03 STEP 1766 H=1766:00 +CPU= 19.559
  19:32:33 STEP 1767 H=1767:00 +CPU= 30.632
  19:32:53 STEP 1768 H=1768:00 +CPU= 19.575
  19:33:12 STEP 1769 H=1769:00 +CPU= 18.936
  19:33:43 STEP 1770 H=1770:00 +CPU= 31.323
  19:34:02 STEP 1771 H=1771:00 +CPU= 18.913
  19:34:22 STEP 1772 H=1772:00 +CPU= 19.528
  19:34:52 STEP 1773 H=1773:00 +CPU= 30.614
  19:35:12 STEP 1774 H=1774:00 +CPU= 19.533
  19:35:31 STEP 1775 H=1775:00 +CPU= 19.443
  19:36:07 STEP 1776 H=1776:00 +CPU= 35.658
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_73.zip
Uploading the intermediate file: upload_file_73.zip
  19:36:26 STEP 1777 H=1777:00 +CPU= 18.990
  19:36:46 STEP 1778 H=1778:00 +CPU= 19.555
  19:37:16 STEP 1779 H=1779:00 +CPU= 30.640
  19:37:36 STEP 1780 H=1780:00 +CPU= 19.548
  19:37:55 STEP 1781 H=1781:00 +CPU= 18.940
  19:38:26 STEP 1782 H=1782:00 +CPU= 31.352
  19:38:45 STEP 1783 H=1783:00 +CPU= 18.957
  19:39:05 STEP 1784 H=1784:00 +CPU= 19.585
  19:39:35 STEP 1785 H=1785:00 +CPU= 30.667
  19:39:55 STEP 1786 H=1786:00 +CPU= 19.599
  19:40:14 STEP 1787 H=1787:00 +CPU= 19.015
  19:40:48 STEP 1788 H=1788:00 +CPU= 33.907
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001788
  19:41:07 STEP 1789 H=1789:00 +CPU= 18.980
  19:41:26 STEP 1790 H=1790:00 +CPU= 19.601
  19:41:57 STEP 1791 H=1791:00 +CPU= 30.691
  19:42:17 STEP 1792 H=1792:00 +CPU= 19.562
  19:42:36 STEP 1793 H=1793:00 +CPU= 18.941
  19:43:07 STEP 1794 H=1794:00 +CPU= 31.340
  19:43:26 STEP 1795 H=1795:00 +CPU= 18.925
  19:43:45 STEP 1796 H=1796:00 +CPU= 19.535
  19:44:16 STEP 1797 H=1797:00 +CPU= 30.643
  19:44:36 STEP 1798 H=1798:00 +CPU= 19.542
  19:44:55 STEP 1799 H=1799:00 +CPU= 19.459
  19:45:31 STEP 1800 H=1800:00 +CPU= 35.634
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_74.zip
Uploading the intermediate file: upload_file_74.zip
  19:45:50 STEP 1801 H=1801:00 +CPU= 18.977
  19:46:09 STEP 1802 H=1802:00 +CPU= 19.548
  19:46:40 STEP 1803 H=1803:00 +CPU= 30.693
  19:46:59 STEP 1804 H=1804:00 +CPU= 19.559
  19:47:18 STEP 1805 H=1805:00 +CPU= 18.942
  19:47:50 STEP 1806 H=1806:00 +CPU= 31.376
  19:48:09 STEP 1807 H=1807:00 +CPU= 18.982
  19:48:28 STEP 1808 H=1808:00 +CPU= 19.588
  19:48:59 STEP 1809 H=1809:00 +CPU= 30.675
  19:49:19 STEP 1810 H=1810:00 +CPU= 19.593
  19:49:38 STEP 1811 H=1811:00 +CPU= 18.974
  19:50:12 STEP 1812 H=1812:00 +CPU= 33.874
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001812
  19:50:30 STEP 1813 H=1813:00 +CPU= 18.973
  19:50:50 STEP 1814 H=1814:00 +CPU= 19.591
  19:51:21 STEP 1815 H=1815:00 +CPU= 30.699
  19:51:40 STEP 1816 H=1816:00 +CPU= 19.596
  19:51:59 STEP 1817 H=1817:00 +CPU= 18.957
  19:52:31 STEP 1818 H=1818:00 +CPU= 31.391
  19:52:50 STEP 1819 H=1819:00 +CPU= 18.940
  19:53:09 STEP 1820 H=1820:00 +CPU= 19.571
  19:53:40 STEP 1821 H=1821:00 +CPU= 30.701
  19:54:00 STEP 1822 H=1822:00 +CPU= 19.543
  19:54:19 STEP 1823 H=1823:00 +CPU= 19.473
  19:54:55 STEP 1824 H=1824:00 +CPU= 35.707
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_75.zip
Uploading the intermediate file: upload_file_75.zip
  19:55:14 STEP 1825 H=1825:00 +CPU= 18.998
  19:55:33 STEP 1826 H=1826:00 +CPU= 19.572
  19:56:04 STEP 1827 H=1827:00 +CPU= 30.708
  19:56:24 STEP 1828 H=1828:00 +CPU= 19.582
  19:56:43 STEP 1829 H=1829:00 +CPU= 18.964
  19:57:14 STEP 1830 H=1830:00 +CPU= 31.400
  19:57:33 STEP 1831 H=1831:00 +CPU= 18.981
  19:57:53 STEP 1832 H=1832:00 +CPU= 19.607
  19:58:23 STEP 1833 H=1833:00 +CPU= 30.708
  19:58:43 STEP 1834 H=1834:00 +CPU= 19.637
  19:59:02 STEP 1835 H=1835:00 +CPU= 18.984
  19:59:36 STEP 1836 H=1836:00 +CPU= 33.905
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001836
  19:59:55 STEP 1837 H=1837:00 +CPU= 18.996
  20:00:14 STEP 1838 H=1838:00 +CPU= 19.623
  20:00:45 STEP 1839 H=1839:00 +CPU= 30.700
  20:01:05 STEP 1840 H=1840:00 +CPU= 19.589
  20:01:24 STEP 1841 H=1841:00 +CPU= 18.989
  20:01:55 STEP 1842 H=1842:00 +CPU= 31.389
  20:02:14 STEP 1843 H=1843:00 +CPU= 18.959
  20:02:34 STEP 1844 H=1844:00 +CPU= 19.587
  20:03:04 STEP 1845 H=1845:00 +CPU= 30.685
  20:03:24 STEP 1846 H=1846:00 +CPU= 19.555
  20:03:43 STEP 1847 H=1847:00 +CPU= 19.501
  20:04:19 STEP 1848 H=1848:00 +CPU= 35.708
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200_76.zip
Uploading the intermediate file: upload_file_76.zip
  20:04:38 STEP 1849 H=1849:00 +CPU= 18.979
  20:04:58 STEP 1850 H=1850:00 +CPU= 19.554
  20:05:28 STEP 1851 H=1851:00 +CPU= 30.676
  20:05:48 STEP 1852 H=1852:00 +CPU= 19.569
  20:06:07 STEP 1853 H=1853:00 +CPU= 18.961
  20:06:38 STEP 1854 H=1854:00 +CPU= 31.428
  20:06:57 STEP 1855 H=1855:00 +CPU= 18.997
  20:07:17 STEP 1856 H=1856:00 +CPU= 19.614
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0046
(argv4) batchid: 972
(argv5) wuid: 12188690
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/2
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12188690
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_729736f1ea4f53dbf2c4f1102340f758 to: /var/lib/boinc/slots/2/oifs_43r3_ps_0046_2003050100_123_972_12188690.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_0046_2003050100_123_972_12188690.zip
ic_ancil_file: ic_ancil_12188690
ifsdata_file: ifsdata_12188690
climate_data_file: clim_data_12188690
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_179027e2adace701c19cf6e45361391a to: /var/lib/boinc/slots/2/ic_ancil_12188690.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/2/ic_ancil_12188690.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_7b2e728072908e2a4b13b5bd836fff2f to: /var/lib/boinc/slots/2/ifsdata/ifsdata_12188690.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/2/ifsdata/ifsdata_12188690.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/2/159l_2/clim_data_12188690.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/2/159l_2/clim_data_12188690.zip
Checking for progress XML file: /var/lib/boinc/slots/2/progress_file_12188690.xml
Opened progress file ok : /var/lib/boinc/slots/2/progress_file_12188690.xml
-- Model is restarting --
Adjusting last_iter, 1857, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/2/progress_file_12188690.xml
last_cpu_time: 45062
upload_file_number: 77
last_iter: 1847
last_upload: 6652800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0046_2003050100_123_972_12188690_0_r141088200
The child process has been launched with process id: 16599
Executing the command: /var/lib/boinc/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:16599:16599] [20230110:220018:1673413218:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:00:55 STEP 1848 H=1848:00 +CPU= 35.261
  22:01:15 STEP 1849 H=1849:00 +CPU= 19.677
  22:01:35 STEP 1850 H=1850:00 +CPU= 20.062
  22:02:06 STEP 1851 H=1851:00 +CPU= 31.581
  22:02:26 STEP 1852 H=1852:00 +CPU= 19.800
  22:02:45 STEP 1853 H=1853:00 +CPU= 19.118
  22:03:17 STEP 1854 H=1854:00 +CPU= 31.735
  22:03:36 STEP 1855 H=1855:00 +CPU= 19.104
  22:03:56 STEP 1856 H=1856:00 +CPU= 19.708
  22:04:27 STEP 1857 H=1857:00 +CPU= 30.996
  22:04:46 STEP 1858 H=1858:00 +CPU= 19.722
  22:05:06 STEP 1859 H=1859:00 +CPU= 19.104
  22:05:40 STEP 1860 H=1860:00 +CPU= 34.193
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001860
  22:05:59 STEP 1861 H=1861:00 +CPU= 19.098
  22:06:19 STEP 1862 H=1862:00 +CPU= 19.725
  22:06:49 STEP 1863 H=1863:00 +CPU= 30.900
  22:07:09 STEP 1864 H=1864:00 +CPU= 19.701
  22:07:28 STEP 1865 H=1865:00 +CPU= 19.103
  22:08:00 STEP 1866 H=1866:00 +CPU= 31.669
  22:08:19 STEP 1867 H=1867:00 +CPU= 19.080
  22:08:39 STEP 1868 H=1868:00 +CPU= 19.674
  22:09:10 STEP 1869 H=1869:00 +CPU= 30.942
  22:09:29 STEP 1870 H=1870:00 +CPU= 19.652
  22:09:49 STEP 1871 H=1871:00 +CPU= 19.572
  22:10:25 STEP 1872 H=1872:00 +CPU= 36.073
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:14:49 1463165 22274187 oifs_43r3_ps_0046_2003050100_123_972_12188690_0 4,406,400 29,184 0.0066
09 May 2023 16:14:45 1463165 22274187 oifs_43r3_ps_0046_2003050100_123_972_12188690_0 3,542,400 23,563 0.0067
09 May 2023 16:14:42 1463165 22274187 oifs_43r3_ps_0046_2003050100_123_972_12188690_0 2,678,400 17,941 0.0067
09 May 2023 16:14:34 1463165 22274187 oifs_43r3_ps_0046_2003050100_123_972_12188690_0 1,728,000 11,251 0.0065
09 May 2023 16:14:30 1463165 22274187 oifs_43r3_ps_0046_2003050100_123_972_12188690_0 864,000 5,649 0.0065


©2024 climateprediction.net