climateprediction.net home page
Task 22305539

Task 22305539

Name oifs_43r3_ps_0657_1987050100_123_956_12173301_2
Workunit 12173301
Created 4 Feb 2023, 14:11:39 UTC
Sent 4 Feb 2023, 14:13:24 UTC
Report deadline 5 Apr 2023, 14:13:24 UTC
Received 5 Feb 2023, 9:08:04 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1534903
Run time 16 hours 54 min 10 sec
CPU time 16 hours 37 min 54 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 5.25 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,039.80 MB
Peak swap size 4,978.07 MB
Peak disk usage 2,822.25 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<stderr_txt>
le_103.zip
  05:12:40 STEP 2497 H=2497:00 +CPU= 15.697
  05:12:57 STEP 2498 H=2498:00 +CPU= 16.332
  05:13:21 STEP 2499 H=2499:00 +CPU= 24.163
  05:13:37 STEP 2500 H=2500:00 +CPU= 16.090
  05:13:53 STEP 2501 H=2501:00 +CPU= 15.950
  05:14:17 STEP 2502 H=2502:00 +CPU= 24.473
  05:14:33 STEP 2503 H=2503:00 +CPU= 15.890
  05:14:49 STEP 2504 H=2504:00 +CPU= 16.315
  05:15:14 STEP 2505 H=2505:00 +CPU= 24.033
  05:15:30 STEP 2506 H=2506:00 +CPU= 16.107
  05:15:47 STEP 2507 H=2507:00 +CPU= 16.159
  05:16:13 STEP 2508 H=2508:00 +CPU= 26.503
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002508
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002508
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002508
  05:16:29 STEP 2509 H=2509:00 +CPU= 15.694
  05:16:45 STEP 2510 H=2510:00 +CPU= 16.302
  05:17:09 STEP 2511 H=2511:00 +CPU= 24.094
  05:17:25 STEP 2512 H=2512:00 +CPU= 16.020
  05:17:41 STEP 2513 H=2513:00 +CPU= 15.910
  05:18:06 STEP 2514 H=2514:00 +CPU= 24.432
  05:18:21 STEP 2515 H=2515:00 +CPU= 15.657
  05:18:38 STEP 2516 H=2516:00 +CPU= 16.284
  05:19:02 STEP 2517 H=2517:00 +CPU= 24.053
  05:19:17 STEP 2518 H=2518:00 +CPU= 15.727
  05:19:34 STEP 2519 H=2519:00 +CPU= 16.585
  05:20:03 STEP 2520 H=2520:00 +CPU= 28.444
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002520
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002520
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002520
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002496
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002496
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002496
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002508
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002508
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002508
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_104.zip
Uploading the intermediate file: upload_file_104.zip
  05:20:19 STEP 2521 H=2521:00 +CPU= 15.738
  05:20:35 STEP 2522 H=2522:00 +CPU= 16.534
  05:20:59 STEP 2523 H=2523:00 +CPU= 24.076
  05:21:15 STEP 2524 H=2524:00 +CPU= 16.143
  05:21:32 STEP 2525 H=2525:00 +CPU= 16.096
  05:21:56 STEP 2526 H=2526:00 +CPU= 24.663
  05:22:12 STEP 2527 H=2527:00 +CPU= 15.749
  05:22:29 STEP 2528 H=2528:00 +CPU= 16.378
  05:22:53 STEP 2529 H=2529:00 +CPU= 24.288
  05:23:09 STEP 2530 H=2530:00 +CPU= 16.419
  05:23:25 STEP 2531 H=2531:00 +CPU= 16.067
  05:23:52 STEP 2532 H=2532:00 +CPU= 26.546
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002532
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002532
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002532
  05:24:08 STEP 2533 H=2533:00 +CPU= 15.710
  05:24:24 STEP 2534 H=2534:00 +CPU= 16.423
  05:24:48 STEP 2535 H=2535:00 +CPU= 23.952
  05:25:04 STEP 2536 H=2536:00 +CPU= 16.024
  05:25:20 STEP 2537 H=2537:00 +CPU= 15.949
  05:25:45 STEP 2538 H=2538:00 +CPU= 24.468
  05:26:00 STEP 2539 H=2539:00 +CPU= 15.777
  05:26:17 STEP 2540 H=2540:00 +CPU= 16.362
  05:26:41 STEP 2541 H=2541:00 +CPU= 24.117
  05:26:57 STEP 2542 H=2542:00 +CPU= 15.941
  05:27:14 STEP 2543 H=2543:00 +CPU= 16.610
  05:27:42 STEP 2544 H=2544:00 +CPU= 28.550
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002544
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002544
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002544
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002520
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002520
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002520
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002532
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002532
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002532
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_105.zip
Uploading the intermediate file: upload_file_105.zip
  05:27:58 STEP 2545 H=2545:00 +CPU= 15.785
  05:28:15 STEP 2546 H=2546:00 +CPU= 16.401
  05:28:39 STEP 2547 H=2547:00 +CPU= 23.914
  05:28:55 STEP 2548 H=2548:00 +CPU= 16.173
  05:29:11 STEP 2549 H=2549:00 +CPU= 15.959
  05:29:35 STEP 2550 H=2550:00 +CPU= 24.493
  05:29:51 STEP 2551 H=2551:00 +CPU= 15.759
  05:30:07 STEP 2552 H=2552:00 +CPU= 16.369
  05:30:31 STEP 2553 H=2553:00 +CPU= 24.011
  05:30:48 STEP 2554 H=2554:00 +CPU= 16.272
  05:31:05 STEP 2555 H=2555:00 +CPU= 16.359
  05:31:32 STEP 2556 H=2556:00 +CPU= 26.431
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002556
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002556
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002556
  05:31:48 STEP 2557 H=2557:00 +CPU= 15.797
  05:32:04 STEP 2558 H=2558:00 +CPU= 16.373
  05:32:28 STEP 2559 H=2559:00 +CPU= 23.842
  05:32:44 STEP 2560 H=2560:00 +CPU= 16.199
  05:33:00 STEP 2561 H=2561:00 +CPU= 15.886
  05:33:24 STEP 2562 H=2562:00 +CPU= 24.288
  05:33:40 STEP 2563 H=2563:00 +CPU= 15.772
  05:33:56 STEP 2564 H=2564:00 +CPU= 16.249
  05:34:20 STEP 2565 H=2565:00 +CPU= 23.994
  05:34:36 STEP 2566 H=2566:00 +CPU= 15.935
  05:34:53 STEP 2567 H=2567:00 +CPU= 16.513
  05:35:21 STEP 2568 H=2568:00 +CPU= 28.361
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002568
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002568
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002568
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002544
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002544
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002544
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002556
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002556
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002556
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_106.zip
Uploading the intermediate file: upload_file_106.zip
  05:35:37 STEP 2569 H=2569:00 +CPU= 15.790
  05:35:54 STEP 2570 H=2570:00 +CPU= 16.312
  05:36:18 STEP 2571 H=2571:00 +CPU= 23.987
  05:36:34 STEP 2572 H=2572:00 +CPU= 16.152
  05:36:50 STEP 2573 H=2573:00 +CPU= 15.930
  05:37:14 STEP 2574 H=2574:00 +CPU= 24.396
  05:37:30 STEP 2575 H=2575:00 +CPU= 15.726
  05:37:46 STEP 2576 H=2576:00 +CPU= 16.318
  05:38:10 STEP 2577 H=2577:00 +CPU= 23.912
  05:38:26 STEP 2578 H=2578:00 +CPU= 15.959
  05:38:42 STEP 2579 H=2579:00 +CPU= 15.968
  05:39:09 STEP 2580 H=2580:00 +CPU= 27.188
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002580
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002580
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002580
  05:39:25 STEP 2581 H=2581:00 +CPU= 15.759
  05:39:42 STEP 2582 H=2582:00 +CPU= 16.656
  05:40:25 STEP 2583 H=2583:00 +CPU= 29.484
  05:40:58 STEP 2584 H=2584:00 +CPU= 20.339
  05:41:23 STEP 2585 H=2585:00 +CPU= 16.835
  05:41:48 STEP 2586 H=2586:00 +CPU= 24.326
  05:42:04 STEP 2587 H=2587:00 +CPU= 16.154
  05:42:32 STEP 2588 H=2588:00 +CPU= 18.335
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0657
(argv4) batchid: 956
(argv5) wuid: 12173301
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /scratch/BOINC/slots/3
Project directory is: /scratch/BOINC/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301
..mkdir for temp folder for results failed
Copying: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /scratch/BOINC/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /scratch/BOINC/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b46540d84f1e4feff3fde9fe07614147 to: /scratch/BOINC/slots/3/oifs_43r3_ps_0657_1987050100_123_956_12173301.zip
Unzipping the namelist zip file: /scratch/BOINC/slots/3/oifs_43r3_ps_0657_1987050100_123_956_12173301.zip
ic_ancil_file: ic_ancil_12173301
ifsdata_file: ifsdata_12173301
climate_data_file: clim_data_12173301
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ed459df37b2246efb2a1cd3eb160e65e to: /scratch/BOINC/slots/3/ic_ancil_12173301.zip
Unzipping the IC ancils zip file: /scratch/BOINC/slots/3/ic_ancil_12173301.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f6aa7b4d431ad4008ad55c1c99163e64 to: /scratch/BOINC/slots/3/ifsdata/ifsdata_12173301.zip
Unzipping the ifsdata_zip file: /scratch/BOINC/slots/3/ifsdata/ifsdata_12173301.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /scratch/BOINC/slots/3/159l_2/clim_data_12173301.zip
Unzipping the climate data zip file: /scratch/BOINC/slots/3/159l_2/clim_data_12173301.zip
Checking for progress XML file: /scratch/BOINC/slots/3/progress_file_12173301.xml
Opened progress file ok : /scratch/BOINC/slots/3/progress_file_12173301.xml
-- Model is restarting --
Adjusting last_iter, 2589, to previous model restart step.
Creating progress file: /scratch/BOINC/slots/3/progress_file_12173301.xml
last_cpu_time: 51726
upload_file_number: 107
last_iter: 2567
last_upload: 9244800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632
The child process has been launched with process id: 19026
Executing the command: /scratch/BOINC/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vaals:1:1:19026:19026] [20230205:054740:1675572460:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:48:17 STEP 2568 H=2568:00 +CPU= 30.168
  05:48:50 STEP 2569 H=2569:00 +CPU= 17.050
  05:49:06 STEP 2570 H=2570:00 +CPU= 16.542
  05:49:30 STEP 2571 H=2571:00 +CPU= 23.888
  05:49:48 STEP 2572 H=2572:00 +CPU= 17.095
  05:50:05 STEP 2573 H=2573:00 +CPU= 17.499
  05:50:31 STEP 2574 H=2574:00 +CPU= 26.134
  05:50:49 STEP 2575 H=2575:00 +CPU= 17.381
  05:51:06 STEP 2576 H=2576:00 +CPU= 17.489
  05:51:33 STEP 2577 H=2577:00 +CPU= 25.672
  05:51:51 STEP 2578 H=2578:00 +CPU= 17.837
  05:52:08 STEP 2579 H=2579:00 +CPU= 17.517
  05:52:37 STEP 2580 H=2580:00 +CPU= 28.761
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002580
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002580
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002580
  05:52:55 STEP 2581 H=2581:00 +CPU= 17.419
  05:53:13 STEP 2582 H=2582:00 +CPU= 17.715
  05:53:40 STEP 2583 H=2583:00 +CPU= 25.861
  05:53:58 STEP 2584 H=2584:00 +CPU= 17.819
  05:54:15 STEP 2585 H=2585:00 +CPU= 17.225
  05:54:42 STEP 2586 H=2586:00 +CPU= 26.337
  05:54:59 STEP 2587 H=2587:00 +CPU= 17.183
  05:55:17 STEP 2588 H=2588:00 +CPU= 17.799
  05:55:42 STEP 2589 H=2589:00 +CPU= 25.643
  05:56:00 STEP 2590 H=2590:00 +CPU= 17.923
  05:56:19 STEP 2591 H=2591:00 +CPU= 18.044
  05:56:52 STEP 2592 H=2592:00 +CPU= 30.904
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002592
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002592
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002592
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002568
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002568
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002568
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002580
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002580
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002580
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_107.zip
Uploading the intermediate file: upload_file_107.zip
  05:57:10 STEP 2593 H=2593:00 +CPU= 17.277
  05:57:29 STEP 2594 H=2594:00 +CPU= 18.203
  05:57:55 STEP 2595 H=2595:00 +CPU= 26.160
  05:58:13 STEP 2596 H=2596:00 +CPU= 18.014
  05:58:31 STEP 2597 H=2597:00 +CPU= 17.503
  05:58:58 STEP 2598 H=2598:00 +CPU= 26.638
  05:59:15 STEP 2599 H=2599:00 +CPU= 17.616
  05:59:33 STEP 2600 H=2600:00 +CPU= 18.038
  05:59:59 STEP 2601 H=2601:00 +CPU= 25.367
  06:00:17 STEP 2602 H=2602:00 +CPU= 17.895
  06:00:35 STEP 2603 H=2603:00 +CPU= 17.333
  06:01:04 STEP 2604 H=2604:00 +CPU= 28.620
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002604
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002604
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002604
  06:01:21 STEP 2605 H=2605:00 +CPU= 17.513
  06:01:39 STEP 2606 H=2606:00 +CPU= 17.946
  06:02:05 STEP 2607 H=2607:00 +CPU= 25.817
  06:02:23 STEP 2608 H=2608:00 +CPU= 18.032
  06:02:41 STEP 2609 H=2609:00 +CPU= 17.481
  06:03:07 STEP 2610 H=2610:00 +CPU= 26.368
  06:03:25 STEP 2611 H=2611:00 +CPU= 17.589
  06:03:43 STEP 2612 H=2612:00 +CPU= 18.066
  06:04:09 STEP 2613 H=2613:00 +CPU= 25.856
  06:04:27 STEP 2614 H=2614:00 +CPU= 17.814
  06:04:45 STEP 2615 H=2615:00 +CPU= 17.962
  06:05:16 STEP 2616 H=2616:00 +CPU= 30.710
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002616
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002616
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002616
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002592
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002592
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002592
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002604
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002604
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002604
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_108.zip
Uploading the intermediate file: upload_file_108.zip
  06:05:34 STEP 2617 H=2617:00 +CPU= 17.565
  06:05:52 STEP 2618 H=2618:00 +CPU= 17.832
  06:06:18 STEP 2619 H=2619:00 +CPU= 26.064
  06:06:36 STEP 2620 H=2620:00 +CPU= 17.821
  06:06:54 STEP 2621 H=2621:00 +CPU= 17.414
  06:07:21 STEP 2622 H=2622:00 +CPU= 26.088
  06:07:38 STEP 2623 H=2623:00 +CPU= 17.257
  06:07:56 STEP 2624 H=2624:00 +CPU= 17.917
  06:08:22 STEP 2625 H=2625:00 +CPU= 25.626
  06:08:41 STEP 2626 H=2626:00 +CPU= 17.846
  06:08:58 STEP 2627 H=2627:00 +CPU= 17.317
  06:09:27 STEP 2628 H=2628:00 +CPU= 28.634
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002628
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002628
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002628
  06:09:44 STEP 2629 H=2629:00 +CPU= 17.512
  06:10:03 STEP 2630 H=2630:00 +CPU= 17.973
  06:10:29 STEP 2631 H=2631:00 +CPU= 25.826
  06:10:47 STEP 2632 H=2632:00 +CPU= 18.034
  06:11:04 STEP 2633 H=2633:00 +CPU= 17.469
  06:11:30 STEP 2634 H=2634:00 +CPU= 26.335
  06:11:48 STEP 2635 H=2635:00 +CPU= 17.408
  06:12:06 STEP 2636 H=2636:00 +CPU= 17.957
  06:12:32 STEP 2637 H=2637:00 +CPU= 25.820
  06:12:50 STEP 2638 H=2638:00 +CPU= 17.839
  06:13:08 STEP 2639 H=2639:00 +CPU= 17.975
  06:13:39 STEP 2640 H=2640:00 +CPU= 30.777
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002640
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002640
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002640
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002616
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002616
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002616
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002628
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002628
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002628
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_109.zip
Uploading the intermediate file: upload_file_109.zip
  06:13:57 STEP 2641 H=2641:00 +CPU= 17.546
  06:14:15 STEP 2642 H=2642:00 +CPU= 17.910
  06:14:41 STEP 2643 H=2643:00 +CPU= 26.123
  06:15:00 STEP 2644 H=2644:00 +CPU= 18.009
  06:15:17 STEP 2645 H=2645:00 +CPU= 17.630
  06:15:44 STEP 2646 H=2646:00 +CPU= 26.631
  06:16:01 STEP 2647 H=2647:00 +CPU= 17.582
  06:16:19 STEP 2648 H=2648:00 +CPU= 17.944
  06:16:45 STEP 2649 H=2649:00 +CPU= 25.622
  06:17:03 STEP 2650 H=2650:00 +CPU= 17.782
  06:17:21 STEP 2651 H=2651:00 +CPU= 17.464
  06:17:50 STEP 2652 H=2652:00 +CPU= 28.600
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002652
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002652
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002652
  06:18:07 STEP 2653 H=2653:00 +CPU= 17.542
  06:18:25 STEP 2654 H=2654:00 +CPU= 18.059
  06:18:51 STEP 2655 H=2655:00 +CPU= 25.753
  06:19:09 STEP 2656 H=2656:00 +CPU= 17.959
  06:19:27 STEP 2657 H=2657:00 +CPU= 17.355
  06:19:53 STEP 2658 H=2658:00 +CPU= 26.345
  06:20:10 STEP 2659 H=2659:00 +CPU= 17.246
  06:20:28 STEP 2660 H=2660:00 +CPU= 17.859
  06:20:54 STEP 2661 H=2661:00 +CPU= 25.677
  06:21:12 STEP 2662 H=2662:00 +CPU= 17.858
  06:21:30 STEP 2663 H=2663:00 +CPU= 18.036
  06:22:02 STEP 2664 H=2664:00 +CPU= 30.877
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002664
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002664
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002664
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002640
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002640
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002640
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002652
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002652
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002652
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_110.zip
Uploading the intermediate file: upload_file_110.zip
  06:22:20 STEP 2665 H=2665:00 +CPU= 17.331
  06:22:38 STEP 2666 H=2666:00 +CPU= 17.993
  06:23:04 STEP 2667 H=2667:00 +CPU= 25.940
  06:23:24 STEP 2668 H=2668:00 +CPU= 17.890
  06:23:41 STEP 2669 H=2669:00 +CPU= 17.246
  06:24:08 STEP 2670 H=2670:00 +CPU= 26.503
  06:24:25 STEP 2671 H=2671:00 +CPU= 17.296
  06:24:43 STEP 2672 H=2672:00 +CPU= 17.749
  06:25:09 STEP 2673 H=2673:00 +CPU= 25.938
  06:25:27 STEP 2674 H=2674:00 +CPU= 17.954
  06:25:45 STEP 2675 H=2675:00 +CPU= 17.513
  06:26:15 STEP 2676 H=2676:00 +CPU= 29.115
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002676
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002676
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002676
  06:26:32 STEP 2677 H=2677:00 +CPU= 17.512
  06:26:51 STEP 2678 H=2678:00 +CPU= 18.029
  06:27:17 STEP 2679 H=2679:00 +CPU= 26.139
  06:27:35 STEP 2680 H=2680:00 +CPU= 18.130
  06:27:53 STEP 2681 H=2681:00 +CPU= 17.692
  06:28:20 STEP 2682 H=2682:00 +CPU= 26.557
  06:28:38 STEP 2683 H=2683:00 +CPU= 17.585
  06:28:56 STEP 2684 H=2684:00 +CPU= 17.917
  06:29:21 STEP 2685 H=2685:00 +CPU= 25.751
  06:29:39 STEP 2686 H=2686:00 +CPU= 17.887
  06:29:57 STEP 2687 H=2687:00 +CPU= 17.826
  06:30:29 STEP 2688 H=2688:00 +CPU= 30.658
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002688
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002688
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002688
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002664
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002664
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002664
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002676
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002676
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002676
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_111.zip
Uploading the intermediate file: upload_file_111.zip
  06:30:46 STEP 2689 H=2689:00 +CPU= 17.072
  06:31:04 STEP 2690 H=2690:00 +CPU= 17.674
  06:31:30 STEP 2691 H=2691:00 +CPU= 25.750
  06:31:47 STEP 2692 H=2692:00 +CPU= 17.814
  06:32:05 STEP 2693 H=2693:00 +CPU= 17.164
  06:32:31 STEP 2694 H=2694:00 +CPU= 26.495
  06:32:49 STEP 2695 H=2695:00 +CPU= 17.179
  06:33:06 STEP 2696 H=2696:00 +CPU= 17.606
  06:33:32 STEP 2697 H=2697:00 +CPU= 25.739
  06:33:50 STEP 2698 H=2698:00 +CPU= 17.940
  06:34:08 STEP 2699 H=2699:00 +CPU= 17.472
  06:34:37 STEP 2700 H=2700:00 +CPU= 28.852
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002700
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002700
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002700
  06:34:54 STEP 2701 H=2701:00 +CPU= 17.432
  06:35:12 STEP 2702 H=2702:00 +CPU= 17.761
  06:35:38 STEP 2703 H=2703:00 +CPU= 25.913
  06:35:56 STEP 2704 H=2704:00 +CPU= 17.623
  06:36:13 STEP 2705 H=2705:00 +CPU= 17.240
  06:36:40 STEP 2706 H=2706:00 +CPU= 26.299
  06:36:57 STEP 2707 H=2707:00 +CPU= 17.303
  06:37:15 STEP 2708 H=2708:00 +CPU= 17.701
  06:37:41 STEP 2709 H=2709:00 +CPU= 25.622
  06:37:59 STEP 2710 H=2710:00 +CPU= 17.881
  06:38:18 STEP 2711 H=2711:00 +CPU= 17.959
  06:38:50 STEP 2712 H=2712:00 +CPU= 30.809
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002712
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002712
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002712
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002688
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002688
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002688
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002700
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002700
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002700
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_112.zip
Uploading the intermediate file: upload_file_112.zip
  06:39:07 STEP 2713 H=2713:00 +CPU= 17.271
  06:39:25 STEP 2714 H=2714:00 +CPU= 17.783
  06:39:51 STEP 2715 H=2715:00 +CPU= 25.984
  06:40:10 STEP 2716 H=2716:00 +CPU= 18.023
  06:40:27 STEP 2717 H=2717:00 +CPU= 17.387
  06:40:54 STEP 2718 H=2718:00 +CPU= 26.747
  06:41:12 STEP 2719 H=2719:00 +CPU= 17.402
  06:41:30 STEP 2720 H=2720:00 +CPU= 17.883
  06:41:56 STEP 2721 H=2721:00 +CPU= 25.791
  06:42:14 STEP 2722 H=2722:00 +CPU= 18.014
  06:42:31 STEP 2723 H=2723:00 +CPU= 17.434
  06:43:00 STEP 2724 H=2724:00 +CPU= 28.885
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002724
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002724
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002724
  06:43:18 STEP 2725 H=2725:00 +CPU= 17.577
  06:43:36 STEP 2726 H=2726:00 +CPU= 18.017
  06:44:03 STEP 2727 H=2727:00 +CPU= 26.168
  06:44:21 STEP 2728 H=2728:00 +CPU= 17.950
  06:44:38 STEP 2729 H=2729:00 +CPU= 17.636
  06:45:26 STEP 2730 H=2730:00 +CPU= 28.712
  06:45:58 STEP 2731 H=2731:00 +CPU= 18.240
  06:46:33 STEP 2732 H=2732:00 +CPU= 18.386
  06:47:00 STEP 2733 H=2733:00 +CPU= 24.501
  06:47:16 STEP 2734 H=2734:00 +CPU= 16.132
  06:47:34 STEP 2735 H=2735:00 +CPU= 17.141
  06:48:05 STEP 2736 H=2736:00 +CPU= 30.151
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002736
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002736
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002736
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002712
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002712
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002712
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002724
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002724
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002724
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_113.zip
Uploading the intermediate file: upload_file_113.zip
  06:48:23 STEP 2737 H=2737:00 +CPU= 17.185
  06:48:40 STEP 2738 H=2738:00 +CPU= 17.705
  06:49:06 STEP 2739 H=2739:00 +CPU= 25.611
  06:49:24 STEP 2740 H=2740:00 +CPU= 17.642
  06:49:41 STEP 2741 H=2741:00 +CPU= 17.396
  06:50:07 STEP 2742 H=2742:00 +CPU= 26.069
  06:50:25 STEP 2743 H=2743:00 +CPU= 17.267
  06:51:06 STEP 2744 H=2744:00 +CPU= 20.039
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0657
(argv4) batchid: 956
(argv5) wuid: 12173301
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /scratch/BOINC/slots/3
Project directory is: /scratch/BOINC/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301
..mkdir for temp folder for results failed
Copying: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /scratch/BOINC/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /scratch/BOINC/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b46540d84f1e4feff3fde9fe07614147 to: /scratch/BOINC/slots/3/oifs_43r3_ps_0657_1987050100_123_956_12173301.zip
Unzipping the namelist zip file: /scratch/BOINC/slots/3/oifs_43r3_ps_0657_1987050100_123_956_12173301.zip
ic_ancil_file: ic_ancil_12173301
ifsdata_file: ifsdata_12173301
climate_data_file: clim_data_12173301
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ed459df37b2246efb2a1cd3eb160e65e to: /scratch/BOINC/slots/3/ic_ancil_12173301.zip
Unzipping the IC ancils zip file: /scratch/BOINC/slots/3/ic_ancil_12173301.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f6aa7b4d431ad4008ad55c1c99163e64 to: /scratch/BOINC/slots/3/ifsdata/ifsdata_12173301.zip
Unzipping the ifsdata_zip file: /scratch/BOINC/slots/3/ifsdata/ifsdata_12173301.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /scratch/BOINC/slots/3/159l_2/clim_data_12173301.zip
Unzipping the climate data zip file: /scratch/BOINC/slots/3/159l_2/clim_data_12173301.zip
Checking for progress XML file: /scratch/BOINC/slots/3/progress_file_12173301.xml
Opened progress file ok : /scratch/BOINC/slots/3/progress_file_12173301.xml
-- Model is restarting --
Adjusting last_iter, 2745, to previous model restart step.
Creating progress file: /scratch/BOINC/slots/3/progress_file_12173301.xml
last_cpu_time: 55406
upload_file_number: 114
last_iter: 2735
last_upload: 9849600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632
The child process has been launched with process id: 8675
Executing the command: /scratch/BOINC/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vaals:1:1:8675:8675] [20230205:074806:1675579686:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:48:43 STEP 2736 H=2736:00 +CPU= 29.324
  07:49:22 STEP 2737 H=2737:00 +CPU= 17.608
  07:49:38 STEP 2738 H=2738:00 +CPU= 15.335
  07:50:03 STEP 2739 H=2739:00 +CPU= 24.224
  07:50:19 STEP 2740 H=2740:00 +CPU= 16.139
  07:50:36 STEP 2741 H=2741:00 +CPU= 17.163
  07:51:03 STEP 2742 H=2742:00 +CPU= 26.524
  07:51:20 STEP 2743 H=2743:00 +CPU= 17.238
  07:51:38 STEP 2744 H=2744:00 +CPU= 17.719
  07:52:04 STEP 2745 H=2745:00 +CPU= 26.038
  07:52:22 STEP 2746 H=2746:00 +CPU= 17.684
  07:52:39 STEP 2747 H=2747:00 +CPU= 17.297
  07:53:09 STEP 2748 H=2748:00 +CPU= 28.908
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002748
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002748
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002748
  07:53:31 STEP 2749 H=2749:00 +CPU= 18.225
  07:54:01 STEP 2750 H=2750:00 +CPU= 18.708
  07:54:26 STEP 2751 H=2751:00 +CPU= 24.434
  07:54:42 STEP 2752 H=2752:00 +CPU= 15.966
  07:55:07 STEP 2753 H=2753:00 +CPU= 18.083
  07:55:53 STEP 2754 H=2754:00 +CPU= 28.067
  07:56:10 STEP 2755 H=2755:00 +CPU= 15.910
  07:56:26 STEP 2756 H=2756:00 +CPU= 16.258
  07:56:51 STEP 2757 H=2757:00 +CPU= 24.966
  07:57:09 STEP 2758 H=2758:00 +CPU= 17.867
  07:57:27 STEP 2759 H=2759:00 +CPU= 17.820
  07:58:02 STEP 2760 H=2760:00 +CPU= 31.402
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002760
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002760
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002760
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002736
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002736
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002736
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002748
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002748
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002748
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_114.zip
Uploading the intermediate file: upload_file_114.zip
  07:58:20 STEP 2761 H=2761:00 +CPU= 17.492
  07:58:37 STEP 2762 H=2762:00 +CPU= 17.257
  07:59:04 STEP 2763 H=2763:00 +CPU= 26.588
  07:59:22 STEP 2764 H=2764:00 +CPU= 17.650
  07:59:39 STEP 2765 H=2765:00 +CPU= 17.066
  08:00:06 STEP 2766 H=2766:00 +CPU= 26.785
  08:00:24 STEP 2767 H=2767:00 +CPU= 17.502
  08:00:42 STEP 2768 H=2768:00 +CPU= 17.720
  08:01:08 STEP 2769 H=2769:00 +CPU= 26.360
  08:01:26 STEP 2770 H=2770:00 +CPU= 17.896
  08:01:43 STEP 2771 H=2771:00 +CPU= 17.002
  08:02:13 STEP 2772 H=2772:00 +CPU= 29.020
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002772
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002772
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002772
  08:02:49 STEP 2773 H=2773:00 +CPU= 20.151
  08:03:30 STEP 2774 H=2774:00 +CPU= 18.485
  08:03:55 STEP 2775 H=2775:00 +CPU= 24.024
  08:04:11 STEP 2776 H=2776:00 +CPU= 16.178
  08:04:27 STEP 2777 H=2777:00 +CPU= 15.903
  08:04:53 STEP 2778 H=2778:00 +CPU= 26.244
  08:05:11 STEP 2779 H=2779:00 +CPU= 17.363
  08:05:28 STEP 2780 H=2780:00 +CPU= 17.108
  08:05:54 STEP 2781 H=2781:00 +CPU= 25.922
  08:06:12 STEP 2782 H=2782:00 +CPU= 17.785
  08:06:29 STEP 2783 H=2783:00 +CPU= 17.249
  08:07:01 STEP 2784 H=2784:00 +CPU= 30.595
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002784
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002784
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002784
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002760
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002760
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002760
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002772
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002772
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002772
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_115.zip
Uploading the intermediate file: upload_file_115.zip
  08:07:18 STEP 2785 H=2785:00 +CPU= 16.998
  08:07:36 STEP 2786 H=2786:00 +CPU= 17.544
  08:08:02 STEP 2787 H=2787:00 +CPU= 25.759
  08:08:21 STEP 2788 H=2788:00 +CPU= 17.605
  08:08:41 STEP 2789 H=2789:00 +CPU= 17.291
  08:09:17 STEP 2790 H=2790:00 +CPU= 27.105
  08:09:51 STEP 2791 H=2791:00 +CPU= 19.019
  08:10:27 STEP 2792 H=2792:00 +CPU= 17.260
  08:10:52 STEP 2793 H=2793:00 +CPU= 24.304
  08:11:08 STEP 2794 H=2794:00 +CPU= 16.352
  08:11:24 STEP 2795 H=2795:00 +CPU= 16.344
  08:11:53 STEP 2796 H=2796:00 +CPU= 28.563
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002796
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002796
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002796
  08:12:10 STEP 2797 H=2797:00 +CPU= 17.039
  08:12:28 STEP 2798 H=2798:00 +CPU= 17.569
  08:12:54 STEP 2799 H=2799:00 +CPU= 25.901
  08:13:11 STEP 2800 H=2800:00 +CPU= 17.411
  08:13:28 STEP 2801 H=2801:00 +CPU= 16.599
  08:13:55 STEP 2802 H=2802:00 +CPU= 26.308
  08:14:12 STEP 2803 H=2803:00 +CPU= 17.293
  08:14:29 STEP 2804 H=2804:00 +CPU= 17.059
  08:14:55 STEP 2805 H=2805:00 +CPU= 25.789
  08:15:13 STEP 2806 H=2806:00 +CPU= 17.653
  08:15:30 STEP 2807 H=2807:00 +CPU= 17.084
  08:16:02 STEP 2808 H=2808:00 +CPU= 30.617
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002808
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002808
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002808
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002784
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002784
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002784
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002796
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002796
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002796
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_116.zip
Uploading the intermediate file: upload_file_116.zip
  08:16:19 STEP 2809 H=2809:00 +CPU= 17.042
  08:16:37 STEP 2810 H=2810:00 +CPU= 17.656
  08:17:03 STEP 2811 H=2811:00 +CPU= 25.940
  08:17:20 STEP 2812 H=2812:00 +CPU= 17.568
  08:17:40 STEP 2813 H=2813:00 +CPU= 17.343
  08:18:06 STEP 2814 H=2814:00 +CPU= 26.376
  08:18:23 STEP 2815 H=2815:00 +CPU= 17.094
  08:18:41 STEP 2816 H=2816:00 +CPU= 17.483
  08:19:07 STEP 2817 H=2817:00 +CPU= 25.801
  08:19:24 STEP 2818 H=2818:00 +CPU= 17.487
  08:19:41 STEP 2819 H=2819:00 +CPU= 17.063
  08:20:10 STEP 2820 H=2820:00 +CPU= 28.647
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002820
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002820
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002820
  08:20:27 STEP 2821 H=2821:00 +CPU= 17.062
  08:20:45 STEP 2822 H=2822:00 +CPU= 17.750
  08:21:11 STEP 2823 H=2823:00 +CPU= 26.099
  08:21:29 STEP 2824 H=2824:00 +CPU= 17.541
  08:21:47 STEP 2825 H=2825:00 +CPU= 16.934
  08:22:13 STEP 2826 H=2826:00 +CPU= 26.658
  08:22:31 STEP 2827 H=2827:00 +CPU= 17.257
  08:22:48 STEP 2828 H=2828:00 +CPU= 17.217
  08:23:14 STEP 2829 H=2829:00 +CPU= 26.003
  08:23:32 STEP 2830 H=2830:00 +CPU= 17.682
  08:23:51 STEP 2831 H=2831:00 +CPU= 17.285
  08:24:23 STEP 2832 H=2832:00 +CPU= 30.630
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002832
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002832
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002832
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002808
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002808
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002808
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002820
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002820
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002820
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_117.zip
Uploading the intermediate file: upload_file_117.zip
  08:24:40 STEP 2833 H=2833:00 +CPU= 17.132
  08:24:58 STEP 2834 H=2834:00 +CPU= 17.746
  08:25:24 STEP 2835 H=2835:00 +CPU= 26.144
  08:25:42 STEP 2836 H=2836:00 +CPU= 17.568
  08:25:59 STEP 2837 H=2837:00 +CPU= 16.986
  08:26:26 STEP 2838 H=2838:00 +CPU= 26.424
  08:26:43 STEP 2839 H=2839:00 +CPU= 17.053
  08:27:00 STEP 2840 H=2840:00 +CPU= 17.452
  08:27:26 STEP 2841 H=2841:00 +CPU= 25.932
  08:27:44 STEP 2842 H=2842:00 +CPU= 17.501
  08:28:01 STEP 2843 H=2843:00 +CPU= 17.065
  08:28:30 STEP 2844 H=2844:00 +CPU= 28.828
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002844
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002844
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002844
  08:28:47 STEP 2845 H=2845:00 +CPU= 17.169
  08:29:05 STEP 2846 H=2846:00 +CPU= 17.716
  08:29:31 STEP 2847 H=2847:00 +CPU= 26.067
  08:29:49 STEP 2848 H=2848:00 +CPU= 17.604
  08:30:07 STEP 2849 H=2849:00 +CPU= 17.052
  08:30:34 STEP 2850 H=2850:00 +CPU= 26.774
  08:30:51 STEP 2851 H=2851:00 +CPU= 17.311
  08:31:09 STEP 2852 H=2852:00 +CPU= 17.353
  08:31:35 STEP 2853 H=2853:00 +CPU= 26.365
  08:31:53 STEP 2854 H=2854:00 +CPU= 17.653
  08:32:11 STEP 2855 H=2855:00 +CPU= 17.577
  08:32:42 STEP 2856 H=2856:00 +CPU= 30.540
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002856
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002856
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002856
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002832
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002832
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002832
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002844
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002844
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002844
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_118.zip
Uploading the intermediate file: upload_file_118.zip
  08:33:00 STEP 2857 H=2857:00 +CPU= 16.760
  08:33:17 STEP 2858 H=2858:00 +CPU= 17.781
  08:33:44 STEP 2859 H=2859:00 +CPU= 26.133
  08:34:01 STEP 2860 H=2860:00 +CPU= 17.279
  08:34:18 STEP 2861 H=2861:00 +CPU= 17.278
  08:34:45 STEP 2862 H=2862:00 +CPU= 26.429
  08:35:02 STEP 2863 H=2863:00 +CPU= 17.219
  08:35:20 STEP 2864 H=2864:00 +CPU= 17.737
  08:35:46 STEP 2865 H=2865:00 +CPU= 25.944
  08:36:04 STEP 2866 H=2866:00 +CPU= 17.570
  08:36:21 STEP 2867 H=2867:00 +CPU= 17.318
  08:36:50 STEP 2868 H=2868:00 +CPU= 28.676
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002868
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002868
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002868
  08:37:07 STEP 2869 H=2869:00 +CPU= 16.886
  08:37:25 STEP 2870 H=2870:00 +CPU= 17.732
  08:37:51 STEP 2871 H=2871:00 +CPU= 26.134
  08:38:08 STEP 2872 H=2872:00 +CPU= 17.277
  08:38:26 STEP 2873 H=2873:00 +CPU= 17.125
  08:38:54 STEP 2874 H=2874:00 +CPU= 26.719
  08:39:12 STEP 2875 H=2875:00 +CPU= 17.192
  08:39:29 STEP 2876 H=2876:00 +CPU= 17.439
  08:39:56 STEP 2877 H=2877:00 +CPU= 26.148
  08:40:13 STEP 2878 H=2878:00 +CPU= 17.620
  08:40:31 STEP 2879 H=2879:00 +CPU= 17.617
  08:41:02 STEP 2880 H=2880:00 +CPU= 30.566
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002880
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002880
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002880
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002856
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002856
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002856
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002868
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002868
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002868
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_119.zip
Uploading the intermediate file: upload_file_119.zip
  08:41:19 STEP 2881 H=2881:00 +CPU= 16.714
  08:41:37 STEP 2882 H=2882:00 +CPU= 17.653
  08:42:03 STEP 2883 H=2883:00 +CPU= 26.154
  08:42:21 STEP 2884 H=2884:00 +CPU= 17.316
  08:42:38 STEP 2885 H=2885:00 +CPU= 17.315
  08:43:05 STEP 2886 H=2886:00 +CPU= 26.447
  08:43:22 STEP 2887 H=2887:00 +CPU= 17.189
  08:43:40 STEP 2888 H=2888:00 +CPU= 17.792
  08:44:06 STEP 2889 H=2889:00 +CPU= 25.967
  08:44:23 STEP 2890 H=2890:00 +CPU= 17.558
  08:44:41 STEP 2891 H=2891:00 +CPU= 17.295
  08:45:09 STEP 2892 H=2892:00 +CPU= 28.723
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002892
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002892
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002892
  08:45:26 STEP 2893 H=2893:00 +CPU= 16.890
  08:45:44 STEP 2894 H=2894:00 +CPU= 17.693
  08:46:10 STEP 2895 H=2895:00 +CPU= 26.115
  08:46:28 STEP 2896 H=2896:00 +CPU= 17.350
  08:46:45 STEP 2897 H=2897:00 +CPU= 17.001
  08:47:12 STEP 2898 H=2898:00 +CPU= 26.710
  08:47:29 STEP 2899 H=2899:00 +CPU= 17.123
  08:47:48 STEP 2900 H=2900:00 +CPU= 17.454
  08:48:14 STEP 2901 H=2901:00 +CPU= 25.892
  08:48:31 STEP 2902 H=2902:00 +CPU= 17.501
  08:48:49 STEP 2903 H=2903:00 +CPU= 17.501
  08:49:20 STEP 2904 H=2904:00 +CPU= 30.422
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002904
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002904
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002904
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002880
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002880
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002880
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002892
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002892
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002892
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_120.zip
Uploading the intermediate file: upload_file_120.zip
  08:49:37 STEP 2905 H=2905:00 +CPU= 16.669
  08:49:55 STEP 2906 H=2906:00 +CPU= 17.712
  08:50:21 STEP 2907 H=2907:00 +CPU= 26.011
  08:50:38 STEP 2908 H=2908:00 +CPU= 17.330
  08:50:56 STEP 2909 H=2909:00 +CPU= 17.332
  08:51:23 STEP 2910 H=2910:00 +CPU= 26.594
  08:51:40 STEP 2911 H=2911:00 +CPU= 17.029
  08:51:58 STEP 2912 H=2912:00 +CPU= 17.662
  08:52:24 STEP 2913 H=2913:00 +CPU= 26.046
  08:52:41 STEP 2914 H=2914:00 +CPU= 17.421
  08:52:58 STEP 2915 H=2915:00 +CPU= 17.316
  08:53:28 STEP 2916 H=2916:00 +CPU= 28.653
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002916
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002916
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002916
  08:53:45 STEP 2917 H=2917:00 +CPU= 16.856
  08:54:03 STEP 2918 H=2918:00 +CPU= 17.657
  08:54:29 STEP 2919 H=2919:00 +CPU= 25.927
  08:54:46 STEP 2920 H=2920:00 +CPU= 17.193
  08:55:03 STEP 2921 H=2921:00 +CPU= 16.953
  08:55:30 STEP 2922 H=2922:00 +CPU= 26.458
  08:55:47 STEP 2923 H=2923:00 +CPU= 17.184
  08:56:05 STEP 2924 H=2924:00 +CPU= 17.496
  08:56:31 STEP 2925 H=2925:00 +CPU= 25.860
  08:56:49 STEP 2926 H=2926:00 +CPU= 17.541
  08:57:06 STEP 2927 H=2927:00 +CPU= 17.513
  08:57:38 STEP 2928 H=2928:00 +CPU= 30.451
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002928
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002928
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002928
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002904
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002904
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002904
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002916
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002916
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002916
Zipping up the intermediate file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_121.zip
Uploading the intermediate file: upload_file_121.zip
  08:57:55 STEP 2929 H=2929:00 +CPU= 16.732
  08:58:13 STEP 2930 H=2930:00 +CPU= 17.786
  08:58:39 STEP 2931 H=2931:00 +CPU= 25.938
  08:58:56 STEP 2932 H=2932:00 +CPU= 17.318
  08:59:14 STEP 2933 H=2933:00 +CPU= 17.312
  08:59:40 STEP 2934 H=2934:00 +CPU= 26.542
  08:59:57 STEP 2935 H=2935:00 +CPU= 16.954
  09:00:15 STEP 2936 H=2936:00 +CPU= 17.686
  09:00:41 STEP 2937 H=2937:00 +CPU= 26.044
  09:00:58 STEP 2938 H=2938:00 +CPU= 17.397
  09:01:16 STEP 2939 H=2939:00 +CPU= 17.286
  09:01:45 STEP 2940 H=2940:00 +CPU= 28.551
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002940
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002940
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002940
  09:02:02 STEP 2941 H=2941:00 +CPU= 16.859
  09:02:19 STEP 2942 H=2942:00 +CPU= 17.745
  09:02:46 STEP 2943 H=2943:00 +CPU= 26.094
  09:03:03 STEP 2944 H=2944:00 +CPU= 17.307
  09:03:20 STEP 2945 H=2945:00 +CPU= 17.012
  09:03:47 STEP 2946 H=2946:00 +CPU= 26.380
  09:04:04 STEP 2947 H=2947:00 +CPU= 17.085
  09:04:22 STEP 2948 H=2948:00 +CPU= 17.524
  09:04:48 STEP 2949 H=2949:00 +CPU= 25.880
  09:05:05 STEP 2950 H=2950:00 +CPU= 17.481
  09:05:23 STEP 2951 H=2951:00 +CPU= 17.496
  09:05:53 STEP 2952 H=2952:00 +CPU= 29.142
The child process terminated with status: 0
Moving to projects directory: /scratch/BOINC/slots/3/ICMGGhq0f+002952
Moving to projects directory: /scratch/BOINC/slots/3/ICMSHhq0f+002952
Moving to projects directory: /scratch/BOINC/slots/3/ICMUAhq0f+002952
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002940
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002928
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002928
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002952
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002928
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002940
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMUAhq0f+002952
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMGGhq0f+002940
Adding to the zip: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_12173301/ICMSHhq0f+002952
Zipping up the final file: /scratch/BOINC/projects/climateprediction.net/oifs_43r3_ps_0657_1987050100_123_956_12173301_2_r496909632_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
09:09:16 (8672): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:48:07 1534903 22305539 oifs_43r3_ps_0657_1987050100_123_956_12173301_2 10,623,600 59,874 0.0056
09 May 2023 16:48:00 1534903 22305539 oifs_43r3_ps_0657_1987050100_123_956_12173301_2 8,640,000 48,126 0.0056
09 May 2023 16:47:59 1534903 22305539 oifs_43r3_ps_0657_1987050100_123_956_12173301_2 7,776,000 43,160 0.0056
09 May 2023 16:47:58 1534903 22305539 oifs_43r3_ps_0657_1987050100_123_956_12173301_2 6,912,000 38,220 0.0055
09 May 2023 16:47:56 1534903 22305539 oifs_43r3_ps_0657_1987050100_123_956_12173301_2 6,048,000 33,257 0.0055
09 May 2023 16:47:55 1534903 22305539 oifs_43r3_ps_0657_1987050100_123_956_12173301_2 5,184,000 28,323 0.0055
09 May 2023 16:47:53 1534903 22305539 oifs_43r3_ps_0657_1987050100_123_956_12173301_2 4,320,000 23,403 0.0054
09 May 2023 16:47:51 1534903 22305539 oifs_43r3_ps_0657_1987050100_123_956_12173301_2 3,456,000 18,592 0.0054
09 May 2023 16:47:50 1534903 22305539 oifs_43r3_ps_0657_1987050100_123_956_12173301_2 2,592,000 14,019 0.0054
09 May 2023 16:47:48 1534903 22305539 oifs_43r3_ps_0657_1987050100_123_956_12173301_2 1,728,000 9,445 0.0055
09 May 2023 16:47:47 1534903 22305539 oifs_43r3_ps_0657_1987050100_123_956_12173301_2 864,000 4,879 0.0056


©2024 climateprediction.net