climateprediction.net home page
Task 22267952

Task 22267952

Name oifs_43r3_ps_0173_1997050100_123_966_12182817_0
Workunit 12182817
Created 22 Dec 2022, 12:54:19 UTC
Sent 22 Dec 2022, 13:03:38 UTC
Report deadline 21 Jan 2023, 13:03:38 UTC
Received 23 Dec 2022, 20:22:21 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1500195
Run time 1 days 0 hours 49 min 47 sec
CPU time 22 hours 30 min 46 sec
Validate state Invalid
Credit 11,612.16
Device peak FLOPS 5.53 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,410.40 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,675.45 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
xeen:1:1:2112279:2112279] [20221223:165528:1671810928:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2112279:2112279] [20221223:165528:1671810928:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2112279:2112279] [20221223:165528:1671810928:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:56:23 STEP 2544 H=2544:00 +CPU= 48.357
  16:56:49 STEP 2545 H=2545:00 +CPU= 25.642
  16:57:18 STEP 2546 H=2546:00 +CPU= 27.930
  16:58:00 STEP 2547 H=2547:00 +CPU= 40.740
  16:58:26 STEP 2548 H=2548:00 +CPU= 25.597
  16:58:52 STEP 2549 H=2549:00 +CPU= 24.824
  16:59:35 STEP 2550 H=2550:00 +CPU= 41.956
  17:00:02 STEP 2551 H=2551:00 +CPU= 26.012
  17:00:29 STEP 2552 H=2552:00 +CPU= 26.768
  17:01:11 STEP 2553 H=2553:00 +CPU= 40.881
  17:01:39 STEP 2554 H=2554:00 +CPU= 26.760
  17:02:05 STEP 2555 H=2555:00 +CPU= 26.213
  17:02:50 STEP 2556 H=2556:00 +CPU= 44.429
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002556
  17:03:16 STEP 2557 H=2557:00 +CPU= 24.685
  17:03:43 STEP 2558 H=2558:00 +CPU= 27.136
  17:04:26 STEP 2559 H=2559:00 +CPU= 41.453
  17:04:55 STEP 2560 H=2560:00 +CPU= 27.142
  17:05:22 STEP 2561 H=2561:00 +CPU= 25.461
  17:06:06 STEP 2562 H=2562:00 +CPU= 41.463
  17:06:31 STEP 2563 H=2563:00 +CPU= 25.100
  17:06:58 STEP 2564 H=2564:00 +CPU= 26.612
  17:07:41 STEP 2565 H=2565:00 +CPU= 41.371
  17:08:07 STEP 2566 H=2566:00 +CPU= 25.920
  17:08:35 STEP 2567 H=2567:00 +CPU= 26.885
  17:09:29 STEP 2568 H=2568:00 +CPU= 48.156
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765_106.zip
Uploading the intermediate file: upload_file_106.zip
  17:09:55 STEP 2569 H=2569:00 +CPU= 24.873
  17:10:22 STEP 2570 H=2570:00 +CPU= 26.892
  17:11:04 STEP 2571 H=2571:00 +CPU= 40.868
  17:11:30 STEP 2572 H=2572:00 +CPU= 25.192
  17:11:59 STEP 2573 H=2573:00 +CPU= 27.638
  17:12:42 STEP 2574 H=2574:00 +CPU= 42.025
  17:13:07 STEP 2575 H=2575:00 +CPU= 24.444
  17:13:35 STEP 2576 H=2576:00 +CPU= 27.309
  17:14:16 STEP 2577 H=2577:00 +CPU= 40.342
  17:14:42 STEP 2578 H=2578:00 +CPU= 25.611
  17:15:09 STEP 2579 H=2579:00 +CPU= 26.287
  17:15:53 STEP 2580 H=2580:00 +CPU= 43.680
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002580
  17:16:19 STEP 2581 H=2581:00 +CPU= 24.754
  17:16:47 STEP 2582 H=2582:00 +CPU= 27.450
  17:17:26 STEP 2583 H=2583:00 +CPU= 38.809
  17:17:53 STEP 2584 H=2584:00 +CPU= 25.688
  17:18:22 STEP 2585 H=2585:00 +CPU= 27.420
  17:19:03 STEP 2586 H=2586:00 +CPU= 40.052
  17:19:28 STEP 2587 H=2587:00 +CPU= 24.664
  17:19:57 STEP 2588 H=2588:00 +CPU= 27.514
  17:20:36 STEP 2589 H=2589:00 +CPU= 38.814
  17:21:02 STEP 2590 H=2590:00 +CPU= 25.796
  17:21:32 STEP 2591 H=2591:00 +CPU= 28.253
  17:22:19 STEP 2592 H=2592:00 +CPU= 46.125
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765_107.zip
Uploading the intermediate file: upload_file_107.zip
  17:22:46 STEP 2593 H=2593:00 +CPU= 26.189
  17:23:13 STEP 2594 H=2594:00 +CPU= 26.191
  17:23:53 STEP 2595 H=2595:00 +CPU= 39.504
  17:24:20 STEP 2596 H=2596:00 +CPU= 26.697
  17:24:46 STEP 2597 H=2597:00 +CPU= 25.479
  17:25:29 STEP 2598 H=2598:00 +CPU= 42.611
  17:25:56 STEP 2599 H=2599:00 +CPU= 25.628
  17:26:22 STEP 2600 H=2600:00 +CPU= 26.132
  17:27:04 STEP 2601 H=2601:00 +CPU= 41.705
  17:27:32 STEP 2602 H=2602:00 +CPU= 26.609
  17:27:56 STEP 2603 H=2603:00 +CPU= 24.434
  17:28:44 STEP 2604 H=2604:00 +CPU= 45.292
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002604
  17:29:11 STEP 2605 H=2605:00 +CPU= 25.312
  17:29:37 STEP 2606 H=2606:00 +CPU= 25.504
  17:30:19 STEP 2607 H=2607:00 +CPU= 41.062
  17:30:48 STEP 2608 H=2608:00 +CPU= 26.127
  17:31:13 STEP 2609 H=2609:00 +CPU= 24.091
  17:31:56 STEP 2610 H=2610:00 +CPU= 41.473
  17:32:21 STEP 2611 H=2611:00 +CPU= 25.093
  17:32:48 STEP 2612 H=2612:00 +CPU= 26.037
  17:33:28 STEP 2613 H=2613:00 +CPU= 39.355
  17:33:54 STEP 2614 H=2614:00 +CPU= 25.594
  17:34:20 STEP 2615 H=2615:00 +CPU= 26.080
  17:35:13 STEP 2616 H=2616:00 +CPU= 48.310
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765_108.zip
Uploading the intermediate file: upload_file_108.zip
  17:35:38 STEP 2617 H=2617:00 +CPU= 24.771
  17:36:06 STEP 2618 H=2618:00 +CPU= 27.089
  17:36:47 STEP 2619 H=2619:00 +CPU= 40.256
  17:37:13 STEP 2620 H=2620:00 +CPU= 25.506
  17:37:40 STEP 2621 H=2621:00 +CPU= 26.016
  17:38:20 STEP 2622 H=2622:00 +CPU= 39.751
  17:38:47 STEP 2623 H=2623:00 +CPU= 26.059
  17:39:15 STEP 2624 H=2624:00 +CPU= 26.639
  17:39:54 STEP 2625 H=2625:00 +CPU= 38.662
  17:40:23 STEP 2626 H=2626:00 +CPU= 27.315
  17:40:48 STEP 2627 H=2627:00 +CPU= 24.766
  17:41:33 STEP 2628 H=2628:00 +CPU= 43.746
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002628
  17:42:00 STEP 2629 H=2629:00 +CPU= 26.187
  17:42:26 STEP 2630 H=2630:00 +CPU= 25.563
  17:43:07 STEP 2631 H=2631:00 +CPU= 40.272
  17:43:34 STEP 2632 H=2632:00 +CPU= 26.013
  17:43:58 STEP 2633 H=2633:00 +CPU= 24.324
  17:44:43 STEP 2634 H=2634:00 +CPU= 41.045
  17:45:10 STEP 2635 H=2635:00 +CPU= 26.622
  17:45:37 STEP 2636 H=2636:00 +CPU= 25.931
  17:46:18 STEP 2637 H=2637:00 +CPU= 40.622
  17:46:47 STEP 2638 H=2638:00 +CPU= 27.459
  17:47:16 STEP 2639 H=2639:00 +CPU= 27.172
  17:48:07 STEP 2640 H=2640:00 +CPU= 47.935
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765_109.zip
Uploading the intermediate file: upload_file_109.zip
  17:48:35 STEP 2641 H=2641:00 +CPU= 26.138
  17:49:02 STEP 2642 H=2642:00 +CPU= 26.711
  17:49:46 STEP 2643 H=2643:00 +CPU= 41.632
  17:50:14 STEP 2644 H=2644:00 +CPU= 27.188
  17:50:39 STEP 2645 H=2645:00 +CPU= 25.181
  17:51:22 STEP 2646 H=2646:00 +CPU= 41.980
  17:51:49 STEP 2647 H=2647:00 +CPU= 25.631
  17:52:18 STEP 2648 H=2648:00 +CPU= 27.223
  17:53:01 STEP 2649 H=2649:00 +CPU= 40.352
  17:53:28 STEP 2650 H=2650:00 +CPU= 26.338
  17:53:53 STEP 2651 H=2651:00 +CPU= 25.520
  17:54:40 STEP 2652 H=2652:00 +CPU= 44.571
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002652
  17:55:08 STEP 2653 H=2653:00 +CPU= 26.158
  17:55:34 STEP 2654 H=2654:00 +CPU= 26.004
  17:56:16 STEP 2655 H=2655:00 +CPU= 41.939
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0173
(argv4) batchid: 966
(argv5) wuid: 12182817
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d82d4c31f14e94be13bba96c335242d4 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0173_1997050100_123_966_12182817.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0173_1997050100_123_966_12182817.zip
ic_ancil_file: ic_ancil_12182817
ifsdata_file: ifsdata_12182817
climate_data_file: clim_data_12182817
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_79e60cbf34509c79781d88015bd51171 to: /var/lib/boinc-client/slots/11/ic_ancil_12182817.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12182817.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e62d99846d617b995ef659c6fa514bfc to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12182817.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12182817.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12182817.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12182817.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12182817.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12182817.xml
-- Model is restarting --
Adjusting last_iter, 2656, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12182817.xml
last_cpu_time: 73001
upload_file_number: 110
last_iter: 2639
last_upload: 9504000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765
The child process has been launched with process id: 2120562
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2120562:2120562] [20221223:175732:1671814652:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:58:27 STEP 2640 H=2640:00 +CPU= 45.987
  17:58:54 STEP 2641 H=2641:00 +CPU= 26.196
  17:59:23 STEP 2642 H=2642:00 +CPU= 27.071
  18:00:04 STEP 2643 H=2643:00 +CPU= 40.106
  18:00:33 STEP 2644 H=2644:00 +CPU= 27.660
  18:00:59 STEP 2645 H=2645:00 +CPU= 25.541
  18:01:42 STEP 2646 H=2646:00 +CPU= 40.441
  18:02:11 STEP 2647 H=2647:00 +CPU= 27.648
  18:02:39 STEP 2648 H=2648:00 +CPU= 26.958
  18:03:19 STEP 2649 H=2649:00 +CPU= 38.580
  18:03:46 STEP 2650 H=2650:00 +CPU= 26.777
  18:04:14 STEP 2651 H=2651:00 +CPU= 26.648
  18:05:01 STEP 2652 H=2652:00 +CPU= 44.124
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002652
  18:05:28 STEP 2653 H=2653:00 +CPU= 26.872
  18:05:54 STEP 2654 H=2654:00 +CPU= 25.776
  18:06:36 STEP 2655 H=2655:00 +CPU= 40.066
  18:07:05 STEP 2656 H=2656:00 +CPU= 28.106
  18:07:31 STEP 2657 H=2657:00 +CPU= 25.424
  18:08:16 STEP 2658 H=2658:00 +CPU= 41.877
  18:08:44 STEP 2659 H=2659:00 +CPU= 26.888
  18:09:12 STEP 2660 H=2660:00 +CPU= 26.664
  18:09:55 STEP 2661 H=2661:00 +CPU= 40.927
  18:10:23 STEP 2662 H=2662:00 +CPU= 26.563
  18:10:49 STEP 2663 H=2663:00 +CPU= 25.376
  18:11:38 STEP 2664 H=2664:00 +CPU= 47.285
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765_110.zip
Uploading the intermediate file: upload_file_110.zip
  18:12:04 STEP 2665 H=2665:00 +CPU= 25.574
  18:12:31 STEP 2666 H=2666:00 +CPU= 26.049
  18:13:12 STEP 2667 H=2667:00 +CPU= 39.722
  18:13:39 STEP 2668 H=2668:00 +CPU= 26.491
  18:14:05 STEP 2669 H=2669:00 +CPU= 26.241
  18:14:47 STEP 2670 H=2670:00 +CPU= 41.105
  18:15:13 STEP 2671 H=2671:00 +CPU= 25.622
  18:15:42 STEP 2672 H=2672:00 +CPU= 27.722
  18:16:21 STEP 2673 H=2673:00 +CPU= 38.632
  18:16:48 STEP 2674 H=2674:00 +CPU= 26.373
  18:17:14 STEP 2675 H=2675:00 +CPU= 25.645
  18:17:58 STEP 2676 H=2676:00 +CPU= 42.479
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002676
  18:18:26 STEP 2677 H=2677:00 +CPU= 27.737
  18:18:52 STEP 2678 H=2678:00 +CPU= 26.026
  18:19:32 STEP 2679 H=2679:00 +CPU= 38.630
  18:20:00 STEP 2680 H=2680:00 +CPU= 27.797
  18:20:25 STEP 2681 H=2681:00 +CPU= 24.799
  18:21:06 STEP 2682 H=2682:00 +CPU= 39.840
  18:21:33 STEP 2683 H=2683:00 +CPU= 25.934
  18:21:59 STEP 2684 H=2684:00 +CPU= 26.684
  18:22:40 STEP 2685 H=2685:00 +CPU= 39.442
  18:23:08 STEP 2686 H=2686:00 +CPU= 27.028
  18:23:35 STEP 2687 H=2687:00 +CPU= 26.279
  18:24:24 STEP 2688 H=2688:00 +CPU= 47.315
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765_111.zip
Uploading the intermediate file: upload_file_111.zip
  18:24:50 STEP 2689 H=2689:00 +CPU= 25.725
  18:25:16 STEP 2690 H=2690:00 +CPU= 25.665
  18:25:59 STEP 2691 H=2691:00 +CPU= 40.400
  18:26:25 STEP 2692 H=2692:00 +CPU= 26.366
  18:26:51 STEP 2693 H=2693:00 +CPU= 24.915
  18:27:33 STEP 2694 H=2694:00 +CPU= 40.278
  18:27:59 STEP 2695 H=2695:00 +CPU= 25.474
  18:28:28 STEP 2696 H=2696:00 +CPU= 27.482
  18:29:09 STEP 2697 H=2697:00 +CPU= 39.748
  18:29:35 STEP 2698 H=2698:00 +CPU= 25.861
  18:30:03 STEP 2699 H=2699:00 +CPU= 26.712
  18:30:48 STEP 2700 H=2700:00 +CPU= 43.926
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002700
  18:31:14 STEP 2701 H=2701:00 +CPU= 25.589
  18:31:42 STEP 2702 H=2702:00 +CPU= 27.421
  18:32:22 STEP 2703 H=2703:00 +CPU= 38.642
  18:32:51 STEP 2704 H=2704:00 +CPU= 27.739
  18:33:16 STEP 2705 H=2705:00 +CPU= 24.776
  18:33:57 STEP 2706 H=2706:00 +CPU= 39.328
  18:34:24 STEP 2707 H=2707:00 +CPU= 26.190
  18:34:50 STEP 2708 H=2708:00 +CPU= 25.525
  18:35:30 STEP 2709 H=2709:00 +CPU= 39.967
  18:35:57 STEP 2710 H=2710:00 +CPU= 26.250
  18:36:23 STEP 2711 H=2711:00 +CPU= 25.454
  18:37:12 STEP 2712 H=2712:00 +CPU= 46.756
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765_112.zip
Uploading the intermediate file: upload_file_112.zip
  18:37:38 STEP 2713 H=2713:00 +CPU= 25.183
  18:38:05 STEP 2714 H=2714:00 +CPU= 26.093
  18:38:46 STEP 2715 H=2715:00 +CPU= 39.647
  18:39:14 STEP 2716 H=2716:00 +CPU= 28.132
  18:39:40 STEP 2717 H=2717:00 +CPU= 25.468
  18:40:19 STEP 2718 H=2718:00 +CPU= 38.359
  18:40:48 STEP 2719 H=2719:00 +CPU= 26.981
  18:41:18 STEP 2720 H=2720:00 +CPU= 26.514
  18:41:58 STEP 2721 H=2721:00 +CPU= 39.181
  18:42:26 STEP 2722 H=2722:00 +CPU= 27.958
  18:42:52 STEP 2723 H=2723:00 +CPU= 25.014
  18:43:36 STEP 2724 H=2724:00 +CPU= 43.249
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002724
  18:44:05 STEP 2725 H=2725:00 +CPU= 27.733
  18:44:31 STEP 2726 H=2726:00 +CPU= 25.825
  18:45:11 STEP 2727 H=2727:00 +CPU= 39.227
  18:45:38 STEP 2728 H=2728:00 +CPU= 26.431
  18:46:03 STEP 2729 H=2729:00 +CPU= 24.913
  18:46:47 STEP 2730 H=2730:00 +CPU= 42.087
  18:47:13 STEP 2731 H=2731:00 +CPU= 25.313
  18:47:39 STEP 2732 H=2732:00 +CPU= 25.516
  18:48:20 STEP 2733 H=2733:00 +CPU= 39.916
  18:48:46 STEP 2734 H=2734:00 +CPU= 25.983
  18:49:12 STEP 2735 H=2735:00 +CPU= 25.391
  18:50:05 STEP 2736 H=2736:00 +CPU= 48.447
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765_113.zip
Uploading the intermediate file: upload_file_113.zip
  18:50:31 STEP 2737 H=2737:00 +CPU= 25.726
  18:50:57 STEP 2738 H=2738:00 +CPU= 25.287
  18:51:38 STEP 2739 H=2739:00 +CPU= 39.641
  18:52:05 STEP 2740 H=2740:00 +CPU= 26.717
  18:52:30 STEP 2741 H=2741:00 +CPU= 24.893
  18:53:12 STEP 2742 H=2742:00 +CPU= 40.313
  18:53:38 STEP 2743 H=2743:00 +CPU= 25.701
  18:54:06 STEP 2744 H=2744:00 +CPU= 26.980
  18:54:46 STEP 2745 H=2745:00 +CPU= 39.422
  18:55:13 STEP 2746 H=2746:00 +CPU= 26.071
  18:55:40 STEP 2747 H=2747:00 +CPU= 25.687
  18:56:23 STEP 2748 H=2748:00 +CPU= 42.542
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002748
  18:56:50 STEP 2749 H=2749:00 +CPU= 26.279
  18:57:16 STEP 2750 H=2750:00 +CPU= 25.872
  18:57:57 STEP 2751 H=2751:00 +CPU= 39.841
  18:58:25 STEP 2752 H=2752:00 +CPU= 26.548
  18:58:50 STEP 2753 H=2753:00 +CPU= 25.197
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0173
(argv4) batchid: 966
(argv5) wuid: 12182817
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d82d4c31f14e94be13bba96c335242d4 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0173_1997050100_123_966_12182817.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0173_1997050100_123_966_12182817.zip
ic_ancil_file: ic_ancil_12182817
ifsdata_file: ifsdata_12182817
climate_data_file: clim_data_12182817
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_79e60cbf34509c79781d88015bd51171 to: /var/lib/boinc-client/slots/11/ic_ancil_12182817.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12182817.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e62d99846d617b995ef659c6fa514bfc to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12182817.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12182817.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12182817.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12182817.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12182817.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12182817.xml
-- Model is restarting --
Adjusting last_iter, 2754, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12182817.xml
last_cpu_time: 76597
upload_file_number: 114
last_iter: 2735
last_upload: 9849600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765
The child process has been launched with process id: 2129776
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2129776:2129776] [20221223:190034:1671818434:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:01:27 STEP 2736 H=2736:00 +CPU= 45.833
  19:01:55 STEP 2737 H=2737:00 +CPU= 27.206
  19:02:22 STEP 2738 H=2738:00 +CPU= 25.928
  19:03:00 STEP 2739 H=2739:00 +CPU= 38.056
  19:03:29 STEP 2740 H=2740:00 +CPU= 27.148
  19:03:54 STEP 2741 H=2741:00 +CPU= 24.844
  19:04:37 STEP 2742 H=2742:00 +CPU= 40.757
  19:05:03 STEP 2743 H=2743:00 +CPU= 25.288
  19:05:29 STEP 2744 H=2744:00 +CPU= 26.326
  19:06:11 STEP 2745 H=2745:00 +CPU= 40.053
  19:06:37 STEP 2746 H=2746:00 +CPU= 25.522
  19:07:02 STEP 2747 H=2747:00 +CPU= 25.240
  19:07:47 STEP 2748 H=2748:00 +CPU= 44.350
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002748
  19:08:13 STEP 2749 H=2749:00 +CPU= 24.700
  19:08:40 STEP 2750 H=2750:00 +CPU= 26.619
  19:09:23 STEP 2751 H=2751:00 +CPU= 40.541
  19:09:49 STEP 2752 H=2752:00 +CPU= 25.425
  19:10:15 STEP 2753 H=2753:00 +CPU= 25.367
  19:10:57 STEP 2754 H=2754:00 +CPU= 40.304
  19:11:22 STEP 2755 H=2755:00 +CPU= 25.149
  19:11:50 STEP 2756 H=2756:00 +CPU= 27.720
  19:12:31 STEP 2757 H=2757:00 +CPU= 39.075
  19:12:57 STEP 2758 H=2758:00 +CPU= 25.969
  19:13:26 STEP 2759 H=2759:00 +CPU= 27.600
  19:14:12 STEP 2760 H=2760:00 +CPU= 45.128
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765_114.zip
Uploading the intermediate file: upload_file_114.zip
  19:14:39 STEP 2761 H=2761:00 +CPU= 26.386
  19:15:06 STEP 2762 H=2762:00 +CPU= 26.170
  19:15:46 STEP 2763 H=2763:00 +CPU= 38.534
  19:16:14 STEP 2764 H=2764:00 +CPU= 27.072
  19:16:40 STEP 2765 H=2765:00 +CPU= 25.131
  19:17:20 STEP 2766 H=2766:00 +CPU= 39.894
  19:17:46 STEP 2767 H=2767:00 +CPU= 25.202
  19:18:12 STEP 2768 H=2768:00 +CPU= 26.114
  19:18:55 STEP 2769 H=2769:00 +CPU= 40.729
  19:19:22 STEP 2770 H=2770:00 +CPU= 26.289
  19:19:48 STEP 2771 H=2771:00 +CPU= 26.041
  19:20:35 STEP 2772 H=2772:00 +CPU= 44.971
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002772
  19:21:03 STEP 2773 H=2773:00 +CPU= 26.495
  19:21:30 STEP 2774 H=2774:00 +CPU= 26.453
  19:22:09 STEP 2775 H=2775:00 +CPU= 37.981
  19:22:38 STEP 2776 H=2776:00 +CPU= 27.613
  19:23:04 STEP 2777 H=2777:00 +CPU= 25.581
  19:23:43 STEP 2778 H=2778:00 +CPU= 38.725
  19:24:12 STEP 2779 H=2779:00 +CPU= 26.843
  19:24:39 STEP 2780 H=2780:00 +CPU= 26.283
  19:25:18 STEP 2781 H=2781:00 +CPU= 39.058
  19:25:47 STEP 2782 H=2782:00 +CPU= 27.911
  19:26:13 STEP 2783 H=2783:00 +CPU= 25.913
  19:27:03 STEP 2784 H=2784:00 +CPU= 47.343
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765_115.zip
Uploading the intermediate file: upload_file_115.zip
  19:27:29 STEP 2785 H=2785:00 +CPU= 25.358
  19:27:56 STEP 2786 H=2786:00 +CPU= 26.665
  19:28:37 STEP 2787 H=2787:00 +CPU= 39.378
  19:29:03 STEP 2788 H=2788:00 +CPU= 26.050
  19:29:30 STEP 2789 H=2789:00 +CPU= 26.177
  19:30:12 STEP 2790 H=2790:00 +CPU= 40.444
  19:30:37 STEP 2791 H=2791:00 +CPU= 24.726
  19:31:06 STEP 2792 H=2792:00 +CPU= 28.219
  19:31:49 STEP 2793 H=2793:00 +CPU= 41.575
  19:32:15 STEP 2794 H=2794:00 +CPU= 25.819
  19:32:42 STEP 2795 H=2795:00 +CPU= 25.477
  19:33:29 STEP 2796 H=2796:00 +CPU= 44.948
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002796
  19:33:54 STEP 2797 H=2797:00 +CPU= 24.960
  19:34:21 STEP 2798 H=2798:00 +CPU= 26.455
  19:35:04 STEP 2799 H=2799:00 +CPU= 40.544
  19:35:30 STEP 2800 H=2800:00 +CPU= 26.166
  19:35:57 STEP 2801 H=2801:00 +CPU= 25.854
  19:36:39 STEP 2802 H=2802:00 +CPU= 40.483
  19:37:05 STEP 2803 H=2803:00 +CPU= 25.203
  19:37:33 STEP 2804 H=2804:00 +CPU= 27.320
  19:38:11 STEP 2805 H=2805:00 +CPU= 38.206
  19:38:38 STEP 2806 H=2806:00 +CPU= 25.725
  19:39:04 STEP 2807 H=2807:00 +CPU= 25.902
  19:39:50 STEP 2808 H=2808:00 +CPU= 45.762
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765_116.zip
Uploading the intermediate file: upload_file_116.zip
  19:40:20 STEP 2809 H=2809:00 +CPU= 27.433
  19:40:46 STEP 2810 H=2810:00 +CPU= 26.038
  19:41:26 STEP 2811 H=2811:00 +CPU= 38.766
  19:41:56 STEP 2812 H=2812:00 +CPU= 27.522
  19:42:25 STEP 2813 H=2813:00 +CPU= 27.690
  19:43:08 STEP 2814 H=2814:00 +CPU= 41.324
  19:43:34 STEP 2815 H=2815:00 +CPU= 25.312
  19:44:02 STEP 2816 H=2816:00 +CPU= 26.773
  19:44:45 STEP 2817 H=2817:00 +CPU= 41.217
  19:45:11 STEP 2818 H=2818:00 +CPU= 26.249
  19:45:38 STEP 2819 H=2819:00 +CPU= 25.909
  19:46:23 STEP 2820 H=2820:00 +CPU= 44.574
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002820
  19:46:48 STEP 2821 H=2821:00 +CPU= 25.034
  19:47:15 STEP 2822 H=2822:00 +CPU= 26.531
  19:47:57 STEP 2823 H=2823:00 +CPU= 40.478
  19:48:24 STEP 2824 H=2824:00 +CPU= 25.889
  19:48:49 STEP 2825 H=2825:00 +CPU= 25.357
  19:49:31 STEP 2826 H=2826:00 +CPU= 40.595
  19:49:57 STEP 2827 H=2827:00 +CPU= 24.851
  19:50:24 STEP 2828 H=2828:00 +CPU= 26.415
  19:51:03 STEP 2829 H=2829:00 +CPU= 38.072
  19:51:29 STEP 2830 H=2830:00 +CPU= 25.810
  19:51:58 STEP 2831 H=2831:00 +CPU= 27.352
  19:52:46 STEP 2832 H=2832:00 +CPU= 46.982
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765_117.zip
Uploading the intermediate file: upload_file_117.zip
  19:53:15 STEP 2833 H=2833:00 +CPU= 26.777
  19:53:41 STEP 2834 H=2834:00 +CPU= 25.725
  19:54:24 STEP 2835 H=2835:00 +CPU= 40.945
  19:54:50 STEP 2836 H=2836:00 +CPU= 25.810
  19:55:18 STEP 2837 H=2837:00 +CPU= 26.381
  19:55:58 STEP 2838 H=2838:00 +CPU= 39.549
  19:56:24 STEP 2839 H=2839:00 +CPU= 25.903
  19:56:52 STEP 2840 H=2840:00 +CPU= 26.490
  19:57:31 STEP 2841 H=2841:00 +CPU= 38.339
  19:57:59 STEP 2842 H=2842:00 +CPU= 27.913
  19:58:25 STEP 2843 H=2843:00 +CPU= 25.357
  19:59:09 STEP 2844 H=2844:00 +CPU= 42.966
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002844
  19:59:37 STEP 2845 H=2845:00 +CPU= 26.678
  20:00:04 STEP 2846 H=2846:00 +CPU= 25.538
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0173
(argv4) batchid: 966
(argv5) wuid: 12182817
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d82d4c31f14e94be13bba96c335242d4 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0173_1997050100_123_966_12182817.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0173_1997050100_123_966_12182817.zip
ic_ancil_file: ic_ancil_12182817
ifsdata_file: ifsdata_12182817
climate_data_file: clim_data_12182817
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_79e60cbf34509c79781d88015bd51171 to: /var/lib/boinc-client/slots/11/ic_ancil_12182817.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12182817.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e62d99846d617b995ef659c6fa514bfc to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12182817.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12182817.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12182817.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12182817.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12182817.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12182817.xml
-- Model is restarting --
Adjusting last_iter, 2847, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12182817.xml
last_cpu_time: 80092
upload_file_number: 118
last_iter: 2831
last_upload: 10195200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765
The child process has been launched with process id: 2138206
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:2138206:2138206] [20221223:200250:1671822170:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:03:45 STEP 2832 H=2832:00 +CPU= 46.929
  20:04:10 STEP 2833 H=2833:00 +CPU= 24.750
  20:04:37 STEP 2834 H=2834:00 +CPU= 26.906
  20:05:20 STEP 2835 H=2835:00 +CPU= 39.779
  20:05:45 STEP 2836 H=2836:00 +CPU= 25.547
  20:06:11 STEP 2837 H=2837:00 +CPU= 25.232
  20:06:52 STEP 2838 H=2838:00 +CPU= 39.996
  20:07:18 STEP 2839 H=2839:00 +CPU= 26.011
  20:07:45 STEP 2840 H=2840:00 +CPU= 26.642
  20:08:25 STEP 2841 H=2841:00 +CPU= 39.464
  20:08:53 STEP 2842 H=2842:00 +CPU= 27.258
  20:09:19 STEP 2843 H=2843:00 +CPU= 25.340
  20:10:04 STEP 2844 H=2844:00 +CPU= 44.287
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002844
  20:10:29 STEP 2845 H=2845:00 +CPU= 24.401
  20:10:55 STEP 2846 H=2846:00 +CPU= 26.187
  20:11:38 STEP 2847 H=2847:00 +CPU= 40.574
  20:12:05 STEP 2848 H=2848:00 +CPU= 25.242
  20:12:31 STEP 2849 H=2849:00 +CPU= 25.240
  20:13:14 STEP 2850 H=2850:00 +CPU= 41.488
  20:13:39 STEP 2851 H=2851:00 +CPU= 25.008
  20:14:06 STEP 2852 H=2852:00 +CPU= 26.175
  20:14:48 STEP 2853 H=2853:00 +CPU= 40.338
  20:15:13 STEP 2854 H=2854:00 +CPU= 25.525
  20:15:41 STEP 2855 H=2855:00 +CPU= 26.443
  20:16:31 STEP 2856 H=2856:00 +CPU= 46.904
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182817/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0173_1997050100_123_966_12182817_0_r2124487765_118.zip
Uploading the intermediate file: upload_file_118.zip
  20:16:56 STEP 2857 H=2857:00 +CPU= 23.882
  20:17:26 STEP 2858 H=2858:00 +CPU= 27.950
  20:18:05 STEP 2859 H=2859:00 +CPU= 39.014
  20:18:31 STEP 2860 H=2860:00 +CPU= 24.623
  20:19:00 STEP 2861 H=2861:00 +CPU= 27.719
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:50:20 1500195 22267952 oifs_43r3_ps_0173_1997050100_123_966_12182817_0 8,294,400 62,013 0.0075
09 May 2023 15:50:09 1500195 22267952 oifs_43r3_ps_0173_1997050100_123_966_12182817_0 7,430,400 56,393 0.0076
09 May 2023 15:50:03 1500195 22267952 oifs_43r3_ps_0173_1997050100_123_966_12182817_0 6,566,400 50,786 0.0077
09 May 2023 15:49:52 1500195 22267952 oifs_43r3_ps_0173_1997050100_123_966_12182817_0 5,702,400 45,169 0.0079
09 May 2023 15:49:42 1500195 22267952 oifs_43r3_ps_0173_1997050100_123_966_12182817_0 4,838,400 39,543 0.0082
09 May 2023 15:49:36 1500195 22267952 oifs_43r3_ps_0173_1997050100_123_966_12182817_0 3,974,400 33,917 0.0085
09 May 2023 15:49:19 1500195 22267952 oifs_43r3_ps_0173_1997050100_123_966_12182817_0 2,678,400 23,613 0.0088
09 May 2023 15:49:08 1500195 22267952 oifs_43r3_ps_0173_1997050100_123_966_12182817_0 1,814,400 16,923 0.0093
09 May 2023 15:48:57 1500195 22267952 oifs_43r3_ps_0173_1997050100_123_966_12182817_0 950,400 10,455 0.0110


©2024 climateprediction.net