climateprediction.net home page
Task 22260412

Task 22260412

Name oifs_43r3_ps_0061_1990050100_123_959_12175705_0
Workunit 12175705
Created 21 Dec 2022, 18:12:15 UTC
Sent 11 Jan 2023, 0:31:31 UTC
Report deadline 10 Feb 2023, 0:31:31 UTC
Received 16 Jan 2023, 0:30:10 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1534673
Run time 1 days 8 hours 42 min 58 sec
CPU time 15 hours 24 min 22 sec
Validate state Invalid
Credit 13,910.40
Device peak FLOPS 4.67 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,414.36 MB
Peak swap size 4,978.16 MB
Peak disk usage 7,716.97 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
nd request received from the BOINC client, suspending the child process
Resuming the child process
  13:58:35 STEP 2599 H=2599:00 +CPU= 36.587
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:02:58 STEP 2600 H=2600:00 +CPU= 38.093
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:06:42 STEP 2601 H=2601:00 +CPU= 57.356
  14:07:22 STEP 2602 H=2602:00 +CPU= 37.822
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:09:29 STEP 2603 H=2603:00 +CPU= 33.624
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:12:13 STEP 2604 H=2604:00 +CPU= 60.524
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002604
  14:12:50 STEP 2605 H=2605:00 +CPU= 35.173
  14:13:30 STEP 2606 H=2606:00 +CPU= 38.457
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:14:40 STEP 2607 H=2607:00 +CPU= 57.229
  14:15:19 STEP 2608 H=2608:00 +CPU= 37.847
  14:15:58 STEP 2609 H=2609:00 +CPU= 37.226
  14:16:55 STEP 2610 H=2610:00 +CPU= 55.632
  14:17:32 STEP 2611 H=2611:00 +CPU= 35.340
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:18:51 STEP 2612 H=2612:00 +CPU= 35.954
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:19:56 STEP 2613 H=2613:00 +CPU= 52.334
  14:20:34 STEP 2614 H=2614:00 +CPU= 36.604
  14:21:10 STEP 2615 H=2615:00 +CPU= 35.485
  14:22:16 STEP 2616 H=2616:00 +CPU= 63.321
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206_42.zip
Uploading the intermediate file: upload_file_42.zip
  14:22:52 STEP 2617 H=2617:00 +CPU= 35.545
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:24:40 STEP 2618 H=2618:00 +CPU= 35.070
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:30:38 STEP 2619 H=2619:00 +CPU= 52.486
  14:31:15 STEP 2620 H=2620:00 +CPU= 35.290
  14:31:50 STEP 2621 H=2621:00 +CPU= 34.211
  14:32:46 STEP 2622 H=2622:00 +CPU= 53.832
  14:33:23 STEP 2623 H=2623:00 +CPU= 35.876
  14:34:01 STEP 2624 H=2624:00 +CPU= 37.218
  14:34:57 STEP 2625 H=2625:00 +CPU= 54.773
  14:35:35 STEP 2626 H=2626:00 +CPU= 36.475
  14:36:11 STEP 2627 H=2627:00 +CPU= 35.379
  14:37:17 STEP 2628 H=2628:00 +CPU= 64.092
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002628
  14:37:55 STEP 2629 H=2629:00 +CPU= 37.312
  14:38:33 STEP 2630 H=2630:00 +CPU= 37.194
  14:39:28 STEP 2631 H=2631:00 +CPU= 53.596
  14:40:04 STEP 2632 H=2632:00 +CPU= 34.891
  14:40:42 STEP 2633 H=2633:00 +CPU= 37.188
  14:41:41 STEP 2634 H=2634:00 +CPU= 57.298
  14:42:19 STEP 2635 H=2635:00 +CPU= 36.081
  14:42:58 STEP 2636 H=2636:00 +CPU= 38.713
  14:43:58 STEP 2637 H=2637:00 +CPU= 58.299
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:51:43 STEP 2638 H=2638:00 +CPU= 38.423
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:58:37 STEP 2639 H=2639:00 +CPU= 37.810
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:03:50 STEP 2640 H=2640:00 +CPU= 65.334
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206_43.zip
Uploading the intermediate file: upload_file_43.zip
  15:05:16 STEP 2641 H=2641:00 +CPU= 25.079
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:37:57 STEP 2642 H=2642:00 +CPU= 34.521
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:39:04 STEP 2643 H=2643:00 +CPU= 54.589
  15:39:42 STEP 2644 H=2644:00 +CPU= 37.025
  15:40:18 STEP 2645 H=2645:00 +CPU= 34.095
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:52:00 STEP 2646 H=2646:00 +CPU= 53.652
  15:52:36 STEP 2647 H=2647:00 +CPU= 35.256
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:56:56 STEP 2648 H=2648:00 +CPU= 35.857
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:58:32 STEP 2649 H=2649:00 +CPU= 53.499
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:02:42 STEP 2650 H=2650:00 +CPU= 34.825
  16:03:17 STEP 2651 H=2651:00 +CPU= 33.403
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:07:48 STEP 2652 H=2652:00 +CPU= 55.130
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002652
  16:08:24 STEP 2653 H=2653:00 +CPU= 34.435
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:09:21 STEP 2654 H=2654:00 +CPU= 34.936
  16:10:18 STEP 2655 H=2655:00 +CPU= 54.220
  16:10:53 STEP 2656 H=2656:00 +CPU= 33.024
  16:11:27 STEP 2657 H=2657:00 +CPU= 32.444
  16:12:21 STEP 2658 H=2658:00 +CPU= 51.408
  16:12:56 STEP 2659 H=2659:00 +CPU= 34.025
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:13:43 STEP 2660 H=2660:00 +CPU= 34.930
  16:14:37 STEP 2661 H=2661:00 +CPU= 52.191
  16:15:13 STEP 2662 H=2662:00 +CPU= 34.308
  16:15:48 STEP 2663 H=2663:00 +CPU= 34.072
  16:16:52 STEP 2664 H=2664:00 +CPU= 61.329
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206_44.zip
Uploading the intermediate file: upload_file_44.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:17:48 STEP 2665 H=2665:00 +CPU= 33.078
  16:18:23 STEP 2666 H=2666:00 +CPU= 34.377
  16:19:16 STEP 2667 H=2667:00 +CPU= 50.699
  16:19:51 STEP 2668 H=2668:00 +CPU= 32.895
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:20:47 STEP 2669 H=2669:00 +CPU= 33.935
  16:21:41 STEP 2670 H=2670:00 +CPU= 52.535
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:22:56 STEP 2671 H=2671:00 +CPU= 32.608
  16:23:32 STEP 2672 H=2672:00 +CPU= 34.288
  16:24:25 STEP 2673 H=2673:00 +CPU= 50.505
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:30:54 STEP 2674 H=2674:00 +CPU= 32.707
  16:31:29 STEP 2675 H=2675:00 +CPU= 33.585
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:34:02 STEP 2676 H=2676:00 +CPU= 57.929
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002676
  16:34:38 STEP 2677 H=2677:00 +CPU= 33.804
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:46 STEP 2678 H=2678:00 +CPU= 32.814
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:39:44 STEP 2679 H=2679:00 +CPU= 52.348
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:41:22 STEP 2680 H=2680:00 +CPU= 32.392
  16:41:57 STEP 2681 H=2681:00 +CPU= 33.614
  16:42:50 STEP 2682 H=2682:00 +CPU= 48.643
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:45:46 STEP 2683 H=2683:00 +CPU= 31.519
  16:46:20 STEP 2684 H=2684:00 +CPU= 32.248
  16:47:12 STEP 2685 H=2685:00 +CPU= 47.441
  16:47:45 STEP 2686 H=2686:00 +CPU= 30.524
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:00:23 STEP 2687 H=2687:00 +CPU= 30.615
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:02:37 STEP 2688 H=2688:00 +CPU= 55.444
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206_45.zip
Uploading the intermediate file: upload_file_45.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:03:30 STEP 2689 H=2689:00 +CPU= 29.712
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:07:29 STEP 2690 H=2690:00 +CPU= 30.787
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:14:36 STEP 2691 H=2691:00 +CPU= 47.322
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:18:15 STEP 2692 H=2692:00 +CPU= 32.895
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:19:43 STEP 2693 H=2693:00 +CPU= 35.607
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:23:42 STEP 2694 H=2694:00 +CPU= 53.730
  18:24:17 STEP 2695 H=2695:00 +CPU= 33.347
  18:24:51 STEP 2696 H=2696:00 +CPU= 33.120
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:27:29 STEP 2697 H=2697:00 +CPU= 52.818
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:28:26 STEP 2698 H=2698:00 +CPU= 35.210
  18:29:01 STEP 2699 H=2699:00 +CPU= 33.486
  18:30:00 STEP 2700 H=2700:00 +CPU= 57.174
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002700
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:30:45 STEP 2701 H=2701:00 +CPU= 33.231
  18:31:20 STEP 2702 H=2702:00 +CPU= 34.511
  18:32:14 STEP 2703 H=2703:00 +CPU= 52.003
  18:32:50 STEP 2704 H=2704:00 +CPU= 34.727
  18:33:24 STEP 2705 H=2705:00 +CPU= 33.193
  18:34:19 STEP 2706 H=2706:00 +CPU= 53.314
  18:34:52 STEP 2707 H=2707:00 +CPU= 32.883
  18:35:27 STEP 2708 H=2708:00 +CPU= 33.908
  18:36:20 STEP 2709 H=2709:00 +CPU= 51.401
  18:36:55 STEP 2710 H=2710:00 +CPU= 33.879
  18:37:32 STEP 2711 H=2711:00 +CPU= 34.521
  18:38:36 STEP 2712 H=2712:00 +CPU= 60.514
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206_46.zip
Uploading the intermediate file: upload_file_46.zip
  18:39:10 STEP 2713 H=2713:00 +CPU= 33.990
  18:39:46 STEP 2714 H=2714:00 +CPU= 34.174
  18:40:40 STEP 2715 H=2715:00 +CPU= 52.509
  18:41:17 STEP 2716 H=2716:00 +CPU= 36.036
  18:41:52 STEP 2717 H=2717:00 +CPU= 34.637
  18:42:49 STEP 2718 H=2718:00 +CPU= 54.789
  18:43:26 STEP 2719 H=2719:00 +CPU= 35.972
  18:44:01 STEP 2720 H=2720:00 +CPU= 34.752
  18:44:57 STEP 2721 H=2721:00 +CPU= 53.274
  18:45:33 STEP 2722 H=2722:00 +CPU= 34.801
  18:46:10 STEP 2723 H=2723:00 +CPU= 36.021
  18:47:09 STEP 2724 H=2724:00 +CPU= 56.904
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002724
  18:47:43 STEP 2725 H=2725:00 +CPU= 32.552
  18:48:19 STEP 2726 H=2726:00 +CPU= 34.688
  18:49:14 STEP 2727 H=2727:00 +CPU= 53.543
  18:49:49 STEP 2728 H=2728:00 +CPU= 34.366
  18:50:25 STEP 2729 H=2729:00 +CPU= 34.285
  18:51:21 STEP 2730 H=2730:00 +CPU= 54.207
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:52:07 STEP 2731 H=2731:00 +CPU= 33.229
  18:52:43 STEP 2732 H=2732:00 +CPU= 35.251
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:53:49 STEP 2733 H=2733:00 +CPU= 51.574
  18:54:22 STEP 2734 H=2734:00 +CPU= 30.570
  18:54:56 STEP 2735 H=2735:00 +CPU= 32.471
  18:56:00 STEP 2736 H=2736:00 +CPU= 61.029
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206_47.zip
Uploading the intermediate file: upload_file_47.zip
  18:56:34 STEP 2737 H=2737:00 +CPU= 31.670
  18:57:10 STEP 2738 H=2738:00 +CPU= 34.238
  18:58:04 STEP 2739 H=2739:00 +CPU= 51.754
  18:58:40 STEP 2740 H=2740:00 +CPU= 33.716
  18:59:18 STEP 2741 H=2741:00 +CPU= 36.666
  19:00:14 STEP 2742 H=2742:00 +CPU= 54.141
  19:00:47 STEP 2743 H=2743:00 +CPU= 32.526
  19:01:22 STEP 2744 H=2744:00 +CPU= 33.194
  19:02:15 STEP 2745 H=2745:00 +CPU= 50.722
  19:02:50 STEP 2746 H=2746:00 +CPU= 34.064
  19:03:24 STEP 2747 H=2747:00 +CPU= 32.577
  19:04:23 STEP 2748 H=2748:00 +CPU= 57.266
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002748
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  19:10:02 STEP 2749 H=2749:00 +CPU= 32.303
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:18:00 STEP 2750 H=2750:00 +CPU= 33.083
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:33:33 STEP 2751 H=2751:00 +CPU= 51.034
  20:34:10 STEP 2752 H=2752:00 +CPU= 35.840
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:35:37 STEP 2753 H=2753:00 +CPU= 33.456
  20:36:33 STEP 2754 H=2754:00 +CPU= 53.519
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:39:11 STEP 2755 H=2755:00 +CPU= 33.387
  20:39:46 STEP 2756 H=2756:00 +CPU= 33.589
  20:40:40 STEP 2757 H=2757:00 +CPU= 51.585
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:43:58 STEP 2758 H=2758:00 +CPU= 34.436
  20:44:35 STEP 2759 H=2759:00 +CPU= 34.577
  20:45:40 STEP 2760 H=2760:00 +CPU= 62.155
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206_48.zip
Uploading the intermediate file: upload_file_48.zip
Uploading trickle at timestep: 9936000
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:46:19 STEP 2761 H=2761:00 +CPU= 27.099
  20:46:54 STEP 2762 H=2762:00 +CPU= 34.272
  20:47:50 STEP 2763 H=2763:00 +CPU= 52.442
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:48:57 STEP 2764 H=2764:00 +CPU= 34.327
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:50:14 STEP 2765 H=2765:00 +CPU= 34.255
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:55:46 STEP 2766 H=2766:00 +CPU= 52.473
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:00:21 STEP 2767 H=2767:00 +CPU= 29.833
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:02:44 STEP 2768 H=2768:00 +CPU= 30.456
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:04:07 STEP 2769 H=2769:00 +CPU= 46.983
  21:04:39 STEP 2770 H=2770:00 +CPU= 30.059
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:05:22 STEP 2771 H=2771:00 +CPU= 29.565
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:07:59 STEP 2772 H=2772:00 +CPU= 51.303
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002772
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:10:03 STEP 2773 H=2773:00 +CPU= 29.575
  21:10:36 STEP 2774 H=2774:00 +CPU= 30.472
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:13:15 STEP 2775 H=2775:00 +CPU= 52.197
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:14:33 STEP 2776 H=2776:00 +CPU= 34.390
  21:15:09 STEP 2777 H=2777:00 +CPU= 33.733
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:16:16 STEP 2778 H=2778:00 +CPU= 53.263
  21:16:52 STEP 2779 H=2779:00 +CPU= 33.491
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:18:20 STEP 2780 H=2780:00 +CPU= 34.610
  21:19:15 STEP 2781 H=2781:00 +CPU= 51.169
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:20:11 STEP 2782 H=2782:00 +CPU= 33.268
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:22:26 STEP 2783 H=2783:00 +CPU= 32.523
  21:23:28 STEP 2784 H=2784:00 +CPU= 57.909
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206_49.zip
Uploading the intermediate file: upload_file_49.zip
  21:24:01 STEP 2785 H=2785:00 +CPU= 32.273
  21:24:36 STEP 2786 H=2786:00 +CPU= 32.792
  21:25:29 STEP 2787 H=2787:00 +CPU= 50.716
  21:26:05 STEP 2788 H=2788:00 +CPU= 33.748
  21:26:38 STEP 2789 H=2789:00 +CPU= 31.729
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:28:53 STEP 2790 H=2790:00 +CPU= 49.635
  21:29:26 STEP 2791 H=2791:00 +CPU= 31.501
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:31:31 STEP 2792 H=2792:00 +CPU= 31.730
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:40:20 STEP 2793 H=2793:00 +CPU= 50.317
  21:40:56 STEP 2794 H=2794:00 +CPU= 34.183
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:42:50 STEP 2795 H=2795:00 +CPU= 32.380
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:44:43 STEP 2796 H=2796:00 +CPU= 56.735
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002796
  21:45:16 STEP 2797 H=2797:00 +CPU= 31.744
  21:45:51 STEP 2798 H=2798:00 +CPU= 32.971
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:54:58 STEP 2799 H=2799:00 +CPU= 49.745
  21:55:33 STEP 2800 H=2800:00 +CPU= 34.043
  21:56:09 STEP 2801 H=2801:00 +CPU= 34.152
  21:57:06 STEP 2802 H=2802:00 +CPU= 54.693
  21:57:42 STEP 2803 H=2803:00 +CPU= 34.190
  21:58:18 STEP 2804 H=2804:00 +CPU= 33.580
  21:59:13 STEP 2805 H=2805:00 +CPU= 52.212
  21:59:47 STEP 2806 H=2806:00 +CPU= 33.338
  22:00:23 STEP 2807 H=2807:00 +CPU= 34.343
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:01:49 STEP 2808 H=2808:00 +CPU= 62.135
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206_50.zip
Uploading the intermediate file: upload_file_50.zip
  22:02:18 STEP 2809 H=2809:00 +CPU= 27.615
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:10:07 STEP 2810 H=2810:00 +CPU= 33.558
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:13:05 STEP 2811 H=2811:00 +CPU= 51.667
  22:13:35 STEP 2812 H=2812:00 +CPU= 29.407
  22:14:02 STEP 2813 H=2813:00 +CPU= 27.532
  22:14:46 STEP 2814 H=2814:00 +CPU= 43.696
  22:15:14 STEP 2815 H=2815:00 +CPU= 28.013
  22:15:43 STEP 2816 H=2816:00 +CPU= 28.768
  22:16:26 STEP 2817 H=2817:00 +CPU= 42.021
  22:16:52 STEP 2818 H=2818:00 +CPU= 26.620
  22:17:19 STEP 2819 H=2819:00 +CPU= 26.600
  22:18:08 STEP 2820 H=2820:00 +CPU= 48.321
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002820
  22:18:35 STEP 2821 H=2821:00 +CPU= 27.033
  22:19:03 STEP 2822 H=2822:00 +CPU= 27.720
  22:19:47 STEP 2823 H=2823:00 +CPU= 43.946
  22:20:14 STEP 2824 H=2824:00 +CPU= 27.002
  22:20:39 STEP 2825 H=2825:00 +CPU= 25.281
  22:21:20 STEP 2826 H=2826:00 +CPU= 40.369
  22:21:47 STEP 2827 H=2827:00 +CPU= 26.400
  22:22:17 STEP 2828 H=2828:00 +CPU= 29.864
  22:23:02 STEP 2829 H=2829:00 +CPU= 44.746
  22:23:29 STEP 2830 H=2830:00 +CPU= 27.475
  22:23:59 STEP 2831 H=2831:00 +CPU= 29.324
  22:24:48 STEP 2832 H=2832:00 +CPU= 49.353
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206_51.zip
Uploading the intermediate file: upload_file_51.zip
  22:25:15 STEP 2833 H=2833:00 +CPU= 26.674
  22:25:42 STEP 2834 H=2834:00 +CPU= 27.069
  22:26:25 STEP 2835 H=2835:00 +CPU= 42.819
  22:26:53 STEP 2836 H=2836:00 +CPU= 27.254
  22:27:20 STEP 2837 H=2837:00 +CPU= 26.767
  22:28:04 STEP 2838 H=2838:00 +CPU= 43.872
  22:28:30 STEP 2839 H=2839:00 +CPU= 25.465
  22:28:55 STEP 2840 H=2840:00 +CPU= 25.584
  22:29:37 STEP 2841 H=2841:00 +CPU= 41.810
  22:30:03 STEP 2842 H=2842:00 +CPU= 25.706
  22:30:30 STEP 2843 H=2843:00 +CPU= 26.603
  22:31:15 STEP 2844 H=2844:00 +CPU= 45.130
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002844
  22:31:42 STEP 2845 H=2845:00 +CPU= 26.650
  22:32:08 STEP 2846 H=2846:00 +CPU= 26.106
  22:32:49 STEP 2847 H=2847:00 +CPU= 39.782
  22:33:14 STEP 2848 H=2848:00 +CPU= 25.170
  22:33:38 STEP 2849 H=2849:00 +CPU= 23.967
  22:34:18 STEP 2850 H=2850:00 +CPU= 39.756
  22:34:42 STEP 2851 H=2851:00 +CPU= 23.789
  22:35:07 STEP 2852 H=2852:00 +CPU= 24.531
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1990050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0061
(argv4) batchid: 959
(argv5) wuid: 12175705
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a98528291788e16e6d843a3730735cbf to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0061_1990050100_123_959_12175705.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0061_1990050100_123_959_12175705.zip
ic_ancil_file: ic_ancil_12175705
ifsdata_file: ifsdata_12175705
climate_data_file: clim_data_12175705
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8756660b186fce6a0e7a67257f6d44fc to: /var/lib/boinc-client/slots/6/ic_ancil_12175705.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12175705.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_af270e1a4f15c360b2038293c366944b to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12175705.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12175705.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12175705.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12175705.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12175705.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12175705.xml
-- Model is restarting --
Adjusting last_iter, 2853, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12175705.xml
last_cpu_time: 52317
upload_file_number: 52
last_iter: 2831
last_upload: 10195200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206
The child process has been launched with process id: 360721
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:360721:360721] [20230115:170822:1673773702:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:09:12 STEP 2832 H=2832:00 +CPU= 45.941
  17:09:38 STEP 2833 H=2833:00 +CPU= 26.480
  17:10:03 STEP 2834 H=2834:00 +CPU= 25.040
  17:10:43 STEP 2835 H=2835:00 +CPU= 38.881
  17:11:08 STEP 2836 H=2836:00 +CPU= 24.844
  17:11:33 STEP 2837 H=2837:00 +CPU= 25.370
  17:12:14 STEP 2838 H=2838:00 +CPU= 41.267
  17:12:39 STEP 2839 H=2839:00 +CPU= 24.881
  17:13:05 STEP 2840 H=2840:00 +CPU= 25.618
  17:13:45 STEP 2841 H=2841:00 +CPU= 39.625
  17:14:12 STEP 2842 H=2842:00 +CPU= 26.969
  17:14:38 STEP 2843 H=2843:00 +CPU= 25.898
  17:15:21 STEP 2844 H=2844:00 +CPU= 43.438
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002844
  17:15:46 STEP 2845 H=2845:00 +CPU= 24.336
  17:16:11 STEP 2846 H=2846:00 +CPU= 24.853
  17:16:50 STEP 2847 H=2847:00 +CPU= 39.570
  17:17:17 STEP 2848 H=2848:00 +CPU= 26.563
  17:17:43 STEP 2849 H=2849:00 +CPU= 25.824
  17:18:22 STEP 2850 H=2850:00 +CPU= 39.175
  17:18:49 STEP 2851 H=2851:00 +CPU= 27.055
  17:19:16 STEP 2852 H=2852:00 +CPU= 26.153
  17:19:56 STEP 2853 H=2853:00 +CPU= 39.931
  17:20:22 STEP 2854 H=2854:00 +CPU= 26.621
  17:20:49 STEP 2855 H=2855:00 +CPU= 26.075
  17:21:35 STEP 2856 H=2856:00 +CPU= 46.664
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206_52.zip
Uploading the intermediate file: upload_file_52.zip
  17:22:01 STEP 2857 H=2857:00 +CPU= 25.422
  17:22:27 STEP 2858 H=2858:00 +CPU= 26.105
  17:23:08 STEP 2859 H=2859:00 +CPU= 41.054
  17:23:36 STEP 2860 H=2860:00 +CPU= 27.219
  17:24:01 STEP 2861 H=2861:00 +CPU= 25.191
  17:24:42 STEP 2862 H=2862:00 +CPU= 40.967
  17:25:08 STEP 2863 H=2863:00 +CPU= 25.849
  17:25:34 STEP 2864 H=2864:00 +CPU= 26.010
  17:26:17 STEP 2865 H=2865:00 +CPU= 43.077
  17:26:43 STEP 2866 H=2866:00 +CPU= 25.732
  17:27:08 STEP 2867 H=2867:00 +CPU= 24.944
  17:27:54 STEP 2868 H=2868:00 +CPU= 45.324
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002868
  17:28:18 STEP 2869 H=2869:00 +CPU= 24.771
  17:28:45 STEP 2870 H=2870:00 +CPU= 26.334
  17:29:27 STEP 2871 H=2871:00 +CPU= 41.789
  17:29:54 STEP 2872 H=2872:00 +CPU= 27.631
  17:30:22 STEP 2873 H=2873:00 +CPU= 27.446
  17:31:05 STEP 2874 H=2874:00 +CPU= 43.329
  17:31:32 STEP 2875 H=2875:00 +CPU= 26.402
  17:32:00 STEP 2876 H=2876:00 +CPU= 27.456
  17:32:40 STEP 2877 H=2877:00 +CPU= 40.638
  17:33:07 STEP 2878 H=2878:00 +CPU= 26.447
  17:33:36 STEP 2879 H=2879:00 +CPU= 28.967
  17:34:24 STEP 2880 H=2880:00 +CPU= 48.364
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206_53.zip
Uploading the intermediate file: upload_file_53.zip
Finished the upload of the intermediate file: upload_file_53.zip
  17:34:50 STEP 2881 H=2881:00 +CPU= 25.406
  17:35:16 STEP 2882 H=2882:00 +CPU= 26.498
  17:36:00 STEP 2883 H=2883:00 +CPU= 43.611
  17:36:28 STEP 2884 H=2884:00 +CPU= 27.469
  17:36:55 STEP 2885 H=2885:00 +CPU= 26.817
  17:37:37 STEP 2886 H=2886:00 +CPU= 41.808
  17:38:03 STEP 2887 H=2887:00 +CPU= 26.607
  17:38:30 STEP 2888 H=2888:00 +CPU= 26.134
  17:39:11 STEP 2889 H=2889:00 +CPU= 40.800
  17:39:38 STEP 2890 H=2890:00 +CPU= 27.584
  17:40:05 STEP 2891 H=2891:00 +CPU= 26.770
  17:40:52 STEP 2892 H=2892:00 +CPU= 46.806
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002892
  17:41:20 STEP 2893 H=2893:00 +CPU= 28.035
  17:41:47 STEP 2894 H=2894:00 +CPU= 26.731
  17:42:30 STEP 2895 H=2895:00 +CPU= 42.443
  17:42:57 STEP 2896 H=2896:00 +CPU= 27.898
  17:43:24 STEP 2897 H=2897:00 +CPU= 26.878
  17:44:08 STEP 2898 H=2898:00 +CPU= 43.084
  17:44:34 STEP 2899 H=2899:00 +CPU= 26.313
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1990050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0061
(argv4) batchid: 959
(argv5) wuid: 12175705
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a98528291788e16e6d843a3730735cbf to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0061_1990050100_123_959_12175705.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0061_1990050100_123_959_12175705.zip
ic_ancil_file: ic_ancil_12175705
ifsdata_file: ifsdata_12175705
climate_data_file: clim_data_12175705
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8756660b186fce6a0e7a67257f6d44fc to: /var/lib/boinc-client/slots/6/ic_ancil_12175705.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12175705.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_af270e1a4f15c360b2038293c366944b to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12175705.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12175705.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12175705.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12175705.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12175705.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12175705.xml
-- Model is restarting --
Adjusting last_iter, 2900, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12175705.xml
last_cpu_time: 54503
upload_file_number: 54
last_iter: 2879
last_upload: 10368000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206
The child process has been launched with process id: 392803
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:392803:392803] [20230116:060004:1673820004:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:00:51 STEP 2880 H=2880:00 +CPU= 43.873
  06:01:16 STEP 2881 H=2881:00 +CPU= 25.621
  06:01:43 STEP 2882 H=2882:00 +CPU= 26.808
  06:02:26 STEP 2883 H=2883:00 +CPU= 43.346
  06:02:53 STEP 2884 H=2884:00 +CPU= 26.024
  06:03:19 STEP 2885 H=2885:00 +CPU= 26.857
  06:04:02 STEP 2886 H=2886:00 +CPU= 42.224
  06:04:27 STEP 2887 H=2887:00 +CPU= 25.095
  06:04:53 STEP 2888 H=2888:00 +CPU= 26.300
  06:05:33 STEP 2889 H=2889:00 +CPU= 40.300
  06:06:04 STEP 2890 H=2890:00 +CPU= 30.270
  06:06:30 STEP 2891 H=2891:00 +CPU= 26.653
  06:07:15 STEP 2892 H=2892:00 +CPU= 44.168
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002892
  06:07:41 STEP 2893 H=2893:00 +CPU= 26.000
  06:08:07 STEP 2894 H=2894:00 +CPU= 26.084
  06:08:46 STEP 2895 H=2895:00 +CPU= 39.616
  06:09:12 STEP 2896 H=2896:00 +CPU= 25.310
  06:09:38 STEP 2897 H=2897:00 +CPU= 26.275
  06:10:19 STEP 2898 H=2898:00 +CPU= 41.300
  06:10:46 STEP 2899 H=2899:00 +CPU= 26.452
  06:11:12 STEP 2900 H=2900:00 +CPU= 26.119
  06:11:50 STEP 2901 H=2901:00 +CPU= 38.525
  06:12:16 STEP 2902 H=2902:00 +CPU= 25.441
  06:12:42 STEP 2903 H=2903:00 +CPU= 26.096
  06:13:32 STEP 2904 H=2904:00 +CPU= 50.434
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175705/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0061_1990050100_123_959_12175705_0_r2115993206_54.zip
Uploading the intermediate file: upload_file_54.zip
  06:13:58 STEP 2905 H=2905:00 +CPU= 25.857
  06:14:27 STEP 2906 H=2906:00 +CPU= 29.092
  06:15:07 STEP 2907 H=2907:00 +CPU= 39.951
  06:15:33 STEP 2908 H=2908:00 +CPU= 25.331
  06:15:58 STEP 2909 H=2909:00 +CPU= 25.750
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:18:35 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 9,936,000 49,007 0.0049
09 May 2023 16:17:30 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 9,072,000 38,494 0.0042
09 May 2023 16:17:22 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 8,035,200 26,240 0.0033
09 May 2023 16:17:22 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 8,035,200 26,240 0.0033
09 May 2023 16:17:22 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 7,171,200 16,652 0.0023
09 May 2023 16:16:17 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 5,702,400 58,120 0.0102
09 May 2023 16:16:05 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 4,838,400 49,848 0.0103
09 May 2023 16:16:05 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 4,838,400 49,848 0.0103
09 May 2023 16:16:05 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 3,456,000 36,290 0.0105
09 May 2023 16:16:05 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 3,456,000 36,290 0.0105
09 May 2023 16:16:05 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 2,592,000 27,668 0.0107
09 May 2023 16:16:05 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 1,728,000 18,566 0.0107
09 May 2023 16:16:05 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 1,728,000 18,566 0.0107
09 May 2023 16:15:23 1534673 22260412 oifs_43r3_ps_0061_1990050100_123_959_12175705_0 864,000 9,084 0.0105


©2024 climateprediction.net