climateprediction.net home page
Task 22258318

Task 22258318

Name oifs_43r3_ps_0094_1988050100_123_957_12173738_0
Workunit 12173738
Created 21 Dec 2022, 13:00:30 UTC
Sent 21 Dec 2022, 13:57:11 UTC
Report deadline 20 Jan 2023, 13:57:11 UTC
Received 25 Dec 2022, 18:27:01 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1534737
Run time 8 hours 34 min 12 sec
CPU time 8 hours 17 min 15 sec
Validate state Invalid
Credit 4,838.40
Device peak FLOPS 5.53 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,393.65 MB
Peak swap size 4,974.12 MB
Peak disk usage 1,213.09 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
431
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  19:40:18 STEP  538 H= 538:00 +CPU= 28.613
  19:40:47 STEP  539 H= 539:00 +CPU= 26.910
  19:41:36 STEP  540 H= 540:00 +CPU= 46.593
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000540
  19:42:04 STEP  541 H= 541:00 +CPU= 26.647
  19:42:34 STEP  542 H= 542:00 +CPU= 28.225
  19:43:18 STEP  543 H= 543:00 +CPU= 41.952
  19:43:47 STEP  544 H= 544:00 +CPU= 27.683
  19:44:16 STEP  545 H= 545:00 +CPU= 27.322
  19:45:01 STEP  546 H= 546:00 +CPU= 43.108
  19:45:30 STEP  547 H= 547:00 +CPU= 27.208
  19:45:59 STEP  548 H= 548:00 +CPU= 27.560
  19:46:45 STEP  549 H= 549:00 +CPU= 42.706
  19:47:14 STEP  550 H= 550:00 +CPU= 27.536
  19:47:43 STEP  551 H= 551:00 +CPU= 28.109
  19:48:37 STEP  552 H= 552:00 +CPU= 50.303
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000540
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_22.zip
Uploading the intermediate file: upload_file_22.zip
  19:49:06 STEP  553 H= 553:00 +CPU= 27.550
  19:49:35 STEP  554 H= 554:00 +CPU= 27.679
  19:50:20 STEP  555 H= 555:00 +CPU= 42.382
  19:50:50 STEP  556 H= 556:00 +CPU= 28.058
  19:51:18 STEP  557 H= 557:00 +CPU= 27.385
  19:52:04 STEP  558 H= 558:00 +CPU= 42.962
  19:52:32 STEP  559 H= 559:00 +CPU= 26.451
  19:53:02 STEP  560 H= 560:00 +CPU= 28.908
  19:53:47 STEP  561 H= 561:00 +CPU= 42.442
  19:54:16 STEP  562 H= 562:00 +CPU= 27.852
  19:54:45 STEP  563 H= 563:00 +CPU= 27.471
  19:55:35 STEP  564 H= 564:00 +CPU= 47.106
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000564
  19:56:04 STEP  565 H= 565:00 +CPU= 26.875
  19:56:33 STEP  566 H= 566:00 +CPU= 27.528
  19:57:17 STEP  567 H= 567:00 +CPU= 41.723
  19:57:47 STEP  568 H= 568:00 +CPU= 28.350
  19:58:16 STEP  569 H= 569:00 +CPU= 27.112
  19:59:02 STEP  570 H= 570:00 +CPU= 43.611
  19:59:32 STEP  571 H= 571:00 +CPU= 27.726
  20:00:02 STEP  572 H= 572:00 +CPU= 28.393
  20:00:46 STEP  573 H= 573:00 +CPU= 42.394
  20:01:16 STEP  574 H= 574:00 +CPU= 28.251
  20:01:46 STEP  575 H= 575:00 +CPU= 27.957
  20:02:39 STEP  576 H= 576:00 +CPU= 50.261
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000564
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_23.zip
Uploading the intermediate file: upload_file_23.zip
  20:03:10 STEP  577 H= 577:00 +CPU= 28.247
  20:03:39 STEP  578 H= 578:00 +CPU= 27.897
  20:04:24 STEP  579 H= 579:00 +CPU= 42.463
  20:04:53 STEP  580 H= 580:00 +CPU= 27.617
  20:05:22 STEP  581 H= 581:00 +CPU= 27.129
  20:06:08 STEP  582 H= 582:00 +CPU= 43.571
  20:06:36 STEP  583 H= 583:00 +CPU= 26.838
  20:07:06 STEP  584 H= 584:00 +CPU= 28.345
  20:07:51 STEP  585 H= 585:00 +CPU= 42.230
  20:08:20 STEP  586 H= 586:00 +CPU= 27.813
  20:08:48 STEP  587 H= 587:00 +CPU= 26.731
  20:09:38 STEP  588 H= 588:00 +CPU= 47.239
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000588
  20:10:07 STEP  589 H= 589:00 +CPU= 27.229
  20:10:36 STEP  590 H= 590:00 +CPU= 27.894
  20:11:22 STEP  591 H= 591:00 +CPU= 42.723
  20:11:50 STEP  592 H= 592:00 +CPU= 27.334
  20:12:19 STEP  593 H= 593:00 +CPU= 27.051
  20:13:05 STEP  594 H= 594:00 +CPU= 43.199
  20:13:33 STEP  595 H= 595:00 +CPU= 26.935
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:15:45 STEP  596 H= 596:00 +CPU= 26.913
  20:16:23 STEP  597 H= 597:00 +CPU= 36.898
  20:16:47 STEP  598 H= 598:00 +CPU= 23.670
  20:17:12 STEP  599 H= 599:00 +CPU= 24.643
  20:17:58 STEP  600 H= 600:00 +CPU= 45.481
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_24.zip
Uploading the intermediate file: upload_file_24.zip
  20:18:24 STEP  601 H= 601:00 +CPU= 24.873
  20:18:49 STEP  602 H= 602:00 +CPU= 24.785
  20:19:27 STEP  603 H= 603:00 +CPU= 37.355
  20:19:52 STEP  604 H= 604:00 +CPU= 25.071
  20:20:17 STEP  605 H= 605:00 +CPU= 24.140
  20:20:55 STEP  606 H= 606:00 +CPU= 37.809
  20:21:19 STEP  607 H= 607:00 +CPU= 23.654
  20:21:44 STEP  608 H= 608:00 +CPU= 24.177
  20:22:20 STEP  609 H= 609:00 +CPU= 35.813
  20:22:45 STEP  610 H= 610:00 +CPU= 24.442
  20:23:09 STEP  611 H= 611:00 +CPU= 24.058
  20:23:52 STEP  612 H= 612:00 +CPU= 41.583
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000612
  20:24:16 STEP  613 H= 613:00 +CPU= 23.991
  20:24:41 STEP  614 H= 614:00 +CPU= 24.495
  20:25:19 STEP  615 H= 615:00 +CPU= 37.603
  20:25:45 STEP  616 H= 616:00 +CPU= 25.226
  20:26:08 STEP  617 H= 617:00 +CPU= 23.255
  20:26:48 STEP  618 H= 618:00 +CPU= 38.654
  20:27:13 STEP  619 H= 619:00 +CPU= 24.476
  20:27:37 STEP  620 H= 620:00 +CPU= 24.044
  20:28:15 STEP  621 H= 621:00 +CPU= 37.562
  20:28:40 STEP  622 H= 622:00 +CPU= 24.100
  20:29:05 STEP  623 H= 623:00 +CPU= 24.661
  20:29:48 STEP  624 H= 624:00 +CPU= 42.563
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000612
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_25.zip
Uploading the intermediate file: upload_file_25.zip
  20:30:12 STEP  625 H= 625:00 +CPU= 23.312
  20:30:36 STEP  626 H= 626:00 +CPU= 23.967
  20:31:16 STEP  627 H= 627:00 +CPU= 38.887
  20:31:40 STEP  628 H= 628:00 +CPU= 24.202
  20:32:04 STEP  629 H= 629:00 +CPU= 23.285
  20:32:42 STEP  630 H= 630:00 +CPU= 37.601
  20:33:07 STEP  631 H= 631:00 +CPU= 24.491
  20:33:35 STEP  632 H= 632:00 +CPU= 26.919
  20:34:17 STEP  633 H= 633:00 +CPU= 41.136
  20:34:42 STEP  634 H= 634:00 +CPU= 24.287
  20:35:06 STEP  635 H= 635:00 +CPU= 24.039
  20:35:48 STEP  636 H= 636:00 +CPU= 41.208
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000636
  20:36:12 STEP  637 H= 637:00 +CPU= 23.214
  20:36:36 STEP  638 H= 638:00 +CPU= 24.082
  20:37:13 STEP  639 H= 639:00 +CPU= 36.427
  20:37:38 STEP  640 H= 640:00 +CPU= 24.252
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:53:49 STEP  641 H= 641:00 +CPU= 22.472
  20:54:23 STEP  642 H= 642:00 +CPU= 32.981
  20:54:47 STEP  643 H= 643:00 +CPU= 23.435
  20:55:11 STEP  644 H= 644:00 +CPU= 24.355
  20:55:51 STEP  645 H= 645:00 +CPU= 38.366
  20:56:16 STEP  646 H= 646:00 +CPU= 24.437
  20:56:41 STEP  647 H= 647:00 +CPU= 24.826
  20:57:27 STEP  648 H= 648:00 +CPU= 44.267
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_26.zip
Uploading the intermediate file: upload_file_26.zip
  20:57:51 STEP  649 H= 649:00 +CPU= 23.918
  20:58:15 STEP  650 H= 650:00 +CPU= 23.844
  20:58:53 STEP  651 H= 651:00 +CPU= 37.432
  20:59:18 STEP  652 H= 652:00 +CPU= 24.716
  20:59:43 STEP  653 H= 653:00 +CPU= 24.537
  21:00:23 STEP  654 H= 654:00 +CPU= 38.510
  21:00:46 STEP  655 H= 655:00 +CPU= 23.228
  21:01:12 STEP  656 H= 656:00 +CPU= 25.432
  21:01:49 STEP  657 H= 657:00 +CPU= 36.171
  21:02:14 STEP  658 H= 658:00 +CPU= 24.840
  21:02:38 STEP  659 H= 659:00 +CPU= 23.819
  21:03:20 STEP  660 H= 660:00 +CPU= 41.695
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000660
  21:03:45 STEP  661 H= 661:00 +CPU= 24.086
  21:04:10 STEP  662 H= 662:00 +CPU= 24.852
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:07:46 STEP  663 H= 663:00 +CPU= 38.850
  21:08:15 STEP  664 H= 664:00 +CPU= 27.363
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:09:47 STEP  665 H= 665:00 +CPU= 27.165
  21:10:32 STEP  666 H= 666:00 +CPU= 42.692
  21:11:00 STEP  667 H= 667:00 +CPU= 27.013
  21:11:29 STEP  668 H= 668:00 +CPU= 27.292
  21:12:16 STEP  669 H= 669:00 +CPU= 43.345
  21:12:46 STEP  670 H= 670:00 +CPU= 28.141
  21:13:17 STEP  671 H= 671:00 +CPU= 29.029
  21:14:11 STEP  672 H= 672:00 +CPU= 51.048
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_27.zip
Uploading the intermediate file: upload_file_27.zip
  21:14:40 STEP  673 H= 673:00 +CPU= 27.355
  21:15:10 STEP  674 H= 674:00 +CPU= 27.981
  21:15:55 STEP  675 H= 675:00 +CPU= 42.924
  21:16:25 STEP  676 H= 676:00 +CPU= 27.627
  21:16:53 STEP  677 H= 677:00 +CPU= 27.155
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:19:19 STEP  678 H= 678:00 +CPU= 42.521
  21:19:48 STEP  679 H= 679:00 +CPU= 26.805
  21:20:17 STEP  680 H= 680:00 +CPU= 27.761
  21:21:02 STEP  681 H= 681:00 +CPU= 42.275
  21:21:31 STEP  682 H= 682:00 +CPU= 28.217
  21:22:00 STEP  683 H= 683:00 +CPU= 27.286
  21:22:50 STEP  684 H= 684:00 +CPU= 47.050
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000684
  21:23:19 STEP  685 H= 685:00 +CPU= 27.493
  21:23:48 STEP  686 H= 686:00 +CPU= 27.738
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:25:45 STEP  687 H= 687:00 +CPU= 42.516
  21:26:15 STEP  688 H= 688:00 +CPU= 28.049
  21:26:44 STEP  689 H= 689:00 +CPU= 27.291
  21:27:29 STEP  690 H= 690:00 +CPU= 43.331
  21:27:58 STEP  691 H= 691:00 +CPU= 27.416
  21:28:28 STEP  692 H= 692:00 +CPU= 28.124
  21:29:14 STEP  693 H= 693:00 +CPU= 42.873
  21:29:43 STEP  694 H= 694:00 +CPU= 27.838
  21:30:13 STEP  695 H= 695:00 +CPU= 28.373
  21:31:07 STEP  696 H= 696:00 +CPU= 50.590
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_28.zip
Uploading the intermediate file: upload_file_28.zip
  21:31:35 STEP  697 H= 697:00 +CPU= 26.935
  21:32:05 STEP  698 H= 698:00 +CPU= 27.868
  21:32:50 STEP  699 H= 699:00 +CPU= 42.768
  21:33:20 STEP  700 H= 700:00 +CPU= 27.981
  21:33:48 STEP  701 H= 701:00 +CPU= 27.145
  21:34:34 STEP  702 H= 702:00 +CPU= 43.271
  21:35:02 STEP  703 H= 703:00 +CPU= 26.777
  21:35:32 STEP  704 H= 704:00 +CPU= 28.307
  21:36:17 STEP  705 H= 705:00 +CPU= 42.699
  21:36:47 STEP  706 H= 706:00 +CPU= 28.249
  21:37:16 STEP  707 H= 707:00 +CPU= 27.312
  21:38:06 STEP  708 H= 708:00 +CPU= 47.338
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000708
  21:38:35 STEP  709 H= 709:00 +CPU= 27.204
  21:39:04 STEP  710 H= 710:00 +CPU= 27.860
  21:39:49 STEP  711 H= 711:00 +CPU= 42.207
  21:40:18 STEP  712 H= 712:00 +CPU= 27.995
  21:40:48 STEP  713 H= 713:00 +CPU= 27.968
  21:41:34 STEP  714 H= 714:00 +CPU= 43.311
  21:42:02 STEP  715 H= 715:00 +CPU= 27.138
  21:42:32 STEP  716 H= 716:00 +CPU= 27.930
  21:43:16 STEP  717 H= 717:00 +CPU= 42.119
  21:43:46 STEP  718 H= 718:00 +CPU= 28.006
  21:44:16 STEP  719 H= 719:00 +CPU= 28.141
  21:45:10 STEP  720 H= 720:00 +CPU= 50.704
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_29.zip
Uploading the intermediate file: upload_file_29.zip
Uploading trickle at timestep: 2592000
  21:45:39 STEP  721 H= 721:00 +CPU= 27.539
  21:46:08 STEP  722 H= 722:00 +CPU= 27.806
  21:46:53 STEP  723 H= 723:00 +CPU= 42.215
  21:47:22 STEP  724 H= 724:00 +CPU= 27.668
  21:47:51 STEP  725 H= 725:00 +CPU= 27.492
  21:48:37 STEP  726 H= 726:00 +CPU= 43.767
  21:49:05 STEP  727 H= 727:00 +CPU= 26.683
  21:49:34 STEP  728 H= 728:00 +CPU= 27.740
  21:50:20 STEP  729 H= 729:00 +CPU= 43.173
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:52:33 STEP  730 H= 730:00 +CPU= 27.811
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:56:55 STEP  731 H= 731:00 +CPU= 25.762
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:00:02 STEP  732 H= 732:00 +CPU= 46.544
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000732
  22:00:30 STEP  733 H= 733:00 +CPU= 26.350
  22:01:00 STEP  734 H= 734:00 +CPU= 27.982
  22:01:43 STEP  735 H= 735:00 +CPU= 41.656
  22:02:13 STEP  736 H= 736:00 +CPU= 27.830
  22:02:41 STEP  737 H= 737:00 +CPU= 26.939
  22:03:27 STEP  738 H= 738:00 +CPU= 43.005
  22:03:55 STEP  739 H= 739:00 +CPU= 26.741
  22:04:25 STEP  740 H= 740:00 +CPU= 28.153
  22:05:10 STEP  741 H= 741:00 +CPU= 42.536
  22:05:39 STEP  742 H= 742:00 +CPU= 27.724
  22:06:09 STEP  743 H= 743:00 +CPU= 28.051
  22:07:03 STEP  744 H= 744:00 +CPU= 50.606
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000732
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_30.zip
Uploading the intermediate file: upload_file_30.zip
  22:07:31 STEP  745 H= 745:00 +CPU= 26.712
  22:08:00 STEP  746 H= 746:00 +CPU= 27.576
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:10:39 STEP  747 H= 747:00 +CPU= 42.244
  22:11:09 STEP  748 H= 748:00 +CPU= 28.387
  22:11:37 STEP  749 H= 749:00 +CPU= 26.744
  22:12:23 STEP  750 H= 750:00 +CPU= 43.052
  22:12:51 STEP  751 H= 751:00 +CPU= 26.831
  22:13:21 STEP  752 H= 752:00 +CPU= 28.039
  22:14:06 STEP  753 H= 753:00 +CPU= 42.858
  22:14:37 STEP  754 H= 754:00 +CPU= 28.348
  22:15:06 STEP  755 H= 755:00 +CPU= 27.768
  22:15:56 STEP  756 H= 756:00 +CPU= 47.159
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000756
  22:16:27 STEP  757 H= 757:00 +CPU= 27.860
  22:16:56 STEP  758 H= 758:00 +CPU= 28.225
  22:17:41 STEP  759 H= 759:00 +CPU= 42.308
  22:18:11 STEP  760 H= 760:00 +CPU= 28.171
  22:18:40 STEP  761 H= 761:00 +CPU= 27.176
  22:19:25 STEP  762 H= 762:00 +CPU= 42.927
  22:19:53 STEP  763 H= 763:00 +CPU= 26.680
  22:20:22 STEP  764 H= 764:00 +CPU= 27.934
  22:21:08 STEP  765 H= 765:00 +CPU= 42.921
  22:21:38 STEP  766 H= 766:00 +CPU= 28.246
  22:22:08 STEP  767 H= 767:00 +CPU= 28.012
  22:23:01 STEP  768 H= 768:00 +CPU= 50.129
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_31.zip
Uploading the intermediate file: upload_file_31.zip
  22:23:29 STEP  769 H= 769:00 +CPU= 26.810
  22:23:59 STEP  770 H= 770:00 +CPU= 27.757
  22:24:44 STEP  771 H= 771:00 +CPU= 42.512
  22:25:13 STEP  772 H= 772:00 +CPU= 27.697
  22:25:42 STEP  773 H= 773:00 +CPU= 27.108
  22:26:27 STEP  774 H= 774:00 +CPU= 42.919
  22:26:55 STEP  775 H= 775:00 +CPU= 27.007
  22:27:24 STEP  776 H= 776:00 +CPU= 27.561
  22:28:09 STEP  777 H= 777:00 +CPU= 42.578
  22:28:39 STEP  778 H= 778:00 +CPU= 27.875
  22:29:07 STEP  779 H= 779:00 +CPU= 27.166
  22:29:57 STEP  780 H= 780:00 +CPU= 46.978
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000780
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:31:28 STEP  781 H= 781:00 +CPU= 26.757
  22:31:57 STEP  782 H= 782:00 +CPU= 27.298
  22:32:41 STEP  783 H= 783:00 +CPU= 41.716
  22:33:10 STEP  784 H= 784:00 +CPU= 27.953
  22:33:39 STEP  785 H= 785:00 +CPU= 26.900
  22:34:24 STEP  786 H= 786:00 +CPU= 43.208
  22:34:53 STEP  787 H= 787:00 +CPU= 27.285
  22:35:22 STEP  788 H= 788:00 +CPU= 27.579
  22:36:08 STEP  789 H= 789:00 +CPU= 42.832
  22:36:37 STEP  790 H= 790:00 +CPU= 27.836
  22:37:06 STEP  791 H= 791:00 +CPU= 27.770
  22:38:00 STEP  792 H= 792:00 +CPU= 50.633
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_32.zip
Uploading the intermediate file: upload_file_32.zip
  22:38:28 STEP  793 H= 793:00 +CPU= 26.573
  22:38:58 STEP  794 H= 794:00 +CPU= 28.009
  22:39:42 STEP  795 H= 795:00 +CPU= 41.958
  22:40:12 STEP  796 H= 796:00 +CPU= 28.048
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:42:31 STEP  797 H= 797:00 +CPU= 26.230
  22:43:17 STEP  798 H= 798:00 +CPU= 43.343
  22:43:45 STEP  799 H= 799:00 +CPU= 26.783
  22:44:15 STEP  800 H= 800:00 +CPU= 27.895
  22:44:59 STEP  801 H= 801:00 +CPU= 42.488
  22:45:29 STEP  802 H= 802:00 +CPU= 28.080
  22:45:58 STEP  803 H= 803:00 +CPU= 27.329
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:47:51 STEP  804 H= 804:00 +CPU= 46.997
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000804
  22:48:20 STEP  805 H= 805:00 +CPU= 26.949
  22:48:50 STEP  806 H= 806:00 +CPU= 27.808
  22:49:34 STEP  807 H= 807:00 +CPU= 42.333
  22:50:04 STEP  808 H= 808:00 +CPU= 27.976
  22:50:33 STEP  809 H= 809:00 +CPU= 27.326
  22:51:20 STEP  810 H= 810:00 +CPU= 43.902
  22:51:48 STEP  811 H= 811:00 +CPU= 26.747
  22:52:17 STEP  812 H= 812:00 +CPU= 27.973
  22:53:02 STEP  813 H= 813:00 +CPU= 42.079
  22:53:32 STEP  814 H= 814:00 +CPU= 28.044
  22:54:01 STEP  815 H= 815:00 +CPU= 27.824
  22:54:55 STEP  816 H= 816:00 +CPU= 50.961
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000804
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_33.zip
Uploading the intermediate file: upload_file_33.zip
  22:55:24 STEP  817 H= 817:00 +CPU= 27.350
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:57:19 STEP  818 H= 818:00 +CPU= 27.343
  22:57:57 STEP  819 H= 819:00 +CPU= 36.949
  22:58:22 STEP  820 H= 820:00 +CPU= 25.196
  22:58:47 STEP  821 H= 821:00 +CPU= 24.371
  22:59:25 STEP  822 H= 822:00 +CPU= 37.218
  22:59:50 STEP  823 H= 823:00 +CPU= 24.436
  23:00:14 STEP  824 H= 824:00 +CPU= 24.059
  23:00:52 STEP  825 H= 825:00 +CPU= 36.929
  23:01:16 STEP  826 H= 826:00 +CPU= 23.950
  23:01:40 STEP  827 H= 827:00 +CPU= 23.620
  23:02:21 STEP  828 H= 828:00 +CPU= 40.798
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000828
  23:02:46 STEP  829 H= 829:00 +CPU= 23.852
  23:03:10 STEP  830 H= 830:00 +CPU= 24.481
  23:03:48 STEP  831 H= 831:00 +CPU= 36.990
  23:04:14 STEP  832 H= 832:00 +CPU= 24.905
  23:04:37 STEP  833 H= 833:00 +CPU= 23.301
  23:05:16 STEP  834 H= 834:00 +CPU= 38.169
  23:05:40 STEP  835 H= 835:00 +CPU= 23.695
  23:06:05 STEP  836 H= 836:00 +CPU= 24.039
  23:06:42 STEP  837 H= 837:00 +CPU= 36.793
  23:07:06 STEP  838 H= 838:00 +CPU= 23.692
  23:07:30 STEP  839 H= 839:00 +CPU= 23.520
  23:08:17 STEP  840 H= 840:00 +CPU= 45.579
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_34.zip
Uploading the intermediate file: upload_file_34.zip
  23:08:41 STEP  841 H= 841:00 +CPU= 23.683
  23:09:07 STEP  842 H= 842:00 +CPU= 25.424
  23:09:44 STEP  843 H= 843:00 +CPU= 36.299
  23:10:09 STEP  844 H= 844:00 +CPU= 24.783
  23:10:33 STEP  845 H= 845:00 +CPU= 23.390
  23:11:12 STEP  846 H= 846:00 +CPU= 38.393
  23:11:36 STEP  847 H= 847:00 +CPU= 23.805
  23:12:01 STEP  848 H= 848:00 +CPU= 24.579
  23:12:39 STEP  849 H= 849:00 +CPU= 37.225
  23:13:04 STEP  850 H= 850:00 +CPU= 24.423
  23:13:28 STEP  851 H= 851:00 +CPU= 24.015
  23:14:11 STEP  852 H= 852:00 +CPU= 42.285
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000852
  23:14:35 STEP  853 H= 853:00 +CPU= 24.143
  23:15:00 STEP  854 H= 854:00 +CPU= 24.306
  23:15:38 STEP  855 H= 855:00 +CPU= 37.429
  23:16:04 STEP  856 H= 856:00 +CPU= 25.416
  23:16:28 STEP  857 H= 857:00 +CPU= 23.654
  23:17:08 STEP  858 H= 858:00 +CPU= 39.045
  23:17:32 STEP  859 H= 859:00 +CPU= 24.260
  23:17:57 STEP  860 H= 860:00 +CPU= 24.577
  23:18:35 STEP  861 H= 861:00 +CPU= 37.414
  23:19:00 STEP  862 H= 862:00 +CPU= 24.273
  23:19:24 STEP  863 H= 863:00 +CPU= 23.623
  23:20:10 STEP  864 H= 864:00 +CPU= 44.433
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_35.zip
Uploading the intermediate file: upload_file_35.zip
  23:20:33 STEP  865 H= 865:00 +CPU= 23.597
  23:20:58 STEP  866 H= 866:00 +CPU= 24.039
  23:21:35 STEP  867 H= 867:00 +CPU= 36.757
  23:22:00 STEP  868 H= 868:00 +CPU= 24.552
  23:22:25 STEP  869 H= 869:00 +CPU= 24.005
  23:23:03 STEP  870 H= 870:00 +CPU= 37.457
  23:23:27 STEP  871 H= 871:00 +CPU= 23.751
  23:23:51 STEP  872 H= 872:00 +CPU= 23.434
  23:24:29 STEP  873 H= 873:00 +CPU= 37.973
  23:24:55 STEP  874 H= 874:00 +CPU= 24.833
  23:25:18 STEP  875 H= 875:00 +CPU= 23.218
  23:26:00 STEP  876 H= 876:00 +CPU= 40.988
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000876
  23:26:24 STEP  877 H= 877:00 +CPU= 23.337
  23:26:48 STEP  878 H= 878:00 +CPU= 23.763
  23:27:26 STEP  879 H= 879:00 +CPU= 37.000
  23:27:50 STEP  880 H= 880:00 +CPU= 23.653
  23:28:14 STEP  881 H= 881:00 +CPU= 24.334
  23:28:53 STEP  882 H= 882:00 +CPU= 37.910
  23:29:17 STEP  883 H= 883:00 +CPU= 23.873
  23:29:42 STEP  884 H= 884:00 +CPU= 24.496
  23:30:20 STEP  885 H= 885:00 +CPU= 37.550
  23:30:45 STEP  886 H= 886:00 +CPU= 24.583
  23:31:10 STEP  887 H= 887:00 +CPU= 24.085
  23:31:56 STEP  888 H= 888:00 +CPU= 45.579
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_36.zip
Uploading the intermediate file: upload_file_36.zip
  23:32:21 STEP  889 H= 889:00 +CPU= 24.074
  23:32:46 STEP  890 H= 890:00 +CPU= 24.744
  23:33:22 STEP  891 H= 891:00 +CPU= 35.595
  23:33:47 STEP  892 H= 892:00 +CPU= 25.011
  23:34:11 STEP  893 H= 893:00 +CPU= 22.983
  23:34:51 STEP  894 H= 894:00 +CPU= 39.034
  23:35:15 STEP  895 H= 895:00 +CPU= 23.858
  23:35:40 STEP  896 H= 896:00 +CPU= 24.708
  23:36:18 STEP  897 H= 897:00 +CPU= 37.174
  23:36:44 STEP  898 H= 898:00 +CPU= 25.448
  23:37:07 STEP  899 H= 899:00 +CPU= 22.764
  23:37:49 STEP  900 H= 900:00 +CPU= 41.773
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000900
  23:38:13 STEP  901 H= 901:00 +CPU= 23.249
  23:38:38 STEP  902 H= 902:00 +CPU= 24.622
  23:39:16 STEP  903 H= 903:00 +CPU= 37.474
  23:39:41 STEP  904 H= 904:00 +CPU= 24.497
  23:40:05 STEP  905 H= 905:00 +CPU= 23.744
  23:40:45 STEP  906 H= 906:00 +CPU= 38.914
  23:41:08 STEP  907 H= 907:00 +CPU= 23.158
  23:41:34 STEP  908 H= 908:00 +CPU= 25.323
  23:42:11 STEP  909 H= 909:00 +CPU= 36.242
  23:42:36 STEP  910 H= 910:00 +CPU= 24.261
  23:42:59 STEP  911 H= 911:00 +CPU= 23.620
  23:43:44 STEP  912 H= 912:00 +CPU= 43.915
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000900
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_37.zip
Uploading the intermediate file: upload_file_37.zip
  23:44:08 STEP  913 H= 913:00 +CPU= 23.683
  23:44:32 STEP  914 H= 914:00 +CPU= 23.363
  23:45:10 STEP  915 H= 915:00 +CPU= 37.105
  23:45:35 STEP  916 H= 916:00 +CPU= 24.538
  23:46:00 STEP  917 H= 917:00 +CPU= 24.556
  23:46:39 STEP  918 H= 918:00 +CPU= 38.455
  23:47:03 STEP  919 H= 919:00 +CPU= 24.185
  23:47:27 STEP  920 H= 920:00 +CPU= 23.405
  23:48:04 STEP  921 H= 921:00 +CPU= 36.031
  23:48:28 STEP  922 H= 922:00 +CPU= 23.809
  23:48:53 STEP  923 H= 923:00 +CPU= 24.695
  23:49:36 STEP  924 H= 924:00 +CPU= 42.160
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000924
  23:50:00 STEP  925 H= 925:00 +CPU= 23.948
  23:50:26 STEP  926 H= 926:00 +CPU= 25.435
  23:51:04 STEP  927 H= 927:00 +CPU= 37.334
  23:51:28 STEP  928 H= 928:00 +CPU= 24.057
  23:51:53 STEP  929 H= 929:00 +CPU= 24.389
  23:52:31 STEP  930 H= 930:00 +CPU= 37.303
  23:52:55 STEP  931 H= 931:00 +CPU= 23.398
  23:53:19 STEP  932 H= 932:00 +CPU= 24.251
  23:53:58 STEP  933 H= 933:00 +CPU= 38.288
  23:54:23 STEP  934 H= 934:00 +CPU= 24.531
  23:54:48 STEP  935 H= 935:00 +CPU= 24.551
  23:55:33 STEP  936 H= 936:00 +CPU= 43.674
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000924
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_38.zip
Uploading the intermediate file: upload_file_38.zip
  23:55:57 STEP  937 H= 937:00 +CPU= 23.711
  23:56:22 STEP  938 H= 938:00 +CPU= 24.645
  23:57:01 STEP  939 H= 939:00 +CPU= 37.462
  23:57:25 STEP  940 H= 940:00 +CPU= 24.654
  23:57:49 STEP  941 H= 941:00 +CPU= 23.109
  23:58:27 STEP  942 H= 942:00 +CPU= 37.909
  23:58:51 STEP  943 H= 943:00 +CPU= 23.373
  23:59:16 STEP  944 H= 944:00 +CPU= 24.808
  23:59:54 STEP  945 H= 945:00 +CPU= 37.226
  00:00:19 STEP  946 H= 946:00 +CPU= 24.101
  00:00:43 STEP  947 H= 947:00 +CPU= 23.499
  00:01:24 STEP  948 H= 948:00 +CPU= 40.760
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000948
  00:01:48 STEP  949 H= 949:00 +CPU= 23.740
  00:02:14 STEP  950 H= 950:00 +CPU= 25.495
  00:02:52 STEP  951 H= 951:00 +CPU= 37.244
  00:03:18 STEP  952 H= 952:00 +CPU= 25.520
  00:03:42 STEP  953 H= 953:00 +CPU= 23.885
  00:04:21 STEP  954 H= 954:00 +CPU= 38.144
  00:04:45 STEP  955 H= 955:00 +CPU= 23.987
  00:05:09 STEP  956 H= 956:00 +CPU= 23.835
  00:05:46 STEP  957 H= 957:00 +CPU= 36.250
  00:06:11 STEP  958 H= 958:00 +CPU= 24.216
  00:06:36 STEP  959 H= 959:00 +CPU= 25.039
  00:07:22 STEP  960 H= 960:00 +CPU= 44.978
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_39.zip
Uploading the intermediate file: upload_file_39.zip
Uploading trickle at timestep: 3456000
  00:07:47 STEP  961 H= 961:00 +CPU= 24.134
  00:08:12 STEP  962 H= 962:00 +CPU= 24.888
  00:08:50 STEP  963 H= 963:00 +CPU= 37.643
  00:09:16 STEP  964 H= 964:00 +CPU= 24.702
  00:09:40 STEP  965 H= 965:00 +CPU= 24.122
  00:10:18 STEP  966 H= 966:00 +CPU= 36.998
  00:10:42 STEP  967 H= 967:00 +CPU= 24.121
  00:11:07 STEP  968 H= 968:00 +CPU= 25.015
  00:11:44 STEP  969 H= 969:00 +CPU= 36.200
  00:12:08 STEP  970 H= 970:00 +CPU= 23.392
  00:12:32 STEP  971 H= 971:00 +CPU= 23.504
  00:13:13 STEP  972 H= 972:00 +CPU= 40.255
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000972
  00:13:37 STEP  973 H= 973:00 +CPU= 23.861
  00:14:02 STEP  974 H= 974:00 +CPU= 24.472
  00:14:40 STEP  975 H= 975:00 +CPU= 37.215
  00:15:05 STEP  976 H= 976:00 +CPU= 24.667
  00:15:29 STEP  977 H= 977:00 +CPU= 24.000
  00:16:07 STEP  978 H= 978:00 +CPU= 37.018
  00:16:30 STEP  979 H= 979:00 +CPU= 23.235
  00:16:54 STEP  980 H= 980:00 +CPU= 23.660
  00:17:32 STEP  981 H= 981:00 +CPU= 37.282
  00:17:57 STEP  982 H= 982:00 +CPU= 24.588
  00:18:22 STEP  983 H= 983:00 +CPU= 24.210
  00:19:07 STEP  984 H= 984:00 +CPU= 44.105
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311_40.zip
Uploading the intermediate file: upload_file_40.zip
  00:19:31 STEP  985 H= 985:00 +CPU= 23.652
  00:19:56 STEP  986 H= 986:00 +CPU= 24.528
  00:20:34 STEP  987 H= 987:00 +CPU= 37.334
  00:20:59 STEP  988 H= 988:00 +CPU= 24.387
  00:21:23 STEP  989 H= 989:00 +CPU= 23.701
  00:22:02 STEP  990 H= 990:00 +CPU= 39.216
  00:22:27 STEP  991 H= 991:00 +CPU= 23.862
  00:22:52 STEP  992 H= 992:00 +CPU= 24.564
  00:23:29 STEP  993 H= 993:00 +CPU= 36.656
  00:23:53 STEP  994 H= 994:00 +CPU= 23.626
  00:24:18 STEP  995 H= 995:00 +CPU= 24.243
  00:24:59 STEP  996 H= 996:00 +CPU= 40.706
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.277] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4745MB (heap), 4399MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.278] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.278] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.278] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.278] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.278] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4745 MB (maxheap), 4399 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     1        (1004) 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               LAPINEA 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                LARMES 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 LARCINA 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002516:1671665116:33809.280] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  LAITLI 
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002517:1671665117:33810.284] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1011590:1011590] [20221222:002517:1671665117:33810.284] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000C1A916  laitli_                   151  laitli.F90
oifs_43r3_model.e  0000000000DEE544  larcina_                  405  larcina.F90
oifs_43r3_model.e  0000000000DEABF8  larmes_                   603  larmes.F90
oifs_43r3_model.e  0000000000DE50FD  lapinea_                  270  lapinea.F90
oifs_43r3_model.e  0000000000D7A4EC  call_sl_                  300  call_sl.F90
oifs_43r3_model.e  0000000000D784FD  call_sl_heap_              94  call_sl_heap.F90
oifs_43r3_model.e  0000000000BF9958  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1988050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0094
(argv4) batchid: 957
(argv5) wuid: 12173738
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173738
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_49bea26f0cc4a18f20fcf7b1a4df4f6d to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0094_1988050100_123_957_12173738.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0094_1988050100_123_957_12173738.zip
ic_ancil_file: ic_ancil_12173738
ifsdata_file: ifsdata_12173738
climate_data_file: clim_data_12173738
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_00aabc955a58c54ef25582ec957e0c6d to: /var/lib/boinc-client/slots/2/ic_ancil_12173738.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12173738.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_db79711eaa5a5031b0694ea8b0cdcd19 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12173738.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12173738.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12173738.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12173738.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12173738.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12173738.xml
-- Model is restarting --
Adjusting last_iter, 996, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12173738.xml
last_cpu_time: 29815
upload_file_number: 41
last_iter: 983
last_upload: 3542400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0094_1988050100_123_957_12173738_0_r1895949311
The child process has been launched with process id: 1945160
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1945160:1945160] [20221225:181117:1671988277:0.008] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:46:45 1534737 22258318 oifs_43r3_ps_0094_1988050100_123_957_12173738_0 3,456,000 28,777 0.0083
09 May 2023 15:46:32 1534737 22258318 oifs_43r3_ps_0094_1988050100_123_957_12173738_0 2,592,000 21,414 0.0083
09 May 2023 15:46:19 1534737 22258318 oifs_43r3_ps_0094_1988050100_123_957_12173738_0 1,728,000 13,734 0.0079
09 May 2023 15:46:09 1534737 22258318 oifs_43r3_ps_0094_1988050100_123_957_12173738_0 864,000 6,861 0.0079


©2024 climateprediction.net