climateprediction.net home page
Task 22283622

Task 22283622

Name oifs_43r3_ps_0921_2011050100_123_980_12197565_0
Workunit 12197565
Created 24 Dec 2022, 3:37:01 UTC
Sent 5 Jan 2023, 4:18:03 UTC
Report deadline 4 Feb 2023, 4:18:03 UTC
Received 6 Jan 2023, 11:55:53 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1443505
Run time 23 hours 59 min 49 sec
CPU time 23 hours 52 min 47 sec
Validate state Invalid
Credit 5,080.32
Device peak FLOPS 2.12 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,478.28 MB
Peak swap size 4,978.07 MB
Peak disk usage 2,610.05 MB
Stderr
<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
+CPU=109.443
  11:58:21 STEP  514 H= 514:00 +CPU= 70.478
  11:59:29 STEP  515 H= 515:00 +CPU= 67.050
  12:01:31 STEP  516 H= 516:00 +CPU=122.260
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000516
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000516
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000516
  12:02:39 STEP  517 H= 517:00 +CPU= 67.440
  12:03:50 STEP  518 H= 518:00 +CPU= 69.882
  12:05:40 STEP  519 H= 519:00 +CPU=109.377
  12:06:50 STEP  520 H= 520:00 +CPU= 69.727
  12:07:57 STEP  521 H= 521:00 +CPU= 66.874
  12:09:50 STEP  522 H= 522:00 +CPU=112.502
  12:10:57 STEP  523 H= 523:00 +CPU= 66.677
  12:12:07 STEP  524 H= 524:00 +CPU= 70.016
  12:13:58 STEP  525 H= 525:00 +CPU=110.049
  12:15:08 STEP  526 H= 526:00 +CPU= 69.876
  12:16:18 STEP  527 H= 527:00 +CPU= 69.534
  12:18:27 STEP  528 H= 528:00 +CPU=126.526
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000528
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000528
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000528
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000504
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000504
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000504
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000516
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000516
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000516
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_21.zip
Uploading the intermediate file: upload_file_21.zip
  12:19:34 STEP  529 H= 529:00 +CPU= 66.798
  12:20:44 STEP  530 H= 530:00 +CPU= 69.539
  12:22:35 STEP  531 H= 531:00 +CPU=110.262
  12:23:45 STEP  532 H= 532:00 +CPU= 69.700
  12:24:52 STEP  533 H= 533:00 +CPU= 67.206
  12:26:45 STEP  534 H= 534:00 +CPU=112.519
  12:27:53 STEP  535 H= 535:00 +CPU= 67.228
  12:29:03 STEP  536 H= 536:00 +CPU= 69.906
  12:30:52 STEP  537 H= 537:00 +CPU=108.820
  12:32:02 STEP  538 H= 538:00 +CPU= 69.818
  12:33:09 STEP  539 H= 539:00 +CPU= 67.142
  12:35:11 STEP  540 H= 540:00 +CPU=121.814
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000540
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000540
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000540
  12:36:19 STEP  541 H= 541:00 +CPU= 67.250
  12:37:29 STEP  542 H= 542:00 +CPU= 69.708
  12:39:20 STEP  543 H= 543:00 +CPU=110.008
  12:40:30 STEP  544 H= 544:00 +CPU= 69.740
  12:41:37 STEP  545 H= 545:00 +CPU= 66.862
  12:43:29 STEP  546 H= 546:00 +CPU=111.966
  12:44:36 STEP  547 H= 547:00 +CPU= 66.824
  12:45:46 STEP  548 H= 548:00 +CPU= 69.917
  12:47:36 STEP  549 H= 549:00 +CPU=109.722
  12:48:46 STEP  550 H= 550:00 +CPU= 69.340
  12:49:55 STEP  551 H= 551:00 +CPU= 68.929
  12:52:01 STEP  552 H= 552:00 +CPU=126.182
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000552
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000552
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000552
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000528
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000528
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000528
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000540
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000540
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000540
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_22.zip
Uploading the intermediate file: upload_file_22.zip
  12:53:09 STEP  553 H= 553:00 +CPU= 67.069
  12:54:18 STEP  554 H= 554:00 +CPU= 69.417
  12:56:08 STEP  555 H= 555:00 +CPU=109.874
  12:57:18 STEP  556 H= 556:00 +CPU= 69.713
  12:58:26 STEP  557 H= 557:00 +CPU= 67.091
  13:00:19 STEP  558 H= 558:00 +CPU=112.719
  13:01:26 STEP  559 H= 559:00 +CPU= 67.071
  13:02:36 STEP  560 H= 560:00 +CPU= 69.555
  13:04:26 STEP  561 H= 561:00 +CPU=109.581
  13:05:36 STEP  562 H= 562:00 +CPU= 69.763
  13:06:44 STEP  563 H= 563:00 +CPU= 67.309
  13:08:46 STEP  564 H= 564:00 +CPU=121.933
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000564
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000564
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000564
  13:09:54 STEP  565 H= 565:00 +CPU= 67.304
  13:11:03 STEP  566 H= 566:00 +CPU= 69.344
  13:12:53 STEP  567 H= 567:00 +CPU=109.439
  13:14:03 STEP  568 H= 568:00 +CPU= 69.709
  13:15:10 STEP  569 H= 569:00 +CPU= 67.031
  13:17:03 STEP  570 H= 570:00 +CPU=112.247
  13:18:10 STEP  571 H= 571:00 +CPU= 66.798
  13:19:20 STEP  572 H= 572:00 +CPU= 69.777
  13:21:09 STEP  573 H= 573:00 +CPU=109.516
  13:22:19 STEP  574 H= 574:00 +CPU= 69.274
  13:23:28 STEP  575 H= 575:00 +CPU= 69.122
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           21  MAX ETADOT VERTICAL VEL.=   3.648738550485386E-007
  LON  =    158.750000000000       degrees
  LAT  =   -36.4483927977945       degrees
  13:25:35 STEP  576 H= 576:00 +CPU=126.118
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000576
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000576
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000576
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000552
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000552
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000552
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000564
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000564
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000564
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_23.zip
Uploading the intermediate file: upload_file_23.zip
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           21  MAX ETADOT VERTICAL VEL.=   2.755919086602404E-007
  LON  =    158.750000000000       degrees
  LAT  =   -36.4483927977945       degrees
  13:26:42 STEP  577 H= 577:00 +CPU= 66.612
  13:27:51 STEP  578 H= 578:00 +CPU= 69.181
  13:29:41 STEP  579 H= 579:00 +CPU=109.289
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           22  MAX ETADOT VERTICAL VEL.=   2.645149798496979E-007
  LON  =    157.500000000000       degrees
  LAT  =   -38.6913661222016       degrees
  13:30:50 STEP  580 H= 580:00 +CPU= 69.491
  13:31:57 STEP  581 H= 581:00 +CPU= 66.662
  13:33:50 STEP  582 H= 582:00 +CPU=112.376
  13:34:57 STEP  583 H= 583:00 +CPU= 66.834
  13:36:07 STEP  584 H= 584:00 +CPU= 69.514
  13:37:56 STEP  585 H= 585:00 +CPU=109.069
  13:39:06 STEP  586 H= 586:00 +CPU= 69.780
  13:40:13 STEP  587 H= 587:00 +CPU= 66.786
  13:42:15 STEP  588 H= 588:00 +CPU=121.574
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000588
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000588
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000588
  13:43:22 STEP  589 H= 589:00 +CPU= 66.978
  13:44:32 STEP  590 H= 590:00 +CPU= 69.515
  13:46:21 STEP  591 H= 591:00 +CPU=108.974
  13:47:31 STEP  592 H= 592:00 +CPU= 69.603
  13:48:38 STEP  593 H= 593:00 +CPU= 66.899
  13:50:31 STEP  594 H= 594:00 +CPU=112.147
  13:51:38 STEP  595 H= 595:00 +CPU= 66.785
  13:52:47 STEP  596 H= 596:00 +CPU= 69.200
  13:54:37 STEP  597 H= 597:00 +CPU=109.319
  13:55:46 STEP  598 H= 598:00 +CPU= 69.415
  13:56:55 STEP  599 H= 599:00 +CPU= 68.839
  13:59:02 STEP  600 H= 600:00 +CPU=126.227
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000600
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000600
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000600
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000576
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000576
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000576
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000588
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000588
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000588
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_24.zip
Uploading the intermediate file: upload_file_24.zip
  14:00:09 STEP  601 H= 601:00 +CPU= 66.503
  14:01:18 STEP  602 H= 602:00 +CPU= 68.961
  14:03:07 STEP  603 H= 603:00 +CPU=109.086
  14:04:17 STEP  604 H= 604:00 +CPU= 69.656
  14:05:24 STEP  605 H= 605:00 +CPU= 66.776
  14:07:17 STEP  606 H= 606:00 +CPU=112.126
  14:08:24 STEP  607 H= 607:00 +CPU= 66.849
  14:09:34 STEP  608 H= 608:00 +CPU= 69.443
  14:11:23 STEP  609 H= 609:00 +CPU=109.337
  14:12:33 STEP  610 H= 610:00 +CPU= 69.441
  14:13:40 STEP  611 H= 611:00 +CPU= 66.702
  14:15:42 STEP  612 H= 612:00 +CPU=121.903
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000612
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000612
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000612
  14:16:48 STEP  613 H= 613:00 +CPU= 66.527
  14:17:58 STEP  614 H= 614:00 +CPU= 69.715
  14:19:48 STEP  615 H= 615:00 +CPU=108.937
  14:20:57 STEP  616 H= 616:00 +CPU= 69.573
  14:22:04 STEP  617 H= 617:00 +CPU= 66.666
  14:23:57 STEP  618 H= 618:00 +CPU=112.196
  14:25:04 STEP  619 H= 619:00 +CPU= 66.651
  14:26:13 STEP  620 H= 620:00 +CPU= 69.488
  14:28:03 STEP  621 H= 621:00 +CPU=109.407
  14:29:12 STEP  622 H= 622:00 +CPU= 69.165
  14:30:21 STEP  623 H= 623:00 +CPU= 69.089
  14:32:28 STEP  624 H= 624:00 +CPU=126.544
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000624
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000624
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000624
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000600
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000600
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000600
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000612
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000612
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000612
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_25.zip
Uploading the intermediate file: upload_file_25.zip
  14:33:35 STEP  625 H= 625:00 +CPU= 66.840
  14:34:45 STEP  626 H= 626:00 +CPU= 69.489
  14:36:34 STEP  627 H= 627:00 +CPU=108.874
  14:37:44 STEP  628 H= 628:00 +CPU= 69.331
  14:38:51 STEP  629 H= 629:00 +CPU= 66.837
  14:40:43 STEP  630 H= 630:00 +CPU=111.914
  14:41:50 STEP  631 H= 631:00 +CPU= 66.992
  14:43:00 STEP  632 H= 632:00 +CPU= 69.345
  14:44:49 STEP  633 H= 633:00 +CPU=109.131
  14:45:59 STEP  634 H= 634:00 +CPU= 69.549
  14:47:06 STEP  635 H= 635:00 +CPU= 66.828
  14:49:08 STEP  636 H= 636:00 +CPU=121.332
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000636
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000636
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000636
  14:50:16 STEP  637 H= 637:00 +CPU= 67.445
  14:51:26 STEP  638 H= 638:00 +CPU= 69.581
  14:53:15 STEP  639 H= 639:00 +CPU=108.501
  14:54:24 STEP  640 H= 640:00 +CPU= 69.608
  14:55:31 STEP  641 H= 641:00 +CPU= 66.849
  14:57:24 STEP  642 H= 642:00 +CPU=112.005
  14:58:32 STEP  643 H= 643:00 +CPU= 66.852
  14:59:42 STEP  644 H= 644:00 +CPU= 69.616
  15:01:31 STEP  645 H= 645:00 +CPU=108.654
  15:02:40 STEP  646 H= 646:00 +CPU= 68.983
  15:03:49 STEP  647 H= 647:00 +CPU= 68.964
  15:05:55 STEP  648 H= 648:00 +CPU=125.504
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000648
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000648
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000648
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000624
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000624
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000624
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000636
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000636
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000636
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_26.zip
Uploading the intermediate file: upload_file_26.zip
  15:07:02 STEP  649 H= 649:00 +CPU= 66.180
  15:08:11 STEP  650 H= 650:00 +CPU= 69.409
  15:10:01 STEP  651 H= 651:00 +CPU=109.309
  15:11:11 STEP  652 H= 652:00 +CPU= 69.403
  15:12:17 STEP  653 H= 653:00 +CPU= 66.477
  15:14:09 STEP  654 H= 654:00 +CPU=111.623
  15:15:16 STEP  655 H= 655:00 +CPU= 66.944
  15:16:26 STEP  656 H= 656:00 +CPU= 69.395
  15:18:16 STEP  657 H= 657:00 +CPU=109.329
  15:19:25 STEP  658 H= 658:00 +CPU= 69.398
  15:20:33 STEP  659 H= 659:00 +CPU= 66.859
  15:22:35 STEP  660 H= 660:00 +CPU=122.080
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000660
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000660
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000660
  15:23:42 STEP  661 H= 661:00 +CPU= 66.936
  15:24:52 STEP  662 H= 662:00 +CPU= 69.654
  15:26:42 STEP  663 H= 663:00 +CPU=108.965
  15:27:52 STEP  664 H= 664:00 +CPU= 69.541
  15:28:59 STEP  665 H= 665:00 +CPU= 67.059
  15:30:51 STEP  666 H= 666:00 +CPU=112.047
  15:31:58 STEP  667 H= 667:00 +CPU= 66.762
  15:33:08 STEP  668 H= 668:00 +CPU= 69.507
  15:34:57 STEP  669 H= 669:00 +CPU=108.792
  15:36:07 STEP  670 H= 670:00 +CPU= 69.554
  15:37:16 STEP  671 H= 671:00 +CPU= 68.746
  15:39:22 STEP  672 H= 672:00 +CPU=125.723
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000672
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000672
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000672
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000648
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000648
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000648
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000660
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000660
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000660
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_27.zip
Uploading the intermediate file: upload_file_27.zip
  15:40:29 STEP  673 H= 673:00 +CPU= 66.389
  15:41:39 STEP  674 H= 674:00 +CPU= 69.467
  15:43:28 STEP  675 H= 675:00 +CPU=108.956
  15:44:38 STEP  676 H= 676:00 +CPU= 69.274
  15:45:45 STEP  677 H= 677:00 +CPU= 66.714
  15:47:37 STEP  678 H= 678:00 +CPU=111.552
  15:48:43 STEP  679 H= 679:00 +CPU= 66.569
  15:49:53 STEP  680 H= 680:00 +CPU= 69.598
  15:51:43 STEP  681 H= 681:00 +CPU=108.900
  15:52:52 STEP  682 H= 682:00 +CPU= 69.570
  15:54:00 STEP  683 H= 683:00 +CPU= 67.046
  15:56:02 STEP  684 H= 684:00 +CPU=121.436
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000684
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000684
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000684
  15:57:09 STEP  685 H= 685:00 +CPU= 66.960
  15:58:19 STEP  686 H= 686:00 +CPU= 69.556
  16:00:08 STEP  687 H= 687:00 +CPU=109.070
  16:01:17 STEP  688 H= 688:00 +CPU= 69.232
  16:02:24 STEP  689 H= 689:00 +CPU= 66.834
  16:04:16 STEP  690 H= 690:00 +CPU=111.615
  16:05:23 STEP  691 H= 691:00 +CPU= 66.938
  16:06:32 STEP  692 H= 692:00 +CPU= 68.906
  16:08:22 STEP  693 H= 693:00 +CPU=108.906
  16:09:31 STEP  694 H= 694:00 +CPU= 69.537
  16:10:41 STEP  695 H= 695:00 +CPU= 69.155
  16:12:47 STEP  696 H= 696:00 +CPU=125.866
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000696
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000696
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000696
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000672
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000672
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000672
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000684
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000684
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000684
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_28.zip
Uploading the intermediate file: upload_file_28.zip
  16:13:54 STEP  697 H= 697:00 +CPU= 66.791
  16:15:04 STEP  698 H= 698:00 +CPU= 69.386
  16:16:53 STEP  699 H= 699:00 +CPU=109.148
  16:18:03 STEP  700 H= 700:00 +CPU= 69.183
  16:19:10 STEP  701 H= 701:00 +CPU= 66.861
  16:21:02 STEP  702 H= 702:00 +CPU=111.716
  16:22:09 STEP  703 H= 703:00 +CPU= 66.897
  16:23:19 STEP  704 H= 704:00 +CPU= 69.706
  16:25:09 STEP  705 H= 705:00 +CPU=109.527
  16:26:19 STEP  706 H= 706:00 +CPU= 69.860
  16:27:26 STEP  707 H= 707:00 +CPU= 66.733
  16:29:28 STEP  708 H= 708:00 +CPU=121.743
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000708
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000708
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000708
  16:30:36 STEP  709 H= 709:00 +CPU= 66.930
  16:31:46 STEP  710 H= 710:00 +CPU= 69.618
  16:33:35 STEP  711 H= 711:00 +CPU=109.008
  16:34:44 STEP  712 H= 712:00 +CPU= 69.355
  16:35:51 STEP  713 H= 713:00 +CPU= 66.789
  16:37:44 STEP  714 H= 714:00 +CPU=112.476
  16:38:51 STEP  715 H= 715:00 +CPU= 66.670
  16:40:01 STEP  716 H= 716:00 +CPU= 69.459
  16:41:50 STEP  717 H= 717:00 +CPU=109.345
  16:43:00 STEP  718 H= 718:00 +CPU= 69.525
  16:44:10 STEP  719 H= 719:00 +CPU= 69.156
  16:46:16 STEP  720 H= 720:00 +CPU=126.109
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000720
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000720
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000720
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000696
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000696
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000696
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000708
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000708
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000708
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_29.zip
Uploading the intermediate file: upload_file_29.zip
Uploading trickle at timestep: 2592000
  16:47:23 STEP  721 H= 721:00 +CPU= 66.946
  16:48:33 STEP  722 H= 722:00 +CPU= 69.621
  16:50:22 STEP  723 H= 723:00 +CPU=108.912
  16:51:32 STEP  724 H= 724:00 +CPU= 69.493
  16:52:39 STEP  725 H= 725:00 +CPU= 66.728
  16:54:31 STEP  726 H= 726:00 +CPU=112.021
  16:55:38 STEP  727 H= 727:00 +CPU= 66.864
  16:56:48 STEP  728 H= 728:00 +CPU= 69.430
  16:58:38 STEP  729 H= 729:00 +CPU=109.196
  16:59:48 STEP  730 H= 730:00 +CPU= 69.710
  17:00:54 STEP  731 H= 731:00 +CPU= 66.767
  17:02:57 STEP  732 H= 732:00 +CPU=121.564
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000732
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000732
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000732
  17:04:04 STEP  733 H= 733:00 +CPU= 66.993
  17:05:14 STEP  734 H= 734:00 +CPU= 69.955
  17:07:04 STEP  735 H= 735:00 +CPU=109.504
  17:08:14 STEP  736 H= 736:00 +CPU= 69.589
  17:09:21 STEP  737 H= 737:00 +CPU= 66.849
  17:11:14 STEP  738 H= 738:00 +CPU=112.337
  17:12:21 STEP  739 H= 739:00 +CPU= 66.812
  17:13:30 STEP  740 H= 740:00 +CPU= 69.350
  17:15:20 STEP  741 H= 741:00 +CPU=109.570
  17:16:30 STEP  742 H= 742:00 +CPU= 69.468
  17:17:39 STEP  743 H= 743:00 +CPU= 69.046
  17:19:46 STEP  744 H= 744:00 +CPU=125.767
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000744
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000744
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000744
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000720
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000720
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000720
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000732
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000732
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000732
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_30.zip
Uploading the intermediate file: upload_file_30.zip
  17:20:52 STEP  745 H= 745:00 +CPU= 66.498
  17:22:02 STEP  746 H= 746:00 +CPU= 69.543
  17:23:52 STEP  747 H= 747:00 +CPU=109.552
  17:25:02 STEP  748 H= 748:00 +CPU= 69.486
  17:26:09 STEP  749 H= 749:00 +CPU= 66.810
  17:28:01 STEP  750 H= 750:00 +CPU=112.400
  17:29:08 STEP  751 H= 751:00 +CPU= 66.806
  17:30:18 STEP  752 H= 752:00 +CPU= 69.701
  17:32:08 STEP  753 H= 753:00 +CPU=110.137
  17:33:18 STEP  754 H= 754:00 +CPU= 69.405
  17:34:25 STEP  755 H= 755:00 +CPU= 67.036
  17:36:29 STEP  756 H= 756:00 +CPU=122.366
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000756
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000756
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000756
  17:37:36 STEP  757 H= 757:00 +CPU= 66.910
  17:38:46 STEP  758 H= 758:00 +CPU= 69.846
  17:40:36 STEP  759 H= 759:00 +CPU=109.679
  17:41:46 STEP  760 H= 760:00 +CPU= 69.476
  17:42:53 STEP  761 H= 761:00 +CPU= 66.622
  17:44:46 STEP  762 H= 762:00 +CPU=112.628
  17:45:53 STEP  763 H= 763:00 +CPU= 66.910
  17:47:03 STEP  764 H= 764:00 +CPU= 69.561
  17:48:53 STEP  765 H= 765:00 +CPU=109.562
  17:50:02 STEP  766 H= 766:00 +CPU= 69.154
  17:51:11 STEP  767 H= 767:00 +CPU= 68.946
  17:53:18 STEP  768 H= 768:00 +CPU=126.728
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000768
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000768
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000768
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000744
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000744
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000744
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000756
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000756
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000756
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_31.zip
Uploading the intermediate file: upload_file_31.zip
  17:54:25 STEP  769 H= 769:00 +CPU= 66.543
  17:55:34 STEP  770 H= 770:00 +CPU= 69.205
  17:57:24 STEP  771 H= 771:00 +CPU=109.365
  17:58:34 STEP  772 H= 772:00 +CPU= 69.748
  17:59:41 STEP  773 H= 773:00 +CPU= 66.862
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2011050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0921
(argv4) batchid: 980
(argv5) wuid: 12197565
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /media/stuff/programs/boinc_data/slots/3
Project directory is: /media/stuff/programs/boinc_data/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565
..mkdir for temp folder for results failed
Copying: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /media/stuff/programs/boinc_data/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /media/stuff/programs/boinc_data/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9fd43ab0a9deb57eabc45a54073a86b4 to: /media/stuff/programs/boinc_data/slots/3/oifs_43r3_ps_0921_2011050100_123_980_12197565.zip
Unzipping the namelist zip file: /media/stuff/programs/boinc_data/slots/3/oifs_43r3_ps_0921_2011050100_123_980_12197565.zip
ic_ancil_file: ic_ancil_12197565
ifsdata_file: ifsdata_12197565
climate_data_file: clim_data_12197565
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_67d7c9efcda6efce162f49c54681698d to: /media/stuff/programs/boinc_data/slots/3/ic_ancil_12197565.zip
Unzipping the IC ancils zip file: /media/stuff/programs/boinc_data/slots/3/ic_ancil_12197565.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8bc681220aba7690fbec093e19bfcbaa to: /media/stuff/programs/boinc_data/slots/3/ifsdata/ifsdata_12197565.zip
Unzipping the ifsdata_zip file: /media/stuff/programs/boinc_data/slots/3/ifsdata/ifsdata_12197565.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /media/stuff/programs/boinc_data/slots/3/159l_2/clim_data_12197565.zip
Unzipping the climate data zip file: /media/stuff/programs/boinc_data/slots/3/159l_2/clim_data_12197565.zip
Checking for progress XML file: /media/stuff/programs/boinc_data/slots/3/progress_file_12197565.xml
Opened progress file ok : /media/stuff/programs/boinc_data/slots/3/progress_file_12197565.xml
-- Model is restarting --
Adjusting last_iter, 774, to previous model restart step.
Creating progress file: /media/stuff/programs/boinc_data/slots/3/progress_file_12197565.xml
last_cpu_time: 64846
upload_file_number: 32
last_iter: 767
last_upload: 2764800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407
The child process has been launched with process id: 23292
Executing the command: /media/stuff/programs/boinc_data/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pilot51-server2:1:1:23292:23292] [20230106:000008:1672981208:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:02:13 STEP  768 H= 768:00 +CPU=119.218
  00:03:20 STEP  769 H= 769:00 +CPU= 66.627
  00:04:30 STEP  770 H= 770:00 +CPU= 69.230
  00:06:19 STEP  771 H= 771:00 +CPU=109.350
  00:07:29 STEP  772 H= 772:00 +CPU= 69.615
  00:08:36 STEP  773 H= 773:00 +CPU= 66.753
  00:10:30 STEP  774 H= 774:00 +CPU=112.618
  00:11:37 STEP  775 H= 775:00 +CPU= 67.052
  00:12:48 STEP  776 H= 776:00 +CPU= 69.914
  00:14:38 STEP  777 H= 777:00 +CPU=109.596
  00:15:48 STEP  778 H= 778:00 +CPU= 69.464
  00:16:55 STEP  779 H= 779:00 +CPU= 66.866
  00:18:59 STEP  780 H= 780:00 +CPU=122.974
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000780
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000780
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000780
  00:20:06 STEP  781 H= 781:00 +CPU= 66.796
  00:21:16 STEP  782 H= 782:00 +CPU= 70.033
  00:23:06 STEP  783 H= 783:00 +CPU=109.435
  00:24:16 STEP  784 H= 784:00 +CPU= 69.647
  00:25:23 STEP  785 H= 785:00 +CPU= 66.931
  00:27:16 STEP  786 H= 786:00 +CPU=112.165
  00:28:23 STEP  787 H= 787:00 +CPU= 66.661
  00:29:33 STEP  788 H= 788:00 +CPU= 69.732
  00:31:23 STEP  789 H= 789:00 +CPU=109.751
  00:32:33 STEP  790 H= 790:00 +CPU= 69.155
  00:33:42 STEP  791 H= 791:00 +CPU= 69.054
  00:35:48 STEP  792 H= 792:00 +CPU=126.044
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000792
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000792
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000792
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000768
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000768
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000768
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000780
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000780
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000780
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_32.zip
Uploading the intermediate file: upload_file_32.zip
  00:36:56 STEP  793 H= 793:00 +CPU= 66.688
  00:38:06 STEP  794 H= 794:00 +CPU= 69.509
  00:39:56 STEP  795 H= 795:00 +CPU=109.634
  00:41:06 STEP  796 H= 796:00 +CPU= 69.190
  00:42:13 STEP  797 H= 797:00 +CPU= 66.815
  00:44:07 STEP  798 H= 798:00 +CPU=112.959
  00:45:13 STEP  799 H= 799:00 +CPU= 66.637
  00:46:24 STEP  800 H= 800:00 +CPU= 69.975
  00:48:14 STEP  801 H= 801:00 +CPU=109.882
  00:49:24 STEP  802 H= 802:00 +CPU= 69.576
  00:50:31 STEP  803 H= 803:00 +CPU= 66.899
  00:52:35 STEP  804 H= 804:00 +CPU=122.886
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000804
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000804
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000804
  00:53:42 STEP  805 H= 805:00 +CPU= 66.846
  00:54:52 STEP  806 H= 806:00 +CPU= 69.812
  00:56:43 STEP  807 H= 807:00 +CPU=109.886
  00:57:53 STEP  808 H= 808:00 +CPU= 69.630
  00:59:00 STEP  809 H= 809:00 +CPU= 66.922
  01:00:55 STEP  810 H= 810:00 +CPU=113.342
  01:02:01 STEP  811 H= 811:00 +CPU= 66.568
  01:03:11 STEP  812 H= 812:00 +CPU= 69.286
  01:05:01 STEP  813 H= 813:00 +CPU=109.978
  01:06:11 STEP  814 H= 814:00 +CPU= 69.332
  01:07:20 STEP  815 H= 815:00 +CPU= 68.901
  01:09:27 STEP  816 H= 816:00 +CPU=126.515
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000816
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000816
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000816
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000792
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000792
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000792
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000804
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000804
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000804
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_33.zip
Uploading the intermediate file: upload_file_33.zip
  01:10:34 STEP  817 H= 817:00 +CPU= 66.579
  01:11:45 STEP  818 H= 818:00 +CPU= 69.885
  01:13:35 STEP  819 H= 819:00 +CPU=109.711
  01:14:45 STEP  820 H= 820:00 +CPU= 69.347
  01:15:52 STEP  821 H= 821:00 +CPU= 66.907
  01:17:45 STEP  822 H= 822:00 +CPU=112.561
  01:18:53 STEP  823 H= 823:00 +CPU= 67.021
  01:20:02 STEP  824 H= 824:00 +CPU= 69.050
  01:21:52 STEP  825 H= 825:00 +CPU=109.879
  01:23:02 STEP  826 H= 826:00 +CPU= 69.792
  01:24:10 STEP  827 H= 827:00 +CPU= 67.080
  01:26:13 STEP  828 H= 828:00 +CPU=122.512
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000828
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000828
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000828
  01:27:20 STEP  829 H= 829:00 +CPU= 66.791
  01:28:30 STEP  830 H= 830:00 +CPU= 69.343
  01:30:20 STEP  831 H= 831:00 +CPU=110.170
  01:31:29 STEP  832 H= 832:00 +CPU= 69.268
  01:32:36 STEP  833 H= 833:00 +CPU= 66.247
  01:34:29 STEP  834 H= 834:00 +CPU=112.696
  01:35:37 STEP  835 H= 835:00 +CPU= 66.996
  01:36:46 STEP  836 H= 836:00 +CPU= 69.334
  01:38:37 STEP  837 H= 837:00 +CPU=110.031
  01:39:47 STEP  838 H= 838:00 +CPU= 69.357
  01:40:56 STEP  839 H= 839:00 +CPU= 68.950
  01:43:03 STEP  840 H= 840:00 +CPU=126.720
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000840
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000840
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000840
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000816
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000816
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000816
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000828
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000828
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000828
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_34.zip
Uploading the intermediate file: upload_file_34.zip
  01:44:10 STEP  841 H= 841:00 +CPU= 66.918
  01:45:20 STEP  842 H= 842:00 +CPU= 69.351
  01:47:10 STEP  843 H= 843:00 +CPU=109.618
  01:48:20 STEP  844 H= 844:00 +CPU= 69.907
  01:49:28 STEP  845 H= 845:00 +CPU= 66.956
  01:51:22 STEP  846 H= 846:00 +CPU=113.102
  01:52:28 STEP  847 H= 847:00 +CPU= 66.778
  01:53:38 STEP  848 H= 848:00 +CPU= 69.334
  01:55:30 STEP  849 H= 849:00 +CPU=110.555
  01:56:40 STEP  850 H= 850:00 +CPU= 69.891
  01:57:48 STEP  851 H= 851:00 +CPU= 67.038
  01:59:50 STEP  852 H= 852:00 +CPU=122.018
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000852
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000852
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000852
  02:00:57 STEP  853 H= 853:00 +CPU= 67.074
  02:02:07 STEP  854 H= 854:00 +CPU= 70.005
  02:03:58 STEP  855 H= 855:00 +CPU=110.296
  02:05:07 STEP  856 H= 856:00 +CPU= 69.169
  02:06:15 STEP  857 H= 857:00 +CPU= 66.819
  02:08:08 STEP  858 H= 858:00 +CPU=112.777
  02:09:16 STEP  859 H= 859:00 +CPU= 67.135
  02:10:25 STEP  860 H= 860:00 +CPU= 69.179
  02:12:15 STEP  861 H= 861:00 +CPU=109.463
  02:13:25 STEP  862 H= 862:00 +CPU= 69.755
  02:14:35 STEP  863 H= 863:00 +CPU= 69.374
  02:16:42 STEP  864 H= 864:00 +CPU=126.445
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000864
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000864
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000864
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000840
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000840
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000840
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000852
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000852
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000852
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_35.zip
Uploading the intermediate file: upload_file_35.zip
  02:17:49 STEP  865 H= 865:00 +CPU= 66.634
  02:18:59 STEP  866 H= 866:00 +CPU= 69.507
  02:20:50 STEP  867 H= 867:00 +CPU=109.706
  02:21:59 STEP  868 H= 868:00 +CPU= 69.535
  02:23:06 STEP  869 H= 869:00 +CPU= 66.683
  02:25:00 STEP  870 H= 870:00 +CPU=113.221
  02:26:07 STEP  871 H= 871:00 +CPU= 66.984
  02:27:17 STEP  872 H= 872:00 +CPU= 69.811
  02:29:07 STEP  873 H= 873:00 +CPU=109.425
  02:30:18 STEP  874 H= 874:00 +CPU= 69.615
  02:31:25 STEP  875 H= 875:00 +CPU= 67.083
  02:33:29 STEP  876 H= 876:00 +CPU=122.459
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000876
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000876
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000876
  02:34:35 STEP  877 H= 877:00 +CPU= 66.489
  02:35:45 STEP  878 H= 878:00 +CPU= 69.634
  02:37:35 STEP  879 H= 879:00 +CPU=109.683
  02:38:46 STEP  880 H= 880:00 +CPU= 69.676
  02:39:53 STEP  881 H= 881:00 +CPU= 66.950
  02:41:46 STEP  882 H= 882:00 +CPU=112.695
  02:42:54 STEP  883 H= 883:00 +CPU= 66.816
  02:44:04 STEP  884 H= 884:00 +CPU= 69.966
  02:45:55 STEP  885 H= 885:00 +CPU=109.649
  02:47:05 STEP  886 H= 886:00 +CPU= 69.655
  02:48:15 STEP  887 H= 887:00 +CPU= 69.495
  02:50:23 STEP  888 H= 888:00 +CPU=126.943
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000888
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000888
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000888
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000864
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000864
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000864
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000876
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000876
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000876
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_36.zip
Uploading the intermediate file: upload_file_36.zip
  02:51:31 STEP  889 H= 889:00 +CPU= 66.607
  02:52:41 STEP  890 H= 890:00 +CPU= 69.584
  02:54:32 STEP  891 H= 891:00 +CPU=110.679
  02:55:42 STEP  892 H= 892:00 +CPU= 69.733
  02:56:49 STEP  893 H= 893:00 +CPU= 66.837
  02:58:42 STEP  894 H= 894:00 +CPU=112.780
  02:59:49 STEP  895 H= 895:00 +CPU= 66.700
  03:01:00 STEP  896 H= 896:00 +CPU= 69.926
  03:02:50 STEP  897 H= 897:00 +CPU=109.844
  03:04:00 STEP  898 H= 898:00 +CPU= 70.031
  03:05:08 STEP  899 H= 899:00 +CPU= 66.958
  03:07:11 STEP  900 H= 900:00 +CPU=122.726
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000900
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000900
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000900
  03:08:18 STEP  901 H= 901:00 +CPU= 67.145
  03:09:28 STEP  902 H= 902:00 +CPU= 69.436
  03:11:18 STEP  903 H= 903:00 +CPU=109.765
  03:12:28 STEP  904 H= 904:00 +CPU= 69.673
  03:13:35 STEP  905 H= 905:00 +CPU= 66.782
  03:15:28 STEP  906 H= 906:00 +CPU=113.184
  03:16:35 STEP  907 H= 907:00 +CPU= 66.586
  03:17:45 STEP  908 H= 908:00 +CPU= 69.612
  03:19:35 STEP  909 H= 909:00 +CPU=109.805
  03:20:45 STEP  910 H= 910:00 +CPU= 69.734
  03:21:54 STEP  911 H= 911:00 +CPU= 68.987
  03:24:02 STEP  912 H= 912:00 +CPU=126.973
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000912
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000912
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000912
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000888
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000888
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000888
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000900
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000900
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000900
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_37.zip
Uploading the intermediate file: upload_file_37.zip
  03:25:09 STEP  913 H= 913:00 +CPU= 66.672
  03:26:19 STEP  914 H= 914:00 +CPU= 69.578
  03:28:08 STEP  915 H= 915:00 +CPU=109.265
  03:29:18 STEP  916 H= 916:00 +CPU= 69.183
  03:30:25 STEP  917 H= 917:00 +CPU= 67.052
  03:32:18 STEP  918 H= 918:00 +CPU=113.215
  03:33:26 STEP  919 H= 919:00 +CPU= 67.197
  03:34:36 STEP  920 H= 920:00 +CPU= 69.779
  03:36:26 STEP  921 H= 921:00 +CPU=109.628
  03:37:35 STEP  922 H= 922:00 +CPU= 69.270
  03:38:42 STEP  923 H= 923:00 +CPU= 66.670
  03:40:44 STEP  924 H= 924:00 +CPU=121.757
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000924
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000924
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000924
  03:41:51 STEP  925 H= 925:00 +CPU= 66.437
  03:43:00 STEP  926 H= 926:00 +CPU= 69.406
  03:44:50 STEP  927 H= 927:00 +CPU=109.302
  03:46:00 STEP  928 H= 928:00 +CPU= 69.846
  03:47:07 STEP  929 H= 929:00 +CPU= 67.030
  03:49:01 STEP  930 H= 930:00 +CPU=112.768
  03:50:08 STEP  931 H= 931:00 +CPU= 66.980
  03:51:18 STEP  932 H= 932:00 +CPU= 69.316
  03:53:08 STEP  933 H= 933:00 +CPU=109.596
  03:54:17 STEP  934 H= 934:00 +CPU= 69.392
  03:55:27 STEP  935 H= 935:00 +CPU= 69.406
  03:57:33 STEP  936 H= 936:00 +CPU=126.237
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000936
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000936
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000936
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000912
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000912
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000912
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000924
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000924
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000924
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_38.zip
Uploading the intermediate file: upload_file_38.zip
  03:58:40 STEP  937 H= 937:00 +CPU= 66.716
  03:59:50 STEP  938 H= 938:00 +CPU= 69.463
  04:01:40 STEP  939 H= 939:00 +CPU=109.548
  04:02:50 STEP  940 H= 940:00 +CPU= 69.831
  04:03:56 STEP  941 H= 941:00 +CPU= 66.477
  04:05:50 STEP  942 H= 942:00 +CPU=112.913
  04:06:57 STEP  943 H= 943:00 +CPU= 67.147
  04:08:07 STEP  944 H= 944:00 +CPU= 70.045
  04:09:57 STEP  945 H= 945:00 +CPU=109.349
  04:11:07 STEP  946 H= 946:00 +CPU= 69.884
  04:12:15 STEP  947 H= 947:00 +CPU= 67.117
  04:14:17 STEP  948 H= 948:00 +CPU=122.013
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000948
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000948
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000948
  04:15:24 STEP  949 H= 949:00 +CPU= 67.085
  04:16:34 STEP  950 H= 950:00 +CPU= 69.977
  04:18:24 STEP  951 H= 951:00 +CPU=109.600
  04:19:34 STEP  952 H= 952:00 +CPU= 69.318
  04:20:41 STEP  953 H= 953:00 +CPU= 66.786
  04:22:34 STEP  954 H= 954:00 +CPU=112.631
  04:23:41 STEP  955 H= 955:00 +CPU= 67.170
  04:24:51 STEP  956 H= 956:00 +CPU= 69.526
  04:26:41 STEP  957 H= 957:00 +CPU=109.964
  04:27:51 STEP  958 H= 958:00 +CPU= 69.447
  04:29:00 STEP  959 H= 959:00 +CPU= 69.291
  04:31:09 STEP  960 H= 960:00 +CPU=126.556
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000960
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000960
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000960
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000936
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000936
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000936
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000948
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000948
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000948
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_39.zip
Uploading the intermediate file: upload_file_39.zip
  04:32:16 STEP  961 H= 961:00 +CPU= 66.599
  04:33:26 STEP  962 H= 962:00 +CPU= 69.693
  04:35:16 STEP  963 H= 963:00 +CPU=109.386
  04:36:26 STEP  964 H= 964:00 +CPU= 69.609
  04:37:33 STEP  965 H= 965:00 +CPU= 66.914
  04:39:26 STEP  966 H= 966:00 +CPU=113.015
  04:40:34 STEP  967 H= 967:00 +CPU= 67.014
  04:41:43 STEP  968 H= 968:00 +CPU= 69.449
  04:43:32 STEP  969 H= 969:00 +CPU=109.328
  04:44:42 STEP  970 H= 970:00 +CPU= 69.709
  04:45:50 STEP  971 H= 971:00 +CPU= 67.147
  04:47:52 STEP  972 H= 972:00 +CPU=121.882
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000972
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000972
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000972
  04:48:59 STEP  973 H= 973:00 +CPU= 66.842
  04:50:09 STEP  974 H= 974:00 +CPU= 69.803
  04:51:59 STEP  975 H= 975:00 +CPU=109.734
  04:53:09 STEP  976 H= 976:00 +CPU= 69.596
  04:54:17 STEP  977 H= 977:00 +CPU= 67.125
  04:56:10 STEP  978 H= 978:00 +CPU=112.867
  04:57:17 STEP  979 H= 979:00 +CPU= 66.788
  04:58:27 STEP  980 H= 980:00 +CPU= 70.000
  05:00:18 STEP  981 H= 981:00 +CPU=110.516
  05:01:28 STEP  982 H= 982:00 +CPU= 69.791
  05:02:37 STEP  983 H= 983:00 +CPU= 68.893
  05:04:44 STEP  984 H= 984:00 +CPU=126.960
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000984
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000984
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000984
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000960
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000960
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000960
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000972
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000972
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000972
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_40.zip
Uploading the intermediate file: upload_file_40.zip
  05:05:52 STEP  985 H= 985:00 +CPU= 66.849
  05:07:01 STEP  986 H= 986:00 +CPU= 69.251
  05:08:51 STEP  987 H= 987:00 +CPU=109.895
  05:10:01 STEP  988 H= 988:00 +CPU= 69.614
  05:11:08 STEP  989 H= 989:00 +CPU= 66.972
  05:13:01 STEP  990 H= 990:00 +CPU=113.090
  05:14:09 STEP  991 H= 991:00 +CPU= 67.139
  05:15:20 STEP  992 H= 992:00 +CPU= 70.070
  05:17:10 STEP  993 H= 993:00 +CPU=109.676
  05:18:19 STEP  994 H= 994:00 +CPU= 69.669
  05:19:26 STEP  995 H= 995:00 +CPU= 66.703
  05:21:28 STEP  996 H= 996:00 +CPU=122.097
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+000996
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+000996
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+000996
  05:22:36 STEP  997 H= 997:00 +CPU= 67.197
  05:23:46 STEP  998 H= 998:00 +CPU= 69.709
  05:25:36 STEP  999 H= 999:00 +CPU=109.619
  05:26:46 STEP 1000 H=1000:00 +CPU= 69.740
  05:27:53 STEP 1001 H=1001:00 +CPU= 67.165
  05:29:46 STEP 1002 H=1002:00 +CPU=112.603
  05:30:53 STEP 1003 H=1003:00 +CPU= 66.452
  05:32:03 STEP 1004 H=1004:00 +CPU= 69.669
  05:33:52 STEP 1005 H=1005:00 +CPU=108.765
  05:35:02 STEP 1006 H=1006:00 +CPU= 69.705
  05:36:11 STEP 1007 H=1007:00 +CPU= 69.056
  05:38:18 STEP 1008 H=1008:00 +CPU=126.398
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMGGhq0f+001008
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMSHhq0f+001008
Moving to projects directory: /media/stuff/programs/boinc_data/slots/3/ICMUAhq0f+001008
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000984
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000984
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000984
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMGGhq0f+000996
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMSHhq0f+000996
Adding to the zip: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_12197565/ICMUAhq0f+000996
Zipping up the intermediate file: /media/stuff/programs/boinc_data/projects/climateprediction.net/oifs_43r3_ps_0921_2011050100_123_980_12197565_0_r248963407_41.zip
Uploading the intermediate file: upload_file_41.zip
Uploading trickle at timestep: 3628800
  05:39:25 STEP 1009 H=1009:00 +CPU= 66.999
  05:40:35 STEP 1010 H=1010:00 +CPU= 69.571
  05:42:25 STEP 1011 H=1011:00 +CPU=109.903
  05:43:35 STEP 1012 H=1012:00 +CPU= 69.687
  05:44:42 STEP 1013 H=1013:00 +CPU= 66.935
  05:46:35 STEP 1014 H=1014:00 +CPU=112.441
  05:47:42 STEP 1015 H=1015:00 +CPU= 67.183
  05:48:52 STEP 1016 H=1016:00 +CPU= 69.561
  05:50:42 STEP 1017 H=1017:00 +CPU=109.502
  05:51:52 STEP 1018 H=1018:00 +CPU= 69.594
  05:52:59 STEP 1019 H=1019:00 +CPU= 67.021
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:10:18 1443505 22283622 oifs_43r3_ps_0921_2011050100_123_980_12197565_0 3,628,800 85,054 0.0234
09 May 2023 16:09:33 1443505 22283622 oifs_43r3_ps_0921_2011050100_123_980_12197565_0 2,592,000 60,437 0.0233
09 May 2023 16:09:11 1443505 22283622 oifs_43r3_ps_0921_2011050100_123_980_12197565_0 1,728,000 40,394 0.0234
09 May 2023 16:08:51 1443505 22283622 oifs_43r3_ps_0921_2011050100_123_980_12197565_0 864,000 20,280 0.0235


©2024 climateprediction.net