climateprediction.net home page
Task 22314770

Task 22314770

Name oifs_43r3_bl_a3pk_2016092300_15_991_12211595_1
Workunit 12211595
Created 24 Feb 2023, 5:36:50 UTC
Sent 24 Feb 2023, 5:37:56 UTC
Report deadline 25 Apr 2023, 5:37:56 UTC
Received 24 Feb 2023, 9:40:44 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1496581
Run time 2 hours 57 min 53 sec
CPU time 2 hours 53 min 24 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 4.71 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 5,070.54 MB
Peak swap size 5,930.79 MB
Peak disk usage 1,362.82 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
port/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x1)
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x1)
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:071913:1677219553:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:19:54 STEP    0 H=   0:00 +CPU= 36.314
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000000
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000000
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000000
  07:20:24 STEP    1 H=   0:15 +CPU= 29.475
  07:20:54 STEP    2 H=   0:30 +CPU= 29.621
  07:21:25 STEP    3 H=   0:45 +CPU= 29.510
  07:21:56 STEP    4 H=   1:00 +CPU= 30.040
  07:22:26 STEP    5 H=   1:15 +CPU= 29.281
  07:22:56 STEP    6 H=   1:30 +CPU= 29.401
  07:23:26 STEP    7 H=   1:45 +CPU= 29.321
  07:23:56 STEP    8 H=   2:00 +CPU= 30.024
  07:24:26 STEP    9 H=   2:15 +CPU= 29.310
  07:24:56 STEP   10 H=   2:30 +CPU= 29.329
  07:25:27 STEP   11 H=   2:45 +CPU= 29.333
  07:26:04 STEP   12 H=   3:00 +CPU= 36.650
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000012
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000012
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000012
  07:26:34 STEP   13 H=   3:15 +CPU= 29.327
  07:27:04 STEP   14 H=   3:30 +CPU= 29.464
  07:27:34 STEP   15 H=   3:45 +CPU= 29.452
  07:28:05 STEP   16 H=   4:00 +CPU= 30.196
  07:28:35 STEP   17 H=   4:15 +CPU= 29.501
  07:29:05 STEP   18 H=   4:30 +CPU= 29.615
  07:29:35 STEP   19 H=   4:45 +CPU= 29.410
  07:30:05 STEP   20 H=   5:00 +CPU= 30.122
  07:30:35 STEP   21 H=   5:15 +CPU= 29.475
  07:31:05 STEP   22 H=   5:30 +CPU= 29.449
  07:31:35 STEP   23 H=   5:45 +CPU= 29.398
  07:32:13 STEP   24 H=   6:00 +CPU= 36.865
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000024
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000024
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000024
  07:32:43 STEP   25 H=   6:15 +CPU= 29.493
  07:33:13 STEP   26 H=   6:30 +CPU= 29.486
  07:33:43 STEP   27 H=   6:45 +CPU= 29.492
  07:34:14 STEP   28 H=   7:00 +CPU= 30.198
  07:34:44 STEP   29 H=   7:15 +CPU= 29.435
  07:35:14 STEP   30 H=   7:30 +CPU= 29.531
  07:35:44 STEP   31 H=   7:45 +CPU= 29.574
  07:36:15 STEP   32 H=   8:00 +CPU= 30.247
  07:36:45 STEP   33 H=   8:15 +CPU= 29.510
  07:37:15 STEP   34 H=   8:30 +CPU= 29.470
  07:37:45 STEP   35 H=   8:45 +CPU= 29.505
  07:38:22 STEP   36 H=   9:00 +CPU= 36.774
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000036
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000036
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000036
  07:38:53 STEP   37 H=   9:15 +CPU= 29.622
  07:39:23 STEP   38 H=   9:30 +CPU= 29.679
  07:39:53 STEP   39 H=   9:45 +CPU= 29.675
  07:40:24 STEP   40 H=  10:00 +CPU= 30.385
  07:40:54 STEP   41 H=  10:15 +CPU= 29.681
  07:41:25 STEP   42 H=  10:30 +CPU= 29.715
  07:41:55 STEP   43 H=  10:45 +CPU= 29.631
  07:42:26 STEP   44 H=  11:00 +CPU= 30.329
  07:42:56 STEP   45 H=  11:15 +CPU= 29.577
  07:43:26 STEP   46 H=  11:30 +CPU= 29.580
  07:43:56 STEP   47 H=  11:45 +CPU= 29.690
  07:44:39 STEP   48 H=  12:00 +CPU= 41.004
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000048
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000048
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000048
  07:45:09 STEP   49 H=  12:15 +CPU= 29.699
  07:45:39 STEP   50 H=  12:30 +CPU= 29.736
  07:46:09 STEP   51 H=  12:45 +CPU= 29.795
  07:46:41 STEP   52 H=  13:00 +CPU= 30.493
  07:47:11 STEP   53 H=  13:15 +CPU= 29.811
  07:47:41 STEP   54 H=  13:30 +CPU= 29.841
  07:48:12 STEP   55 H=  13:45 +CPU= 29.857
  07:48:43 STEP   56 H=  14:00 +CPU= 30.545
  07:49:13 STEP   57 H=  14:15 +CPU= 29.729
  07:49:43 STEP   58 H=  14:30 +CPU= 29.732
  07:50:14 STEP   59 H=  14:45 +CPU= 29.674
  07:50:52 STEP   60 H=  15:00 +CPU= 37.161
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000060
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000060
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000060
  07:51:22 STEP   61 H=  15:15 +CPU= 29.747
  07:51:52 STEP   62 H=  15:30 +CPU= 29.835
  07:52:23 STEP   63 H=  15:45 +CPU= 29.842
  07:52:54 STEP   64 H=  16:00 +CPU= 30.540
  07:53:24 STEP   65 H=  16:15 +CPU= 29.824
  07:53:55 STEP   66 H=  16:30 +CPU= 29.790
  07:54:25 STEP   67 H=  16:45 +CPU= 29.791
  07:54:56 STEP   68 H=  17:00 +CPU= 30.418
  07:55:26 STEP   69 H=  17:15 +CPU= 29.764
  07:55:57 STEP   70 H=  17:30 +CPU= 29.698
  07:56:27 STEP   71 H=  17:45 +CPU= 29.689
  07:57:05 STEP   72 H=  18:00 +CPU= 37.023
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000072
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000072
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000072
  07:57:35 STEP   73 H=  18:15 +CPU= 29.668
  07:58:05 STEP   74 H=  18:30 +CPU= 29.654
  07:58:36 STEP   75 H=  18:45 +CPU= 29.748
  07:59:07 STEP   76 H=  19:00 +CPU= 30.442
  07:59:37 STEP   77 H=  19:15 +CPU= 29.802
  08:00:07 STEP   78 H=  19:30 +CPU= 29.684
  08:00:38 STEP   79 H=  19:45 +CPU= 29.684
  08:01:09 STEP   80 H=  20:00 +CPU= 30.481
  08:01:39 STEP   81 H=  20:15 +CPU= 29.703
  08:02:09 STEP   82 H=  20:30 +CPU= 29.732
  08:02:39 STEP   83 H=  20:45 +CPU= 29.628
  08:03:17 STEP   84 H=  21:00 +CPU= 37.004
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000084
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000084
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000084
  08:03:48 STEP   85 H=  21:15 +CPU= 29.678
  08:04:19 STEP   86 H=  21:30 +CPU= 29.762
  08:04:50 STEP   87 H=  21:45 +CPU= 29.710
  08:05:21 STEP   88 H=  22:00 +CPU= 30.501
  08:05:51 STEP   89 H=  22:15 +CPU= 29.773
  08:06:21 STEP   90 H=  22:30 +CPU= 29.783
  08:06:52 STEP   91 H=  22:45 +CPU= 29.823
  08:07:23 STEP   92 H=  23:00 +CPU= 30.574
  08:07:53 STEP   93 H=  23:15 +CPU= 29.755
  08:08:24 STEP   94 H=  23:30 +CPU= 29.788
  08:08:54 STEP   95 H=  23:45 +CPU= 29.826
  08:09:36 STEP   96 H=  24:00 +CPU= 40.801
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000096
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000096
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000096
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000000
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000000
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000000
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000012
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000012
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000012
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000024
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000024
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000024
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000036
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000036
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000036
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000048
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000048
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000048
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000060
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000060
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000060
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000072
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000072
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000072
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000084
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000084
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000084
Zipping up the intermediate file: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_a3pk_2016092300_15_991_12211595_1_r1333941547_0.zip
Uploading the intermediate file: upload_file_0.zip
  08:10:09 STEP   97 H=  24:15 +CPU= 29.854
  08:10:39 STEP   98 H=  24:30 +CPU= 29.889
  08:11:10 STEP   99 H=  24:45 +CPU= 29.877
  08:11:41 STEP  100 H=  25:00 +CPU= 30.578
  08:12:11 STEP  101 H=  25:15 +CPU= 29.906
  08:12:42 STEP  102 H=  25:30 +CPU= 29.894
  08:13:12 STEP  103 H=  25:45 +CPU= 29.928
  08:13:44 STEP  104 H=  26:00 +CPU= 30.604
  08:14:14 STEP  105 H=  26:15 +CPU= 29.862
  08:14:45 STEP  106 H=  26:30 +CPU= 29.909
  08:15:15 STEP  107 H=  26:45 +CPU= 29.852
  08:15:53 STEP  108 H=  27:00 +CPU= 37.217
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000108
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000108
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000108
  08:16:24 STEP  109 H=  27:15 +CPU= 29.874
  08:16:54 STEP  110 H=  27:30 +CPU= 29.875
  08:17:24 STEP  111 H=  27:45 +CPU= 29.932
  08:17:56 STEP  112 H=  28:00 +CPU= 30.694
  08:18:26 STEP  113 H=  28:15 +CPU= 29.925
  08:18:57 STEP  114 H=  28:30 +CPU= 29.978
  08:19:28 STEP  115 H=  28:45 +CPU= 29.964
  08:19:59 STEP  116 H=  29:00 +CPU= 30.565
  08:20:29 STEP  117 H=  29:15 +CPU= 29.943
  08:21:00 STEP  118 H=  29:30 +CPU= 30.019
  08:21:30 STEP  119 H=  29:45 +CPU= 29.905
  08:22:08 STEP  120 H=  30:00 +CPU= 37.285
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000120
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000120
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000120
  08:22:39 STEP  121 H=  30:15 +CPU= 29.912
  08:23:09 STEP  122 H=  30:30 +CPU= 29.924
  08:23:39 STEP  123 H=  30:45 +CPU= 29.943
  08:24:11 STEP  124 H=  31:00 +CPU= 30.698
  08:24:41 STEP  125 H=  31:15 +CPU= 30.006
  08:25:12 STEP  126 H=  31:30 +CPU= 29.976
  08:25:42 STEP  127 H=  31:45 +CPU= 29.917
  08:26:13 STEP  128 H=  32:00 +CPU= 30.696
  08:26:44 STEP  129 H=  32:15 +CPU= 29.838
  08:27:14 STEP  130 H=  32:30 +CPU= 30.009
  08:27:45 STEP  131 H=  32:45 +CPU= 29.842
  08:28:22 STEP  132 H=  33:00 +CPU= 37.194
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000132
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000132
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000132
  08:28:53 STEP  133 H=  33:15 +CPU= 29.918
  08:29:23 STEP  134 H=  33:30 +CPU= 29.939
  08:29:54 STEP  135 H=  33:45 +CPU= 29.910
  08:30:25 STEP  136 H=  34:00 +CPU= 30.648
  08:30:56 STEP  137 H=  34:15 +CPU= 29.979
  08:31:26 STEP  138 H=  34:30 +CPU= 29.929
  08:31:57 STEP  139 H=  34:45 +CPU= 29.936
  08:32:28 STEP  140 H=  35:00 +CPU= 30.629
  08:32:58 STEP  141 H=  35:15 +CPU= 29.880
  08:33:29 STEP  142 H=  35:30 +CPU= 29.921
  08:33:59 STEP  143 H=  35:45 +CPU= 29.764
  08:34:41 STEP  144 H=  36:00 +CPU= 40.892
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000144
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000144
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000144
  08:35:11 STEP  145 H=  36:15 +CPU= 29.936
  08:35:42 STEP  146 H=  36:30 +CPU= 29.916
  08:36:12 STEP  147 H=  36:45 +CPU= 29.765
  08:36:43 STEP  148 H=  37:00 +CPU= 30.475
  08:37:13 STEP  149 H=  37:15 +CPU= 29.740
  08:37:44 STEP  150 H=  37:30 +CPU= 29.713
  08:38:14 STEP  151 H=  37:45 +CPU= 29.798
  08:38:45 STEP  152 H=  38:00 +CPU= 30.498
  08:39:15 STEP  153 H=  38:15 +CPU= 29.804
  08:39:45 STEP  154 H=  38:30 +CPU= 29.754
  08:40:16 STEP  155 H=  38:45 +CPU= 29.782
  08:40:54 STEP  156 H=  39:00 +CPU= 37.032
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000156
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000156
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000156
  08:41:24 STEP  157 H=  39:15 +CPU= 29.715
  08:41:54 STEP  158 H=  39:30 +CPU= 29.735
  08:42:25 STEP  159 H=  39:45 +CPU= 29.671
  08:42:56 STEP  160 H=  40:00 +CPU= 30.398
  08:43:26 STEP  161 H=  40:15 +CPU= 29.707
  08:43:56 STEP  162 H=  40:30 +CPU= 29.700
  08:44:27 STEP  163 H=  40:45 +CPU= 29.663
  08:44:57 STEP  164 H=  41:00 +CPU= 30.365
  08:45:28 STEP  165 H=  41:15 +CPU= 29.707
  08:45:58 STEP  166 H=  41:30 +CPU= 29.710
  08:46:28 STEP  167 H=  41:45 +CPU= 29.799
  08:47:06 STEP  168 H=  42:00 +CPU= 37.243
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000168
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000168
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000168
  08:47:37 STEP  169 H=  42:15 +CPU= 29.844
  08:48:07 STEP  170 H=  42:30 +CPU= 29.831
  08:48:37 STEP  171 H=  42:45 +CPU= 29.792
  08:49:08 STEP  172 H=  43:00 +CPU= 30.541
  08:49:39 STEP  173 H=  43:15 +CPU= 29.879
  08:50:09 STEP  174 H=  43:30 +CPU= 29.779
  08:50:40 STEP  175 H=  43:45 +CPU= 29.841
  08:51:11 STEP  176 H=  44:00 +CPU= 30.523
  08:51:41 STEP  177 H=  44:15 +CPU= 29.813
  08:52:12 STEP  178 H=  44:30 +CPU= 29.865
  08:52:42 STEP  179 H=  44:45 +CPU= 29.891
  08:53:20 STEP  180 H=  45:00 +CPU= 36.849
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000180
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000180
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000180
  08:53:50 STEP  181 H=  45:15 +CPU= 29.851
  08:54:21 STEP  182 H=  45:30 +CPU= 29.895
  08:54:51 STEP  183 H=  45:45 +CPU= 29.848
  08:55:22 STEP  184 H=  46:00 +CPU= 30.558
  08:55:53 STEP  185 H=  46:15 +CPU= 29.880
  08:56:23 STEP  186 H=  46:30 +CPU= 29.878
  08:56:53 STEP  187 H=  46:45 +CPU= 29.853
  08:57:24 STEP  188 H=  47:00 +CPU= 30.472
  08:57:55 STEP  189 H=  47:15 +CPU= 29.798
  08:58:25 STEP  190 H=  47:30 +CPU= 29.818
  08:58:56 STEP  191 H=  47:45 +CPU= 29.811
  08:59:37 STEP  192 H=  48:00 +CPU= 40.716
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000192
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000192
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000192
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000096
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000096
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000096
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000108
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000108
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000108
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000120
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000120
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000120
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000132
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000132
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000132
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000144
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000144
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000144
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000156
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000156
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000156
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000168
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000168
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000168
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000180
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000180
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000180
Zipping up the intermediate file: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_a3pk_2016092300_15_991_12211595_1_r1333941547_1.zip
Uploading the intermediate file: upload_file_1.zip
  09:00:09 STEP  193 H=  48:15 +CPU= 29.772
  09:00:39 STEP  194 H=  48:30 +CPU= 29.852
  09:01:10 STEP  195 H=  48:45 +CPU= 29.771
  09:01:41 STEP  196 H=  49:00 +CPU= 30.552
  09:02:11 STEP  197 H=  49:15 +CPU= 29.639
  09:02:41 STEP  198 H=  49:30 +CPU= 29.625
  09:03:11 STEP  199 H=  49:45 +CPU= 29.726
  09:03:42 STEP  200 H=  50:00 +CPU= 30.495
  09:04:13 STEP  201 H=  50:15 +CPU= 29.774
  09:04:43 STEP  202 H=  50:30 +CPU= 29.835
  09:05:14 STEP  203 H=  50:45 +CPU= 29.761
  09:05:52 STEP  204 H=  51:00 +CPU= 36.919
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000204
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000204
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000204
  09:06:22 STEP  205 H=  51:15 +CPU= 29.733
  09:06:52 STEP  206 H=  51:30 +CPU= 29.582
  09:07:22 STEP  207 H=  51:45 +CPU= 29.710
  09:07:53 STEP  208 H=  52:00 +CPU= 30.481
  09:08:24 STEP  209 H=  52:15 +CPU= 29.861
  09:08:55 STEP  210 H=  52:30 +CPU= 29.850
  09:09:25 STEP  211 H=  52:45 +CPU= 29.555
  09:09:56 STEP  212 H=  53:00 +CPU= 30.153
  09:10:26 STEP  213 H=  53:15 +CPU= 29.470
  09:10:56 STEP  214 H=  53:30 +CPU= 29.636
  09:11:26 STEP  215 H=  53:45 +CPU= 29.458
  09:12:03 STEP  216 H=  54:00 +CPU= 36.771
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000216
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000216
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000216
  09:12:33 STEP  217 H=  54:15 +CPU= 29.411
  09:13:03 STEP  218 H=  54:30 +CPU= 29.566
  09:13:33 STEP  219 H=  54:45 +CPU= 29.613
  09:14:04 STEP  220 H=  55:00 +CPU= 30.169
  09:14:34 STEP  221 H=  55:15 +CPU= 29.455
  09:15:04 STEP  222 H=  55:30 +CPU= 29.551
  09:15:34 STEP  223 H=  55:45 +CPU= 29.519
  09:16:05 STEP  224 H=  56:00 +CPU= 29.993
  09:16:35 STEP  225 H=  56:15 +CPU= 29.492
  09:17:05 STEP  226 H=  56:30 +CPU= 29.526
  09:17:35 STEP  227 H=  56:45 +CPU= 29.464
  09:18:12 STEP  228 H=  57:00 +CPU= 36.657
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000228
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000228
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000228
  09:18:42 STEP  229 H=  57:15 +CPU= 29.434
  09:19:12 STEP  230 H=  57:30 +CPU= 29.653
  09:19:42 STEP  231 H=  57:45 +CPU= 29.633
  09:20:13 STEP  232 H=  58:00 +CPU= 30.337
  09:20:43 STEP  233 H=  58:15 +CPU= 29.657
  09:21:14 STEP  234 H=  58:30 +CPU= 29.662
  09:21:44 STEP  235 H=  58:45 +CPU= 29.689
  09:22:15 STEP  236 H=  59:00 +CPU= 30.384
  09:22:46 STEP  237 H=  59:15 +CPU= 29.721
  09:23:16 STEP  238 H=  59:30 +CPU= 29.704
  09:23:46 STEP  239 H=  59:45 +CPU= 29.632
  09:24:27 STEP  240 H=  60:00 +CPU= 40.463
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000240
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000240
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000240
  09:24:58 STEP  241 H=  60:15 +CPU= 29.717
  09:25:28 STEP  242 H=  60:30 +CPU= 29.611
  09:25:58 STEP  243 H=  60:45 +CPU= 29.694
  09:26:29 STEP  244 H=  61:00 +CPU= 30.272
  09:26:59 STEP  245 H=  61:15 +CPU= 29.671
  09:27:29 STEP  246 H=  61:30 +CPU= 29.685
  09:27:59 STEP  247 H=  61:45 +CPU= 29.633
  09:28:30 STEP  248 H=  62:00 +CPU= 30.259
  09:29:00 STEP  249 H=  62:15 +CPU= 29.492
  09:29:30 STEP  250 H=  62:30 +CPU= 29.653
  09:30:01 STEP  251 H=  62:45 +CPU= 29.642
  09:30:38 STEP  252 H=  63:00 +CPU= 36.828
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000252
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000252
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000252
  09:31:08 STEP  253 H=  63:15 +CPU= 29.587
  09:31:38 STEP  254 H=  63:30 +CPU= 29.578
  09:32:08 STEP  255 H=  63:45 +CPU= 29.610
  09:32:39 STEP  256 H=  64:00 +CPU= 30.342
  09:33:09 STEP  257 H=  64:15 +CPU= 29.642
  09:33:40 STEP  258 H=  64:30 +CPU= 29.615
  09:34:10 STEP  259 H=  64:45 +CPU= 29.660
  09:34:41 STEP  260 H=  65:00 +CPU= 30.414
  09:35:11 STEP  261 H=  65:15 +CPU= 29.680
  09:35:41 STEP  262 H=  65:30 +CPU= 29.709
  09:36:11 STEP  263 H=  65:45 +CPU= 29.680
  09:36:49 STEP  264 H=  66:00 +CPU= 37.026
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000264
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000264
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000264
  09:37:19 STEP  265 H=  66:15 +CPU= 29.657
  09:37:49 STEP  266 H=  66:30 +CPU= 29.741
  09:38:19 STEP  267 H=  66:45 +CPU= 29.608
  09:38:50 STEP  268 H=  67:00 +CPU= 30.433
  09:39:20 STEP  269 H=  67:15 +CPU= 29.742
  09:39:51 STEP  270 H=  67:30 +CPU= 29.754
  09:40:21 STEP  271 H=  67:45 +CPU= 29.746
  09:40:52 STEP  272 H=  68:00 +CPU= 30.514
  09:41:23 STEP  273 H=  68:15 +CPU= 29.806
  09:41:53 STEP  274 H=  68:30 +CPU= 29.806
  09:42:23 STEP  275 H=  68:45 +CPU= 29.790
  09:43:01 STEP  276 H=  69:00 +CPU= 37.127
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000276
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000276
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000276
  09:43:31 STEP  277 H=  69:15 +CPU= 29.802
  09:44:02 STEP  278 H=  69:30 +CPU= 29.893
  09:44:32 STEP  279 H=  69:45 +CPU= 29.835
  09:45:03 STEP  280 H=  70:00 +CPU= 30.534
  09:45:33 STEP  281 H=  70:15 +CPU= 29.873
  09:46:04 STEP  282 H=  70:30 +CPU= 29.821
  09:46:34 STEP  283 H=  70:45 +CPU= 29.922
  09:47:05 STEP  284 H=  71:00 +CPU= 30.617
  09:47:36 STEP  285 H=  71:15 +CPU= 29.888
  09:48:06 STEP  286 H=  71:30 +CPU= 29.901
  09:48:37 STEP  287 H=  71:45 +CPU= 29.922
  09:49:18 STEP  288 H=  72:00 +CPU= 40.800
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000288
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000288
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000288
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000192
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000192
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000192
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000204
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000204
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000204
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000216
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000216
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000216
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000228
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000228
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000228
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000240
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000240
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000240
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000252
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000252
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000252
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000264
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000264
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000264
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMGGh7zg+000276
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMSHh7zg+000276
Adding to the zip: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_12211595/ICMUAh7zg+000276
Zipping up the intermediate file: /mnt/WDred/AnvMorgan/BOINC/projects/climateprediction.net/oifs_43r3_bl_a3pk_2016092300_15_991_12211595_1_r1333941547_2.zip
Uploading the intermediate file: upload_file_2.zip
  09:49:51 STEP  289 H=  72:15 +CPU= 29.996
  09:50:21 STEP  290 H=  72:30 +CPU= 29.918
  09:50:51 STEP  291 H=  72:45 +CPU= 29.455
  09:51:22 STEP  292 H=  73:00 +CPU= 30.458
  09:51:52 STEP  293 H=  73:15 +CPU= 29.549
  09:52:22 STEP  294 H=  73:30 +CPU= 29.627
  09:52:52 STEP  295 H=  73:45 +CPU= 29.739
  09:53:23 STEP  296 H=  74:00 +CPU= 30.352
  09:53:53 STEP  297 H=  74:15 +CPU= 29.583
  09:54:23 STEP  298 H=  74:30 +CPU= 29.662
  09:54:54 STEP  299 H=  74:45 +CPU= 29.726
  09:55:31 STEP  300 H=  75:00 +CPU= 36.773
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000300
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000300
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000300
  09:56:01 STEP  301 H=  75:15 +CPU= 29.541
  09:56:31 STEP  302 H=  75:30 +CPU= 29.716
  09:57:02 STEP  303 H=  75:45 +CPU= 29.641
  09:57:32 STEP  304 H=  76:00 +CPU= 30.247
  09:58:03 STEP  305 H=  76:15 +CPU= 29.696
  09:58:33 STEP  306 H=  76:30 +CPU= 29.687
  09:59:03 STEP  307 H=  76:45 +CPU= 29.777
  09:59:34 STEP  308 H=  77:00 +CPU= 30.547
  10:00:04 STEP  309 H=  77:15 +CPU= 29.734
  10:00:35 STEP  310 H=  77:30 +CPU= 29.764
  10:01:05 STEP  311 H=  77:45 +CPU= 29.786
  10:01:43 STEP  312 H=  78:00 +CPU= 37.017
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000312
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000312
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000312
  10:02:13 STEP  313 H=  78:15 +CPU= 29.690
  10:02:43 STEP  314 H=  78:30 +CPU= 29.774
  10:03:14 STEP  315 H=  78:45 +CPU= 29.776
  10:03:45 STEP  316 H=  79:00 +CPU= 30.486
  10:04:15 STEP  317 H=  79:15 +CPU= 29.578
  10:04:45 STEP  318 H=  79:30 +CPU= 29.785
  10:05:15 STEP  319 H=  79:45 +CPU= 29.646
  10:05:46 STEP  320 H=  80:00 +CPU= 30.378
  10:06:16 STEP  321 H=  80:15 +CPU= 29.908
  10:06:47 STEP  322 H=  80:30 +CPU= 29.619
  10:07:17 STEP  323 H=  80:45 +CPU= 29.784
  10:07:54 STEP  324 H=  81:00 +CPU= 36.728
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000324
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000324
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000324
  10:08:24 STEP  325 H=  81:15 +CPU= 29.646
  10:08:54 STEP  326 H=  81:30 +CPU= 29.708
  10:09:25 STEP  327 H=  81:45 +CPU= 29.706
  10:09:56 STEP  328 H=  82:00 +CPU= 30.449
  10:10:26 STEP  329 H=  82:15 +CPU= 29.653
  10:10:56 STEP  330 H=  82:30 +CPU= 29.642
  10:11:26 STEP  331 H=  82:45 +CPU= 29.766
  10:11:57 STEP  332 H=  83:00 +CPU= 30.357
  10:12:27 STEP  333 H=  83:15 +CPU= 29.769
  10:12:57 STEP  334 H=  83:30 +CPU= 29.787
  10:13:28 STEP  335 H=  83:45 +CPU= 29.777
  10:14:09 STEP  336 H=  84:00 +CPU= 40.445
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMGGh7zg+000336
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMSHh7zg+000336
Moving to projects directory: /mnt/WDred/AnvMorgan/BOINC/slots/8/ICMUAh7zg+000336
  10:14:39 STEP  337 H=  84:15 +CPU= 29.744
  10:15:10 STEP  338 H=  84:30 +CPU= 29.737
  10:15:40 STEP  339 H=  84:45 +CPU= 29.699
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 4362MB (heap), 5076MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f0c0 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc4a8f], nsigs = 1
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3843] 4362 MB (maxheap), 5076 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101600:1677230160:10607.777] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/mnt/WDred/AnvMorgan/BOINC/slots/8/oifs_43r3_model.exe' : sigcontextptr=0x7ffc3daaf6d0
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 24) with addr2line-cmd
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/mnt/WDred/AnvMorgan/BOINC/slots/8/oifs_43r3_model.exe' 0x838803 0x838e09 0x1d8bb20 0x1cc4a8f 0x12d283b 0x12ba8d9 0x12b0762 0x1163fc3 0x10b682f 0x10a915c 0xe680c3 0x15c1e62 0x15ac40e 0x6620b4 0x4a0e7c 0x49dc29 0x49ce72 0x49cd68 0x4078ad 0x40276f 0x40270d 0x1d8287a 0x1d840d7 0x4025e5
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x838803] : LinuxTraceBack() at linuxtrbk.c:264
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x838e09] : LinuxTraceBack() at linuxtrbk.c:236
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1d8bb20] : __restore_rt() at libc_sigaction.o:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1cc4a8f] : __libm_exp_e7() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x12d283b] : vdfmain_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x12ba8d9] : vdfouter_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x12b0762] : turbulence_layer_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x1163fc3] : callpar_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x10b682f] : ec_phys_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x10a915c] : ec_phys_drv_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xe680c3] : gp_model_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x15c1e62] : gp_model_heap_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x15ac40e] : scan2m_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x6620b4] : stepo_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x4a0e7c] : cnt4_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x49dc29] : cnt3_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x49ce72] : cnt2_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x49cd68] : cnt1_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x4078ad] : cnt0_() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x40276f] : MAIN__() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x40270d] : main() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x1d8287a] : __libc_start_call_main() at libc-start.o:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1d840d7] : __libc_start_main_impl() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x4025e5] : _start() at ??:?
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101601:1677230161:10608.778] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/mnt/WDred/AnvMorgan/BOINC/slots/8/oifs_43r3_model.exe' : sigcontextptr=0x7ffc3daaf5e0
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 25) with addr2line-cmd
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/mnt/WDred/AnvMorgan/BOINC/slots/8/oifs_43r3_model.exe' 0x838803 0x838e09 0x820d70 0x1d8bb20 0x1cc4a8f 0x12d283b 0x12ba8d9 0x12b0762 0x1163fc3 0x10b682f 0x10a915c 0xe680c3 0x15c1e62 0x15ac40e 0x6620b4 0x4a0e7c 0x49dc29 0x49ce72 0x49cd68 0x4078ad 0x40276f 0x40270d 0x1d8287a 0x1d840d7 0x4025e5
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x838803] : LinuxTraceBack() at linuxtrbk.c:264
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x838e09] : LinuxTraceBack() at linuxtrbk.c:236
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x820d70] : signal_drhook() at drhook.c:1734
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1d8bb20] : __restore_rt() at libc_sigaction.o:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x1cc4a8f] : __libm_exp_e7() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x12d283b] : vdfmain_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x12ba8d9] : vdfouter_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x12b0762] : turbulence_layer_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x1163fc3] : callpar_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x10b682f] : ec_phys_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x10a915c] : ec_phys_drv_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0xe680c3] : gp_model_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x15c1e62] : gp_model_heap_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x15ac40e] : scan2m_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x6620b4] : stepo_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4a0e7c] : cnt4_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x49dc29] : cnt3_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x49ce72] : cnt2_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x49cd68] : cnt1_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x4078ad] : cnt0_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x40276f] : MAIN__() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x40270d] : main() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1d8287a] : __libc_start_call_main() at libc-start.o:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x1d840d7] : __libc_start_main_impl() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [24]: [0x4025e5] : _start() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101602:1677230162:10608.919] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:svarten.tribun:1:1:3965104:3965104] [20230224:101602:1677230162:10608.919] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x401140 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  000000000081FCF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D8BB20  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC4A8F  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000012D283B  vdfmain_                  874  vdfmain.F90
oifs_43r3_model.e  00000000012BA8D9  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000012B0762  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000001163FC3  callpar_                  869  callpar.F90
oifs_43r3_model.e  00000000010B682F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  00000000010A915C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000E680C3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000015C1E62  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  00000000015AC40E  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  00000000006620B4  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004A0E7C  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  000000000049DC29  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  000000000049CE72  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049CD68  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  00000000004078AD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040276F  MAIN__                     96  master.F90
oifs_43r3_model.e  000000000040270D  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D8287A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D840D7  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000004025E5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.109081635009196E-05 0.000000000000000E+00 0.621721083772728E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.190151413802512E-05 0.000000000000000E+00 0.231071133099864E-03
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.288487579875768E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.828206737218058E-06 0.000000000000000E+00 0.858639071663285E-03
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.611990846269052E-06 0.000000000000000E+00 0.383813891898742E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.552863024364792E-05 0.000000000000000E+00 0.575697498267005E-02
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.322388187868056E-05 0.000000000000000E+00 0.444432635987916E-02
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.167481798489152E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =   336 STEPO   F00000000
 CREATE RESTART FILE ./srf00031200.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00031200.0001 MODE=w
 IO-STREAM CLOSED - ./srf00031200.0001
 MPI-TASK:   1 - ************ BYTES IN     34 RECORDS TRANSFERRED IN    2.4737 SECONDS******** Mbytes/s, TOTAL TIME=    3.1195(79.3%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00030000.0001
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =   336 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+000336 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+000336
 MPI-TASK:   1 -    296484570 BYTES IN      6 RECORDS TRANSFERRED IN    0.0004 SECONDS741211.4 Mbytes/s, TOTAL TIME=    0.0099( 4.0%)
 NSTEP =   336 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+000336 MODE=a
 IO-STREAM CLOSED - ./ICMSHh7zg+000336
 MPI-TASK:   1 -    707765250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0086 SECONDS 82298.3 Mbytes/s, TOTAL TIME=    0.3027( 2.8%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000336 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+000336
 MPI-TASK:   1 -   4743753111 BYTES IN    125 RECORDS TRANSFERRED IN    0.0052 SECONDS912260.2 Mbytes/s, TOTAL TIME=    0.2394( 2.2%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000336 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+000336
 MPI-TASK:   1 -    287910937 BYTES IN      2 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0040( 5.0%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000336 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+000336
 MPI-TASK:   1 -    296484570 BYTES IN      4 RECORDS TRANSFERRED IN    0.0003 SECONDS988281.9 Mbytes/s, TOTAL TIME=    0.0066( 4.5%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =   336 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =   336 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+000336 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+000336
 MPI-TASK:   1 -    296484561 BYTES IN      7 RECORDS TRANSFERRED IN    0.0004 SECONDS741211.4 Mbytes/s, TOTAL TIME=    0.0110( 3.6%)
 10:14:09 STEP  336 H=  84:00 +CPU= 40.445
 NSTEP =   337 STEPO   0AAA00AAA
 10:14:39 STEP  337 H=  84:15 +CPU= 29.744
 NSTEP =   338 STEPO   0AAA00AAA
 10:15:10 STEP  338 H=  84:30 +CPU= 29.737
 NSTEP =   339 STEPO   0AAA00AAA
 10:15:40 STEP  339 H=  84:45 +CPU= 29.699
ISTEP=     340 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=     340 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =   340 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 10:13:37 FULLPOS-V DYNFPOS    336     3.413    3.413    3.485    171:02    174:23 0.14308223185144E-04       2GB       0MB
 10:13:37 FULLPOS-T DYNFPOS    336     0.606    0.606    0.609    171:03    174:24 0.14308223185144E-04       2GB       0MB
 10:13:38 0AAA00AAA STEPO      336     0.622    0.622    0.628    171:04    174:25 0.14308223185144E-04       2GB       0MB
 10:14:09 0AAA00AAA STEPO      337    30.550   30.550   31.174    171:34    174:56 0.14308223185144E-04       2GB       0MB
 10:14:39 0AAA00AAA STEPO      338    29.744   29.744   30.332    172:04    175:26 0.14308223185144E-04       2GB       0MB
 10:15:10 0AAA00AAA STEPO      339    29.737   29.737   30.533    172:34    175:57 0.14308223185144E-04       2GB       0MB
 10:15:40 0AAA00AAA STEPO      340    29.699   29.699   30.183    173:03    176:27 0.14308223185144E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "     336",
 CTIME   = "00031200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,         324, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /mnt/WDred/AnvMorgan/BOINC/slots/8/progress_file_12211595.xml

<running_values>
  <last_cpu_time>10401.910000</last_cpu_time>
  <upload_file_number>3</upload_file_number>
  <last_iter>340</last_iter>
  <last_upload>259200</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net