climateprediction.net home page
Task 22298006

Task 22298006

Name oifs_43r3_ps_0134_2009050100_123_978_12194778_1
Workunit 12194778
Created 14 Jan 2023, 23:50:32 UTC
Sent 20 Jan 2023, 2:19:47 UTC
Report deadline 21 Mar 2023, 2:19:47 UTC
Received 21 Jan 2023, 1:17:24 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1538236
Run time 22 hours 23 min 34 sec
CPU time 22 hours 16 min
Validate state Valid
Credit 14,873.04
Device peak FLOPS 5.98 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.99 MB
Peak swap size 4,978.06 MB
Peak disk usage 2,018.90 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
151
  21:09:01 STEP 2523 H=2523:00 +CPU= 40.922
  21:09:28 STEP 2524 H=2524:00 +CPU= 26.582
  21:09:53 STEP 2525 H=2525:00 +CPU= 25.722
  21:10:34 STEP 2526 H=2526:00 +CPU= 40.380
  21:10:59 STEP 2527 H=2527:00 +CPU= 24.840
  21:11:26 STEP 2528 H=2528:00 +CPU= 27.511
  21:12:07 STEP 2529 H=2529:00 +CPU= 40.568
  21:12:35 STEP 2530 H=2530:00 +CPU= 27.483
  21:13:01 STEP 2531 H=2531:00 +CPU= 26.586
  21:13:46 STEP 2532 H=2532:00 +CPU= 45.014
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002532
  21:14:11 STEP 2533 H=2533:00 +CPU= 25.090
  21:14:39 STEP 2534 H=2534:00 +CPU= 27.364
  21:15:20 STEP 2535 H=2535:00 +CPU= 41.688
  21:15:48 STEP 2536 H=2536:00 +CPU= 27.763
  21:16:15 STEP 2537 H=2537:00 +CPU= 26.901
  21:16:58 STEP 2538 H=2538:00 +CPU= 42.678
  21:17:25 STEP 2539 H=2539:00 +CPU= 27.027
  21:17:53 STEP 2540 H=2540:00 +CPU= 28.282
  21:18:35 STEP 2541 H=2541:00 +CPU= 41.672
  21:19:03 STEP 2542 H=2542:00 +CPU= 28.296
  21:19:31 STEP 2543 H=2543:00 +CPU= 27.632
  21:20:20 STEP 2544 H=2544:00 +CPU= 48.849
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_105.zip
Uploading the intermediate file: upload_file_105.zip
  21:20:46 STEP 2545 H=2545:00 +CPU= 26.792
  21:21:12 STEP 2546 H=2546:00 +CPU= 25.652
  21:21:53 STEP 2547 H=2547:00 +CPU= 41.390
  21:22:22 STEP 2548 H=2548:00 +CPU= 28.648
  21:22:49 STEP 2549 H=2549:00 +CPU= 27.259
  21:23:27 STEP 2550 H=2550:00 +CPU= 37.871
  21:23:52 STEP 2551 H=2551:00 +CPU= 25.185
  21:24:20 STEP 2552 H=2552:00 +CPU= 27.787
  21:25:01 STEP 2553 H=2553:00 +CPU= 40.658
  21:25:29 STEP 2554 H=2554:00 +CPU= 27.833
  21:25:54 STEP 2555 H=2555:00 +CPU= 25.085
  21:26:38 STEP 2556 H=2556:00 +CPU= 43.571
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002556
  21:27:02 STEP 2557 H=2557:00 +CPU= 24.627
  21:27:31 STEP 2558 H=2558:00 +CPU= 28.382
  21:28:10 STEP 2559 H=2559:00 +CPU= 39.143
  21:28:37 STEP 2560 H=2560:00 +CPU= 27.236
  21:29:04 STEP 2561 H=2561:00 +CPU= 26.672
  21:29:47 STEP 2562 H=2562:00 +CPU= 42.894
  21:30:12 STEP 2563 H=2563:00 +CPU= 25.611
  21:30:38 STEP 2564 H=2564:00 +CPU= 25.825
  21:31:16 STEP 2565 H=2565:00 +CPU= 38.311
  21:31:44 STEP 2566 H=2566:00 +CPU= 27.366
  21:32:10 STEP 2567 H=2567:00 +CPU= 26.612
  21:32:59 STEP 2568 H=2568:00 +CPU= 48.834
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_106.zip
Uploading the intermediate file: upload_file_106.zip
  21:33:24 STEP 2569 H=2569:00 +CPU= 25.197
  21:33:48 STEP 2570 H=2570:00 +CPU= 23.781
  21:34:22 STEP 2571 H=2571:00 +CPU= 33.968
  21:34:47 STEP 2572 H=2572:00 +CPU= 24.424
  21:35:11 STEP 2573 H=2573:00 +CPU= 24.379
  21:35:46 STEP 2574 H=2574:00 +CPU= 35.060
  21:36:09 STEP 2575 H=2575:00 +CPU= 22.503
  21:36:32 STEP 2576 H=2576:00 +CPU= 23.372
  21:37:03 STEP 2577 H=2577:00 +CPU= 30.655
  21:37:25 STEP 2578 H=2578:00 +CPU= 22.678
  21:37:47 STEP 2579 H=2579:00 +CPU= 22.120
  21:38:26 STEP 2580 H=2580:00 +CPU= 38.108
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002580
  21:38:47 STEP 2581 H=2581:00 +CPU= 21.045
  21:39:11 STEP 2582 H=2582:00 +CPU= 24.559
  21:39:55 STEP 2583 H=2583:00 +CPU= 43.913
  21:40:22 STEP 2584 H=2584:00 +CPU= 26.584
  21:40:46 STEP 2585 H=2585:00 +CPU= 24.597
  21:41:21 STEP 2586 H=2586:00 +CPU= 34.449
  21:41:43 STEP 2587 H=2587:00 +CPU= 22.551
  21:42:08 STEP 2588 H=2588:00 +CPU= 24.411
  21:42:42 STEP 2589 H=2589:00 +CPU= 34.359
  21:43:05 STEP 2590 H=2590:00 +CPU= 22.737
  21:43:29 STEP 2591 H=2591:00 +CPU= 23.704
  21:44:10 STEP 2592 H=2592:00 +CPU= 40.989
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_107.zip
Uploading the intermediate file: upload_file_107.zip
  21:44:31 STEP 2593 H=2593:00 +CPU= 21.228
  21:44:51 STEP 2594 H=2594:00 +CPU= 20.085
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2009050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0134
(argv4) batchid: 978
(argv5) wuid: 12194778
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/11
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_97a9a96b5d05660e0485b2170479f0ef to: /var/lib/boinc-client_40000/slots/11/oifs_43r3_ps_0134_2009050100_123_978_12194778.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/11/oifs_43r3_ps_0134_2009050100_123_978_12194778.zip
ic_ancil_file: ic_ancil_12194778
ifsdata_file: ifsdata_12194778
climate_data_file: clim_data_12194778
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5b81e8e387e39576bb761b360057f4dd to: /var/lib/boinc-client_40000/slots/11/ic_ancil_12194778.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/11/ic_ancil_12194778.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8ed42ab98a1bf04f7a79b67415cdcedd to: /var/lib/boinc-client_40000/slots/11/ifsdata/ifsdata_12194778.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/11/ifsdata/ifsdata_12194778.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/11/159l_2/clim_data_12194778.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/11/159l_2/clim_data_12194778.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/11/progress_file_12194778.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/11/progress_file_12194778.xml
-- Model is restarting --
Adjusting last_iter, 2595, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/11/progress_file_12194778.xml
last_cpu_time: 69825
upload_file_number: 108
last_iter: 2591
last_upload: 9331200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913
The child process has been launched with process id: 1427
Executing the command: /var/lib/boinc-client_40000/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1427:1427] [20230120:215820:1674251900:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:00:01 STEP 2592 H=2592:00 +CPU= 23.300
  22:00:15 STEP 2593 H=2593:00 +CPU= 13.732
  22:00:31 STEP 2594 H=2594:00 +CPU= 15.536
  22:01:04 STEP 2595 H=2595:00 +CPU= 33.200
  22:01:27 STEP 2596 H=2596:00 +CPU= 23.480
  22:01:51 STEP 2597 H=2597:00 +CPU= 23.536
  22:02:29 STEP 2598 H=2598:00 +CPU= 37.896
  22:02:54 STEP 2599 H=2599:00 +CPU= 24.658
  22:03:14 STEP 2600 H=2600:00 +CPU= 20.747
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2009050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0134
(argv4) batchid: 978
(argv5) wuid: 12194778
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/11
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_97a9a96b5d05660e0485b2170479f0ef to: /var/lib/boinc-client_40000/slots/11/oifs_43r3_ps_0134_2009050100_123_978_12194778.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/11/oifs_43r3_ps_0134_2009050100_123_978_12194778.zip
ic_ancil_file: ic_ancil_12194778
ifsdata_file: ifsdata_12194778
climate_data_file: clim_data_12194778
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5b81e8e387e39576bb761b360057f4dd to: /var/lib/boinc-client_40000/slots/11/ic_ancil_12194778.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/11/ic_ancil_12194778.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8ed42ab98a1bf04f7a79b67415cdcedd to: /var/lib/boinc-client_40000/slots/11/ifsdata/ifsdata_12194778.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/11/ifsdata/ifsdata_12194778.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/11/159l_2/clim_data_12194778.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/11/159l_2/clim_data_12194778.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/11/progress_file_12194778.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/11/progress_file_12194778.xml
-- Model is restarting --
Adjusting last_iter, 2601, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/11/progress_file_12194778.xml
last_cpu_time: 70057
upload_file_number: 108
last_iter: 2591
last_upload: 9331200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913
The child process has been launched with process id: 1428
Executing the command: /var/lib/boinc-client_40000/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1428:1428] [20230120:221531:1674252931:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:17:09 STEP 2592 H=2592:00 +CPU= 24.318
  22:17:23 STEP 2593 H=2593:00 +CPU= 13.877
  22:17:36 STEP 2594 H=2594:00 +CPU= 13.699
  22:18:10 STEP 2595 H=2595:00 +CPU= 34.060
  22:18:33 STEP 2596 H=2596:00 +CPU= 22.056
  22:18:55 STEP 2597 H=2597:00 +CPU= 22.938
  22:19:30 STEP 2598 H=2598:00 +CPU= 34.253
  22:19:53 STEP 2599 H=2599:00 +CPU= 23.611
  22:20:18 STEP 2600 H=2600:00 +CPU= 24.396
  22:20:53 STEP 2601 H=2601:00 +CPU= 35.345
  22:21:15 STEP 2602 H=2602:00 +CPU= 21.464
  22:21:38 STEP 2603 H=2603:00 +CPU= 23.426
  22:22:19 STEP 2604 H=2604:00 +CPU= 41.403
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002604
  22:22:45 STEP 2605 H=2605:00 +CPU= 25.348
  22:23:11 STEP 2606 H=2606:00 +CPU= 25.920
  22:23:49 STEP 2607 H=2607:00 +CPU= 38.644
  22:24:15 STEP 2608 H=2608:00 +CPU= 25.769
  22:24:39 STEP 2609 H=2609:00 +CPU= 24.237
  22:25:17 STEP 2610 H=2610:00 +CPU= 37.260
  22:25:38 STEP 2611 H=2611:00 +CPU= 21.002
  22:25:59 STEP 2612 H=2612:00 +CPU= 21.215
  22:26:32 STEP 2613 H=2613:00 +CPU= 33.566
  22:26:56 STEP 2614 H=2614:00 +CPU= 23.122
  22:27:18 STEP 2615 H=2615:00 +CPU= 22.475
  22:27:54 STEP 2616 H=2616:00 +CPU= 35.975
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_108.zip
Uploading the intermediate file: upload_file_108.zip
  22:28:16 STEP 2617 H=2617:00 +CPU= 21.703
  22:28:38 STEP 2618 H=2618:00 +CPU= 22.432
  22:29:09 STEP 2619 H=2619:00 +CPU= 30.691
  22:29:32 STEP 2620 H=2620:00 +CPU= 22.342
  22:29:55 STEP 2621 H=2621:00 +CPU= 23.489
  22:30:31 STEP 2622 H=2622:00 +CPU= 35.885
  22:30:55 STEP 2623 H=2623:00 +CPU= 24.163
  22:31:19 STEP 2624 H=2624:00 +CPU= 24.010
  22:31:54 STEP 2625 H=2625:00 +CPU= 34.959
  22:32:18 STEP 2626 H=2626:00 +CPU= 23.890
  22:32:38 STEP 2627 H=2627:00 +CPU= 19.859
  22:33:15 STEP 2628 H=2628:00 +CPU= 37.281
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002628
  22:33:38 STEP 2629 H=2629:00 +CPU= 22.811
  22:34:01 STEP 2630 H=2630:00 +CPU= 23.298
  22:34:34 STEP 2631 H=2631:00 +CPU= 32.982
  22:34:54 STEP 2632 H=2632:00 +CPU= 19.641
  22:35:15 STEP 2633 H=2633:00 +CPU= 20.555
  22:35:49 STEP 2634 H=2634:00 +CPU= 33.951
  22:36:11 STEP 2635 H=2635:00 +CPU= 22.246
  22:36:35 STEP 2636 H=2636:00 +CPU= 23.931
  22:37:10 STEP 2637 H=2637:00 +CPU= 35.102
  22:37:33 STEP 2638 H=2638:00 +CPU= 23.322
  22:37:56 STEP 2639 H=2639:00 +CPU= 22.587
  22:38:36 STEP 2640 H=2640:00 +CPU= 40.205
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_109.zip
Uploading the intermediate file: upload_file_109.zip
  22:38:58 STEP 2641 H=2641:00 +CPU= 21.722
  22:39:21 STEP 2642 H=2642:00 +CPU= 23.169
  22:39:55 STEP 2643 H=2643:00 +CPU= 34.057
  22:40:17 STEP 2644 H=2644:00 +CPU= 21.555
  22:40:40 STEP 2645 H=2645:00 +CPU= 22.482
  22:41:19 STEP 2646 H=2646:00 +CPU= 38.716
  22:41:42 STEP 2647 H=2647:00 +CPU= 23.666
  22:42:06 STEP 2648 H=2648:00 +CPU= 23.835
  22:42:42 STEP 2649 H=2649:00 +CPU= 36.159
  22:43:08 STEP 2650 H=2650:00 +CPU= 25.572
  22:43:32 STEP 2651 H=2651:00 +CPU= 23.601
  22:44:10 STEP 2652 H=2652:00 +CPU= 38.878
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002652
  22:44:34 STEP 2653 H=2653:00 +CPU= 23.133
  22:44:57 STEP 2654 H=2654:00 +CPU= 23.214
  22:45:28 STEP 2655 H=2655:00 +CPU= 30.910
  22:45:51 STEP 2656 H=2656:00 +CPU= 22.985
  22:46:13 STEP 2657 H=2657:00 +CPU= 22.111
  22:46:49 STEP 2658 H=2658:00 +CPU= 35.977
  22:47:14 STEP 2659 H=2659:00 +CPU= 25.272
  22:47:38 STEP 2660 H=2660:00 +CPU= 23.707
  22:48:14 STEP 2661 H=2661:00 +CPU= 35.893
  22:48:37 STEP 2662 H=2662:00 +CPU= 23.080
  22:49:03 STEP 2663 H=2663:00 +CPU= 25.702
  22:49:46 STEP 2664 H=2664:00 +CPU= 43.493
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_110.zip
Uploading the intermediate file: upload_file_110.zip
  22:50:11 STEP 2665 H=2665:00 +CPU= 24.642
  22:50:35 STEP 2666 H=2666:00 +CPU= 23.966
  22:51:13 STEP 2667 H=2667:00 +CPU= 37.500
  22:51:39 STEP 2668 H=2668:00 +CPU= 25.791
  22:52:03 STEP 2669 H=2669:00 +CPU= 24.251
  22:52:40 STEP 2670 H=2670:00 +CPU= 37.358
  22:53:02 STEP 2671 H=2671:00 +CPU= 22.052
  22:53:28 STEP 2672 H=2672:00 +CPU= 25.685
  22:54:08 STEP 2673 H=2673:00 +CPU= 40.015
  22:54:37 STEP 2674 H=2674:00 +CPU= 28.626
  22:55:02 STEP 2675 H=2675:00 +CPU= 24.894
  22:55:48 STEP 2676 H=2676:00 +CPU= 46.076
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002676
  22:56:13 STEP 2677 H=2677:00 +CPU= 25.317
  22:56:42 STEP 2678 H=2678:00 +CPU= 29.142
  22:57:21 STEP 2679 H=2679:00 +CPU= 38.078
  22:57:44 STEP 2680 H=2680:00 +CPU= 23.532
  22:58:05 STEP 2681 H=2681:00 +CPU= 20.771
  22:58:43 STEP 2682 H=2682:00 +CPU= 38.105
  22:59:08 STEP 2683 H=2683:00 +CPU= 25.088
  22:59:34 STEP 2684 H=2684:00 +CPU= 25.581
  23:00:13 STEP 2685 H=2685:00 +CPU= 39.549
  23:00:37 STEP 2686 H=2686:00 +CPU= 23.972
  23:01:06 STEP 2687 H=2687:00 +CPU= 29.036
  23:01:52 STEP 2688 H=2688:00 +CPU= 45.581
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_111.zip
Uploading the intermediate file: upload_file_111.zip
  23:02:19 STEP 2689 H=2689:00 +CPU= 27.376
  23:02:44 STEP 2690 H=2690:00 +CPU= 25.169
  23:03:25 STEP 2691 H=2691:00 +CPU= 40.601
  23:03:52 STEP 2692 H=2692:00 +CPU= 26.284
  23:04:16 STEP 2693 H=2693:00 +CPU= 24.330
  23:04:55 STEP 2694 H=2694:00 +CPU= 39.255
  23:05:21 STEP 2695 H=2695:00 +CPU= 25.686
  23:05:44 STEP 2696 H=2696:00 +CPU= 22.807
  23:06:19 STEP 2697 H=2697:00 +CPU= 34.926
  23:06:41 STEP 2698 H=2698:00 +CPU= 22.681
  23:07:04 STEP 2699 H=2699:00 +CPU= 22.779
  23:07:43 STEP 2700 H=2700:00 +CPU= 38.581
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002700
  23:08:04 STEP 2701 H=2701:00 +CPU= 21.189
  23:08:28 STEP 2702 H=2702:00 +CPU= 24.235
  23:09:04 STEP 2703 H=2703:00 +CPU= 35.610
  23:09:25 STEP 2704 H=2704:00 +CPU= 21.130
  23:09:47 STEP 2705 H=2705:00 +CPU= 22.078
  23:10:21 STEP 2706 H=2706:00 +CPU= 34.038
  23:10:44 STEP 2707 H=2707:00 +CPU= 22.829
  23:11:06 STEP 2708 H=2708:00 +CPU= 22.292
  23:11:39 STEP 2709 H=2709:00 +CPU= 32.634
  23:12:04 STEP 2710 H=2710:00 +CPU= 25.116
  23:12:28 STEP 2711 H=2711:00 +CPU= 24.098
  23:13:09 STEP 2712 H=2712:00 +CPU= 41.350
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_112.zip
Uploading the intermediate file: upload_file_112.zip
  23:13:34 STEP 2713 H=2713:00 +CPU= 24.742
  23:13:59 STEP 2714 H=2714:00 +CPU= 25.379
  23:14:37 STEP 2715 H=2715:00 +CPU= 37.371
  23:15:02 STEP 2716 H=2716:00 +CPU= 25.343
  23:15:26 STEP 2717 H=2717:00 +CPU= 23.810
  23:16:05 STEP 2718 H=2718:00 +CPU= 38.773
  23:16:30 STEP 2719 H=2719:00 +CPU= 24.743
  23:16:55 STEP 2720 H=2720:00 +CPU= 25.223
  23:17:33 STEP 2721 H=2721:00 +CPU= 37.861
  23:17:56 STEP 2722 H=2722:00 +CPU= 23.523
  23:18:19 STEP 2723 H=2723:00 +CPU= 22.500
  23:19:01 STEP 2724 H=2724:00 +CPU= 41.265
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002724
  23:19:24 STEP 2725 H=2725:00 +CPU= 22.904
  23:19:46 STEP 2726 H=2726:00 +CPU= 22.112
  23:20:20 STEP 2727 H=2727:00 +CPU= 34.508
  23:20:45 STEP 2728 H=2728:00 +CPU= 24.246
  23:21:08 STEP 2729 H=2729:00 +CPU= 23.285
  23:21:47 STEP 2730 H=2730:00 +CPU= 38.827
  23:22:11 STEP 2731 H=2731:00 +CPU= 24.646
  23:22:35 STEP 2732 H=2732:00 +CPU= 23.010
  23:23:07 STEP 2733 H=2733:00 +CPU= 32.173
  23:23:30 STEP 2734 H=2734:00 +CPU= 23.080
  23:23:52 STEP 2735 H=2735:00 +CPU= 21.973
  23:24:34 STEP 2736 H=2736:00 +CPU= 42.325
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_113.zip
Uploading the intermediate file: upload_file_113.zip
  23:24:58 STEP 2737 H=2737:00 +CPU= 23.693
  23:25:22 STEP 2738 H=2738:00 +CPU= 23.656
  23:25:55 STEP 2739 H=2739:00 +CPU= 33.079
  23:26:17 STEP 2740 H=2740:00 +CPU= 21.858
  23:26:36 STEP 2741 H=2741:00 +CPU= 19.108
  23:27:09 STEP 2742 H=2742:00 +CPU= 33.358
  23:27:30 STEP 2743 H=2743:00 +CPU= 20.618
  23:27:53 STEP 2744 H=2744:00 +CPU= 22.935
  23:28:28 STEP 2745 H=2745:00 +CPU= 35.217
  23:28:51 STEP 2746 H=2746:00 +CPU= 22.572
  23:29:15 STEP 2747 H=2747:00 +CPU= 24.712
  23:29:53 STEP 2748 H=2748:00 +CPU= 37.084
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002748
  23:30:17 STEP 2749 H=2749:00 +CPU= 24.453
  23:30:41 STEP 2750 H=2750:00 +CPU= 23.875
  23:31:18 STEP 2751 H=2751:00 +CPU= 37.237
  23:31:42 STEP 2752 H=2752:00 +CPU= 23.674
  23:32:02 STEP 2753 H=2753:00 +CPU= 20.491
  23:32:37 STEP 2754 H=2754:00 +CPU= 34.269
  23:32:59 STEP 2755 H=2755:00 +CPU= 21.732
  23:33:22 STEP 2756 H=2756:00 +CPU= 23.514
  23:33:57 STEP 2757 H=2757:00 +CPU= 34.816
  23:34:19 STEP 2758 H=2758:00 +CPU= 21.819
  23:34:43 STEP 2759 H=2759:00 +CPU= 24.310
  23:35:23 STEP 2760 H=2760:00 +CPU= 40.023
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_114.zip
Uploading the intermediate file: upload_file_114.zip
  23:35:45 STEP 2761 H=2761:00 +CPU= 21.659
  23:36:09 STEP 2762 H=2762:00 +CPU= 24.437
  23:36:47 STEP 2763 H=2763:00 +CPU= 37.075
  23:37:10 STEP 2764 H=2764:00 +CPU= 22.883
  23:37:34 STEP 2765 H=2765:00 +CPU= 23.787
  23:38:08 STEP 2766 H=2766:00 +CPU= 34.766
  23:38:32 STEP 2767 H=2767:00 +CPU= 23.643
  23:38:54 STEP 2768 H=2768:00 +CPU= 22.424
  23:39:30 STEP 2769 H=2769:00 +CPU= 35.117
  23:39:54 STEP 2770 H=2770:00 +CPU= 24.115
  23:40:17 STEP 2771 H=2771:00 +CPU= 23.518
  23:40:56 STEP 2772 H=2772:00 +CPU= 39.221
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002772
  23:41:18 STEP 2773 H=2773:00 +CPU= 21.522
  23:41:41 STEP 2774 H=2774:00 +CPU= 23.249
  23:42:20 STEP 2775 H=2775:00 +CPU= 38.337
  23:42:44 STEP 2776 H=2776:00 +CPU= 24.312
  23:43:08 STEP 2777 H=2777:00 +CPU= 24.253
  23:43:48 STEP 2778 H=2778:00 +CPU= 39.975
  23:44:12 STEP 2779 H=2779:00 +CPU= 23.501
  23:44:36 STEP 2780 H=2780:00 +CPU= 24.159
  23:45:14 STEP 2781 H=2781:00 +CPU= 38.029
  23:45:35 STEP 2782 H=2782:00 +CPU= 20.726
  23:45:58 STEP 2783 H=2783:00 +CPU= 22.904
  23:46:42 STEP 2784 H=2784:00 +CPU= 44.343
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_115.zip
Uploading the intermediate file: upload_file_115.zip
  23:47:04 STEP 2785 H=2785:00 +CPU= 22.148
  23:47:31 STEP 2786 H=2786:00 +CPU= 26.414
  23:48:15 STEP 2787 H=2787:00 +CPU= 44.280
  23:48:38 STEP 2788 H=2788:00 +CPU= 23.265
  23:49:00 STEP 2789 H=2789:00 +CPU= 21.747
  23:49:33 STEP 2790 H=2790:00 +CPU= 33.255
  23:49:53 STEP 2791 H=2791:00 +CPU= 19.725
  23:50:17 STEP 2792 H=2792:00 +CPU= 23.771
  23:50:52 STEP 2793 H=2793:00 +CPU= 35.045
  23:51:13 STEP 2794 H=2794:00 +CPU= 21.649
  23:51:37 STEP 2795 H=2795:00 +CPU= 23.434
  23:52:19 STEP 2796 H=2796:00 +CPU= 41.617
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002796
  23:52:43 STEP 2797 H=2797:00 +CPU= 24.238
  23:53:08 STEP 2798 H=2798:00 +CPU= 25.071
  23:53:44 STEP 2799 H=2799:00 +CPU= 36.150
  23:54:09 STEP 2800 H=2800:00 +CPU= 24.866
  23:54:34 STEP 2801 H=2801:00 +CPU= 24.668
  23:55:14 STEP 2802 H=2802:00 +CPU= 40.254
  23:55:39 STEP 2803 H=2803:00 +CPU= 25.482
  23:56:01 STEP 2804 H=2804:00 +CPU= 21.927
  23:56:39 STEP 2805 H=2805:00 +CPU= 37.302
  23:57:03 STEP 2806 H=2806:00 +CPU= 24.621
  23:57:28 STEP 2807 H=2807:00 +CPU= 25.030
  23:58:11 STEP 2808 H=2808:00 +CPU= 42.519
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_116.zip
Uploading the intermediate file: upload_file_116.zip
  23:58:35 STEP 2809 H=2809:00 +CPU= 23.772
  23:58:57 STEP 2810 H=2810:00 +CPU= 22.352
  23:59:33 STEP 2811 H=2811:00 +CPU= 35.511
  23:59:55 STEP 2812 H=2812:00 +CPU= 22.748
  00:00:16 STEP 2813 H=2813:00 +CPU= 20.393
  00:00:48 STEP 2814 H=2814:00 +CPU= 31.929
  00:01:12 STEP 2815 H=2815:00 +CPU= 24.202
  00:01:34 STEP 2816 H=2816:00 +CPU= 21.959
  00:02:07 STEP 2817 H=2817:00 +CPU= 32.965
  00:02:30 STEP 2818 H=2818:00 +CPU= 22.795
  00:02:50 STEP 2819 H=2819:00 +CPU= 20.387
  00:03:29 STEP 2820 H=2820:00 +CPU= 38.554
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002820
  00:03:51 STEP 2821 H=2821:00 +CPU= 22.676
  00:04:17 STEP 2822 H=2822:00 +CPU= 26.002
  00:04:54 STEP 2823 H=2823:00 +CPU= 36.609
  00:05:17 STEP 2824 H=2824:00 +CPU= 23.017
  00:05:40 STEP 2825 H=2825:00 +CPU= 22.711
  00:06:16 STEP 2826 H=2826:00 +CPU= 35.976
  00:06:34 STEP 2827 H=2827:00 +CPU= 18.519
  00:06:56 STEP 2828 H=2828:00 +CPU= 21.932
  00:07:33 STEP 2829 H=2829:00 +CPU= 36.579
  00:07:55 STEP 2830 H=2830:00 +CPU= 21.876
  00:08:18 STEP 2831 H=2831:00 +CPU= 23.446
  00:08:57 STEP 2832 H=2832:00 +CPU= 38.231
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_117.zip
Uploading the intermediate file: upload_file_117.zip
  00:09:18 STEP 2833 H=2833:00 +CPU= 21.878
Uploading trickle at timestep: 10195200
  00:09:39 STEP 2834 H=2834:00 +CPU= 21.015
  00:10:15 STEP 2835 H=2835:00 +CPU= 35.609
  00:10:35 STEP 2836 H=2836:00 +CPU= 20.355
  00:10:58 STEP 2837 H=2837:00 +CPU= 22.277
  00:11:30 STEP 2838 H=2838:00 +CPU= 32.036
  00:11:49 STEP 2839 H=2839:00 +CPU= 19.266
  00:12:13 STEP 2840 H=2840:00 +CPU= 23.732
  00:12:50 STEP 2841 H=2841:00 +CPU= 37.315
  00:13:14 STEP 2842 H=2842:00 +CPU= 23.925
  00:13:36 STEP 2843 H=2843:00 +CPU= 21.563
  00:14:16 STEP 2844 H=2844:00 +CPU= 40.632
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002844
  00:14:40 STEP 2845 H=2845:00 +CPU= 24.061
  00:15:06 STEP 2846 H=2846:00 +CPU= 25.919
  00:15:46 STEP 2847 H=2847:00 +CPU= 40.055
  00:16:13 STEP 2848 H=2848:00 +CPU= 26.511
  00:16:36 STEP 2849 H=2849:00 +CPU= 23.335
  00:17:12 STEP 2850 H=2850:00 +CPU= 35.619
  00:17:35 STEP 2851 H=2851:00 +CPU= 22.552
  00:17:59 STEP 2852 H=2852:00 +CPU= 24.075
  00:18:36 STEP 2853 H=2853:00 +CPU= 36.991
  00:18:58 STEP 2854 H=2854:00 +CPU= 22.847
  00:19:22 STEP 2855 H=2855:00 +CPU= 23.916
  00:20:09 STEP 2856 H=2856:00 +CPU= 46.509
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_118.zip
Uploading the intermediate file: upload_file_118.zip
  00:20:34 STEP 2857 H=2857:00 +CPU= 24.764
  00:21:00 STEP 2858 H=2858:00 +CPU= 25.983
  00:21:38 STEP 2859 H=2859:00 +CPU= 38.144
  00:22:06 STEP 2860 H=2860:00 +CPU= 27.605
  00:22:30 STEP 2861 H=2861:00 +CPU= 24.390
  00:23:10 STEP 2862 H=2862:00 +CPU= 39.999
  00:23:36 STEP 2863 H=2863:00 +CPU= 25.463
  00:24:00 STEP 2864 H=2864:00 +CPU= 24.742
  00:24:38 STEP 2865 H=2865:00 +CPU= 37.649
  00:25:01 STEP 2866 H=2866:00 +CPU= 23.132
  00:25:25 STEP 2867 H=2867:00 +CPU= 23.747
  00:26:06 STEP 2868 H=2868:00 +CPU= 41.165
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002868
  00:26:28 STEP 2869 H=2869:00 +CPU= 21.754
  00:26:51 STEP 2870 H=2870:00 +CPU= 23.604
  00:27:32 STEP 2871 H=2871:00 +CPU= 40.714
  00:27:57 STEP 2872 H=2872:00 +CPU= 25.302
  00:28:21 STEP 2873 H=2873:00 +CPU= 23.711
  00:28:57 STEP 2874 H=2874:00 +CPU= 35.756
  00:29:20 STEP 2875 H=2875:00 +CPU= 23.218
  00:29:40 STEP 2876 H=2876:00 +CPU= 19.440
  00:30:13 STEP 2877 H=2877:00 +CPU= 33.446
  00:30:35 STEP 2878 H=2878:00 +CPU= 22.188
  00:30:57 STEP 2879 H=2879:00 +CPU= 22.227
  00:31:43 STEP 2880 H=2880:00 +CPU= 45.100
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_119.zip
Uploading the intermediate file: upload_file_119.zip
  00:32:09 STEP 2881 H=2881:00 +CPU= 26.351
  00:32:36 STEP 2882 H=2882:00 +CPU= 27.114
  00:33:19 STEP 2883 H=2883:00 +CPU= 43.173
  00:33:47 STEP 2884 H=2884:00 +CPU= 27.341
  00:34:12 STEP 2885 H=2885:00 +CPU= 25.703
  00:34:54 STEP 2886 H=2886:00 +CPU= 42.121
  00:35:22 STEP 2887 H=2887:00 +CPU= 27.367
  00:35:44 STEP 2888 H=2888:00 +CPU= 21.992
  00:36:20 STEP 2889 H=2889:00 +CPU= 35.864
  00:36:42 STEP 2890 H=2890:00 +CPU= 22.044
  00:37:05 STEP 2891 H=2891:00 +CPU= 22.821
  00:37:42 STEP 2892 H=2892:00 +CPU= 37.389
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002892
  00:38:05 STEP 2893 H=2893:00 +CPU= 22.667
  00:38:29 STEP 2894 H=2894:00 +CPU= 23.824
  00:39:05 STEP 2895 H=2895:00 +CPU= 36.068
  00:39:28 STEP 2896 H=2896:00 +CPU= 23.000
  00:39:49 STEP 2897 H=2897:00 +CPU= 21.186
  00:40:24 STEP 2898 H=2898:00 +CPU= 35.480
  00:40:48 STEP 2899 H=2899:00 +CPU= 23.373
  00:41:11 STEP 2900 H=2900:00 +CPU= 22.998
  00:41:46 STEP 2901 H=2901:00 +CPU= 35.665
  00:42:11 STEP 2902 H=2902:00 +CPU= 24.068
  00:42:32 STEP 2903 H=2903:00 +CPU= 21.163
  00:43:12 STEP 2904 H=2904:00 +CPU= 40.140
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_120.zip
Uploading the intermediate file: upload_file_120.zip
  00:43:32 STEP 2905 H=2905:00 +CPU= 20.396
  00:43:54 STEP 2906 H=2906:00 +CPU= 21.825
  00:44:26 STEP 2907 H=2907:00 +CPU= 32.049
  00:44:48 STEP 2908 H=2908:00 +CPU= 21.527
  00:45:10 STEP 2909 H=2909:00 +CPU= 22.454
  00:45:44 STEP 2910 H=2910:00 +CPU= 33.314
  00:46:04 STEP 2911 H=2911:00 +CPU= 19.892
  00:46:27 STEP 2912 H=2912:00 +CPU= 23.435
  00:46:59 STEP 2913 H=2913:00 +CPU= 31.848
  00:47:20 STEP 2914 H=2914:00 +CPU= 21.546
  00:47:41 STEP 2915 H=2915:00 +CPU= 21.095
  00:48:14 STEP 2916 H=2916:00 +CPU= 32.546
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002916
  00:48:34 STEP 2917 H=2917:00 +CPU= 19.518
  00:48:51 STEP 2918 H=2918:00 +CPU= 17.827
  00:49:25 STEP 2919 H=2919:00 +CPU= 33.999
  00:49:50 STEP 2920 H=2920:00 +CPU= 24.894
  00:50:16 STEP 2921 H=2921:00 +CPU= 25.685
  00:50:54 STEP 2922 H=2922:00 +CPU= 38.341
  00:51:20 STEP 2923 H=2923:00 +CPU= 25.197
  00:51:43 STEP 2924 H=2924:00 +CPU= 23.017
  00:52:19 STEP 2925 H=2925:00 +CPU= 36.786
  00:52:46 STEP 2926 H=2926:00 +CPU= 26.208
  00:53:13 STEP 2927 H=2927:00 +CPU= 27.521
  00:54:00 STEP 2928 H=2928:00 +CPU= 47.225
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_121.zip
Uploading the intermediate file: upload_file_121.zip
  00:54:25 STEP 2929 H=2929:00 +CPU= 24.775
  00:54:49 STEP 2930 H=2930:00 +CPU= 23.810
  00:55:23 STEP 2931 H=2931:00 +CPU= 34.204
  00:55:47 STEP 2932 H=2932:00 +CPU= 23.840
  00:56:11 STEP 2933 H=2933:00 +CPU= 24.329
  00:56:47 STEP 2934 H=2934:00 +CPU= 35.352
  00:57:10 STEP 2935 H=2935:00 +CPU= 22.962
  00:57:32 STEP 2936 H=2936:00 +CPU= 22.588
  00:58:11 STEP 2937 H=2937:00 +CPU= 38.581
  00:58:34 STEP 2938 H=2938:00 +CPU= 22.877
  00:58:53 STEP 2939 H=2939:00 +CPU= 19.278
  00:59:29 STEP 2940 H=2940:00 +CPU= 35.522
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002940
  00:59:50 STEP 2941 H=2941:00 +CPU= 21.773
  01:00:14 STEP 2942 H=2942:00 +CPU= 23.854
  01:00:54 STEP 2943 H=2943:00 +CPU= 39.285
  01:01:19 STEP 2944 H=2944:00 +CPU= 25.298
  01:01:41 STEP 2945 H=2945:00 +CPU= 22.404
  01:02:21 STEP 2946 H=2946:00 +CPU= 39.938
  01:02:47 STEP 2947 H=2947:00 +CPU= 25.278
  01:03:11 STEP 2948 H=2948:00 +CPU= 24.094
  01:03:48 STEP 2949 H=2949:00 +CPU= 37.486
  01:04:12 STEP 2950 H=2950:00 +CPU= 23.719
  01:04:36 STEP 2951 H=2951:00 +CPU= 23.732
  01:05:15 STEP 2952 H=2952:00 +CPU= 38.810
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client_40000/slots/11/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMGGhq0f+002952
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12194778/ICMSHhq0f+002928
Zipping up the final file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0134_2009050100_123_978_12194778_1_r2066043913_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
01:08:38 (961): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:32:05 1538236 22298006 oifs_43r3_ps_0134_2009050100_123_978_12194778_1 10,623,600 80,160 0.0075
09 May 2023 16:32:00 1538236 22298006 oifs_43r3_ps_0134_2009050100_123_978_12194778_1 10,195,200 76,786 0.0075
09 May 2023 16:31:38 1538236 22298006 oifs_43r3_ps_0134_2009050100_123_978_12194778_1 8,640,000 63,848 0.0074
09 May 2023 16:31:25 1538236 22298006 oifs_43r3_ps_0134_2009050100_123_978_12194778_1 7,776,000 56,568 0.0073
09 May 2023 16:31:12 1538236 22298006 oifs_43r3_ps_0134_2009050100_123_978_12194778_1 6,912,000 49,267 0.0071
09 May 2023 16:31:01 1538236 22298006 oifs_43r3_ps_0134_2009050100_123_978_12194778_1 6,048,000 42,629 0.0070
09 May 2023 16:30:55 1538236 22298006 oifs_43r3_ps_0134_2009050100_123_978_12194778_1 5,184,000 36,421 0.0070
09 May 2023 16:30:44 1538236 22298006 oifs_43r3_ps_0134_2009050100_123_978_12194778_1 4,320,000 30,256 0.0070
09 May 2023 16:30:32 1538236 22298006 oifs_43r3_ps_0134_2009050100_123_978_12194778_1 3,456,000 24,112 0.0070
09 May 2023 16:30:20 1538236 22298006 oifs_43r3_ps_0134_2009050100_123_978_12194778_1 2,592,000 17,987 0.0069
09 May 2023 16:30:14 1538236 22298006 oifs_43r3_ps_0134_2009050100_123_978_12194778_1 1,728,000 11,812 0.0068
09 May 2023 16:30:01 1538236 22298006 oifs_43r3_ps_0134_2009050100_123_978_12194778_1 864,000 5,638 0.0065


©2024 climateprediction.net