climateprediction.net home page
Task 21670795

Task 21670795

Name wah2_sam50_a0o7_198912_24_815_011848811_0
Workunit 11848811
Created 20 May 2019, 10:54:21 UTC
Sent 25 May 2019, 15:11:41 UTC
Report deadline 6 May 2020, 20:31:41 UTC
Received 21 Jun 2019, 12:37:17 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1291402
Run time 4 days 21 hours 57 min 59 sec
CPU time 4 days 20 hours 17 min 57 sec
Validate state Invalid
Credit 12,178.71
Device peak FLOPS 3.40 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 223.93 MB
Peak swap size 189.49 MB
Peak disk usage 378.66 MB
Stderr 
<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
Adresa  - exit code 9 (0x9)</message>
<stderr_txt>


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x776EE084 read attempt to address 0x3060FFFF

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 05/26/19 00:35:31
Install Directory : 
Data Directory    : D:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( D:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\ProgramData\BOINC\slots\0;D:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 D:\ProgramData\BOINC\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000776c0000 0000000000180000 C:\Windows\SysWOW64\ntdll.dll (6.1.7601.24441) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 0000000074dd0000 0000000000110000 C:\Windows\syswow64\kernel32.dll (6.1.7601.24441) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 0000000077150000 0000000000047000 C:\Windows\syswow64\KERNELBASE.dll (6.1.7601.24441) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 0000000076fb0000 0000000000100000 C:\Windows\syswow64\USER32.dll (6.1.7601.23594) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 0000000076ad0000 0000000000090000 C:\Windows\syswow64\GDI32.dll (6.1.7601.24440) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 6.1.7601.24440 (win7sp1_ldr.190416-0600)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.24440

ModLoad: 00000000756f0000 000000000000a000 C:\Windows\syswow64\LPK.dll (6.1.7601.24439) (-exported- Symbols Loaded)
    Linked PDB Filename   : wlpk.pdb
    File Version          : 6.1.7601.24439 (win7sp1_ldr.190413-2027)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.24439

ModLoad: 00000000770b0000 000000000009d000 C:\Windows\syswow64\USP10.dll (1.626.7601.24440) (-exported- Symbols Loaded)
    Linked PDB Filename   : usp10.pdb
    File Version          : 1.0626.7601.24440 (win7sp1_ldr.190416-0600)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Uniscribe Unicode script processor
    Product Version       : 1.0626.7601.24440

ModLoad: 0000000075230000 00000000000ac000 C:\Windows\syswow64\msvcrt.dll (7.0.7601.17744) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.7601.17744 (win7sp1_gdr.111215-1535)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.7601.17744

ModLoad: 0000000076c00000 00000000000a1000 C:\Windows\syswow64\ADVAPI32.dll (6.1.7601.24441) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 6.1.7601.24441 (win7sp1_ldr.190418-1735)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.24441

ModLoad: 0000000075910000 0000000000019000 C:\Windows\SysWOW64\sechost.dll (6.1.7601.18869) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 00000000771b0000 00000000000f0000 C:\Windows\syswow64\RPCRT4.dll (6.1.7601.24441) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 0000000074d70000 0000000000060000 C:\Windows\syswow64\SspiCli.dll (6.1.7601.24441) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 6.1.7601.24441 (win7sp1_ldr.190418-1735)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.24441

ModLoad: 0000000074d60000 000000000000c000 C:\Windows\syswow64\CRYPTBASE.dll (6.1.7601.24441) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 6.1.7601.24441 (win7sp1_ldr.190418-1735)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.24441

ModLoad: 0000000075340000 0000000000060000 C:\Windows\system32\IMM32.DLL (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 0000000075160000 00000000000cd000 C:\Windows\syswow64\MSCTF.dll (6.1.7601.23915) (-exported- Symbols Loaded)
    Linked PDB Filename   : msctf.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 00000000732d0000 0000000000021000 C:\Windows\system32\ntmarta.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntmarta.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 00000000752f0000 0000000000045000 C:\Windows\syswow64\WLDAP32.dll (6.1.7601.23889) (-exported- Symbols Loaded)
    Linked PDB Filename   : wldap32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 0000000070fa0000 000000000004c000 C:\Windows\system32\apphelp.dll (6.1.7601.19050) (-exported- Symbols Loaded)
    Linked PDB Filename   : apphelp.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 000000006f640000 00000000000eb000 C:\Windows\system32\dbghelp.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 0000000073300000 0000000000009000 C:\Windows\system32\version.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 21332, Write: 0, Other 79659

- I/O Transfers Counters -
Read: 0, Write: 2882, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 195304, QuotaPeakPagedPoolUsage: 195304
QuotaNonPagedPoolUsage: 7688, QuotaPeakNonPagedPoolUsage: 7768

- Virtual Memory Usage -
VirtualSize: 106225664, PeakVirtualSize: 106225664

- Pagefile Usage -
PagefileUsage: 2363392, PeakPagefileUsage: 3080192

- Working Set Size -
WorkingSetSize: 78348288, PeakWorkingSetSize: 78868480, PageFaultCount: 20346

*** Dump of thread ID 5584 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2245693.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x776EE084 read attempt to address 0x3060FFFF

- Registers -
eax=3060fff8 ebx=30610000 ecx=102647fa edx=003de8b8 esi=00280000 edi=00000000
eip=776ee084 esp=058cfe5c ebp=058cfe6c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246

- Callstack -
ChildEBP RetAddr  Args to Child
058cfe6c 74de14ad 00280000 00000000 30610000 058cfe94 ntdll!RtlFreeHeap+0x0 
058cfe80 00482c3b 00280000 00000000 30610000 058cfeac kernel32!HeapFree+0x0 
058cfe94 00448311 30610000 00000000 011e0020 80000000 wah2_8.24_windows_intelx86!+0x0 
058cfeac 0044fa68 011e0020 00000000 0051d6b0 058cfefc wah2_8.24_windows_intelx86!+0x0 
058cfec0 004856fc 00000015 15aab906 00000001 00000006 wah2_8.24_windows_intelx86!+0x0 
058cfefc 74e87739 00000006 15ab6674 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
058cff88 74de343d 00000006 058cffd4 776f9802 80000006 kernel32!CtrlRoutine+0x0 
058cff94 776f9802 80000006 7306b5cb 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
058cffd4 776f97d5 74e87584 80000006 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
058cffec 00000000 74e87584 80000006 00000000 4d500cdc ntdll!RtlInitializeExceptionChain+0x0 

*** Dump of thread ID 2748 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 11718750.000000, User Time: 68125000.000000, Wait Time: 2245686.000000

- Registers -
eax=00000000 ebx=00000050 ecx=00000000 edx=00000000 esi=0018ed80 edi=00000000
eip=776dfdc1 esp=0018ed3c ebp=0018eda4
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
0018eda4 77164667 000000c8 00000000 0018edc8 00468ec1 ntdll!NtDelayExecution+0x0 
0018edb4 00468ec1 000000c8 000000c8 00000000 0018ee00 KERNELBASE!Sleep+0x0 
0018edc8 00407a89 a0000000 027f9999 00000000 004d0c50 wah2_8.24_windows_intelx86!+0x0 
0018ee00 004092c5 00000000 103ebab2 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0018fd48 004094b7 0018fef8 00409535 004d1370 0018fd68 wah2_8.24_windows_intelx86!+0x0 
0018fd50 00409535 004d1370 0018fd68 00681ff8 0018fd68 wah2_8.24_windows_intelx86!+0x0 
0018fef8 00485c7b 00400000 00000000 00682036 0000000a wah2_8.24_windows_intelx86!+0x0 
0018ff88 74de343d 7efde000 0018ffd4 776f9802 7efde000 wah2_8.24_windows_intelx86!+0x0 
0018ff94 776f9802 7efde000 7692b5cb 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
0018ffd4 776f97d5 00485cce 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
0018ffec 00000000 00485cce 7efde000 00000000 78746341 ntdll!RtlInitializeExceptionChain+0x0 

*** Dump of thread ID 1332 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2245688.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=011dfd24 edi=00000000
eip=776dfdc1 esp=011dfce0 ebp=011dfd48
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
011dfd48 77164667 00000064 00000000 011dff88 00478b44 ntdll!NtDelayExecution+0x0 
011dfd58 00478b44 00000064 00000000 00000000 00000000 KERNELBASE!Sleep+0x0 
011dff88 74de343d 00000000 011dffd4 776f9802 00000000 wah2_8.24_windows_intelx86!+0x0 
011dff94 776f9802 00000000 7797b5cb 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
011dffd4 776f97d5 00478b20 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
011dffec 00000000 00478b20 00000000 00000000 32686177 ntdll!RtlInitializeExceptionChain+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
No Process Handle
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6776, selfPID=6776, iMonCtr=1
No Process Handle
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6776, selfPID=3552, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4316, selfPID=4316, iMonCtr=1
No Process Handle
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4316, selfPID=2000, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3524, selfPID=5992, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3524, selfPID=3524, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6116, selfPID=5756, iMonCtr=1
No Process Handle
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6116, selfPID=6116, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5948, selfPID=4052, iMonCtr=1
No Process Handle
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5948, selfPID=5948, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5128, selfPID=5128, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5128, selfPID=1744, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5848, selfPID=5848, iMonCtr=1
No Process Handle
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5848, selfPID=3504, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3200, selfPID=5188, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3200, selfPID=3200, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5644, selfPID=5644, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5644, selfPID=6036, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5348, selfPID=5488, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5348, selfPID=5348, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5024, selfPID=6108, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5024, selfPID=5024, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6392, selfPID=6392, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6392, selfPID=6500, iMonCtr=1

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
20 Jun 2019 14:18:51 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 184,619 404,175 2.1892
18 Jun 2019 18:52:59 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 173,099 379,776 2.1940
17 Jun 2019 17:05:45 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 161,579 355,237 2.1985
16 Jun 2019 11:30:04 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 150,059 330,898 2.2051
06 Jun 2019 14:57:47 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 138,539 306,283 2.2108
06 Jun 2019 13:55:31 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 127,019 279,813 2.2029
06 Jun 2019 13:55:31 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 115,499 254,012 2.1993
06 Jun 2019 13:55:31 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 103,979 228,124 2.1939
06 Jun 2019 13:55:31 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 92,459 202,556 2.1908
06 Jun 2019 13:55:31 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 80,939 177,036 2.1873
06 Jun 2019 13:55:31 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 69,419 151,656 2.1846
06 Jun 2019 13:55:31 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 57,899 126,224 2.1801
29 May 2019 08:19:07 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 46,379 101,000 2.1777
27 May 2019 21:47:38 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 34,859 75,896 2.1772
26 May 2019 20:21:01 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 23,339 50,881 2.1801
26 May 2019 13:19:11 1291402 21670795 wah2_sam50_a0o7_198912_24_815_011848811_0 11,819 25,895 2.1910


©2024 climateprediction.net