climateprediction.net home page
Task 16658209

Task 16658209

Name hadcm3n_768y_1980_40_008419381_3
Workunit 8570237
Created 30 May 2014, 19:12:53 UTC
Sent 30 May 2014, 19:13:00 UTC
Report deadline 30 Aug 2014, 2:40:11 UTC
Received 14 Jun 2014, 16:45:59 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status -2 (0xFFFFFFFE) Unknown error code
Computer ID 1385673
Run time 1 days 16 hours 50 min 20 sec
CPU time 1 days 11 hours 0 min 36 sec
Validate state Invalid
Credit 622.08
Device peak FLOPS 2.65 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
windows_intelx86
Stderr 
<core_client_version>7.2.42</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code -2 (0xfffffffe)
</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
23:45:45 (6504): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
03:34:16 (8248): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:37:29 (4184): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:45:44 (8592): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:11:50 (10388): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 22
WARNING: BOINC Windows Runtime Debugger has been disabled.
04:14:42 (10056): start_timer_thread(): CreateThread() failed, errno 0


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x751CC42D

BOINC Windows Runtime Debugger not configured, terminating application...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6728, iMonCtr=1
Model crash detected, will try to restart...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x751CC42D

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 06/14/14 04:15:03
Install Directory : C:\Program Files (x86)\BOINC\
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\Program Files (x86)\BOINC\\dbghelp.dll ): GetLastError = 193
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\Program Files (x86)\BOINC\\symsrv.dll ): GetLastError = 193
LoadLibraryA( symsrv.dll ): GetLastError = 193
LoadLibraryA( C:\Program Files (x86)\BOINC\\srcsrv.dll ): GetLastError = 193
LoadLibraryA( srcsrv.dll ): GetLastError = 193
LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_768y_1980_40_008419381;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 775f0000 00180000 C:\Windows\SysWOW64\ntdll.dll (6.1.7601.18247) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 76da0000 00110000 C:\Windows\syswow64\kernel32.dll (6.1.7601.18409) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 751c0000 00047000 C:\Windows\syswow64\KERNELBASE.dll (6.1.7601.18409) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 770f0000 00100000 C:\Windows\syswow64\USER32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 759e0000 00090000 C:\Windows\syswow64\GDI32.dll (6.1.7601.18275) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 6.1.7601.18275 (win7sp1_gdr.131002-1533)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18275

ModLoad: 751b0000 0000a000 C:\Windows\syswow64\LPK.dll (6.1.7601.18177) (-exported- Symbols Loaded)
    Linked PDB Filename   : wlpk.pdb
    File Version          : 6.1.7601.18177 (win7sp1_gdr.130605-1534)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18177

ModLoad: 758e0000 0009d000 C:\Windows\syswow64\USP10.dll (1.626.7601.18454) (-exported- Symbols Loaded)
    Linked PDB Filename   : usp10.pdb
    File Version          : 1.0626.7601.18454 (win7sp1_gdr.140424-1533)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Uniscribe Unicode script processor
    Product Version       : 1.0626.7601.18454

ModLoad: 75e20000 000ac000 C:\Windows\syswow64\msvcrt.dll (7.0.7601.17744) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.7601.17744 (win7sp1_gdr.111215-1535)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.7601.17744

ModLoad: 75f60000 000a0000 C:\Windows\syswow64\ADVAPI32.dll (6.1.7601.18247) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 76130000 00019000 C:\Windows\SysWOW64\sechost.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 76000000 000f0000 C:\Windows\syswow64\RPCRT4.dll (6.1.7601.18205) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74ff0000 00060000 C:\Windows\syswow64\SspiCli.dll (6.1.7601.18443) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 6.1.7601.18443 (win7sp1_gdr.140411-1533)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18443

ModLoad: 74fe0000 0000c000 C:\Windows\syswow64\CRYPTBASE.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 68db0000 000c8000 C:\Windows\system32\OPENGL32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : opengl32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 6c080000 00022000 C:\Windows\system32\GLU32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : glu32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 6bf90000 000e7000 C:\Windows\system32\DDRAW.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : ddraw.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74cf0000 00006000 C:\Windows\system32\DCIMAN32.dll (6.1.7601.18177) (-exported- Symbols Loaded)
    Linked PDB Filename   : dciman32.pdb
    File Version          : 6.1.7601.18177 (win7sp1_gdr.130605-1534)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18177

ModLoad: 75c50000 0019d000 C:\Windows\syswow64\SETUPAPI.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : setupapi.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75f30000 00027000 C:\Windows\syswow64\CFGMGR32.dll (6.1.7601.17621) (-exported- Symbols Loaded)
    Linked PDB Filename   : cfgmgr32.pdb
    File Version          : 6.1.7601.17621 (win7sp1_gdr.110523-2108)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17621

Get Product Name Failed.
ModLoad: 75480000 0008f000 C:\Windows\syswow64\OLEAUT32.dll (6.1.7601.17676) (-exported- Symbols Loaded)
    Linked PDB Filename   : oleaut32.pdb
    File Version          : 6.1.7601.17676
    Company Name          : Microsoft Corporation
    Product Name          : 
    Product Version       : 6.1.7601.17676

ModLoad: 76eb0000 0015c000 C:\Windows\syswow64\ole32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : ole32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 76110000 00012000 C:\Windows\syswow64\DEVOBJ.dll (6.1.7601.17621) (-exported- Symbols Loaded)
    Linked PDB Filename   : devobj.pdb
    File Version          : 6.1.7601.17621 (win7sp1_gdr.110523-2108)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17621

ModLoad: 72450000 00013000 C:\Windows\system32\dwmapi.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : dwmapi.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75180000 0002b000 C:\Windows\syswow64\imagehlp.dll (6.1.7601.18288) (-exported- Symbols Loaded)
    Linked PDB Filename   : imagehlp.pdb
    File Version          : 6.1.7601.18288 (win7sp1_gdr.131018-1533)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18288

ModLoad: 75ed0000 00060000 C:\Windows\system32\IMM32.DLL (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 75210000 000cc000 C:\Windows\syswow64\MSCTF.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : msctf.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 71980000 000eb000 C:\Windows\system32\dbghelp.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 74f10000 00009000 C:\Windows\system32\version.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 0, Write: 0, Other 0

- I/O Transfers Counters -
Read: 0, Write: 0, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0
QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0

- Virtual Memory Usage -
VirtualSize: 0, PeakVirtualSize: 0

- Pagefile Usage -
PagefileUsage: 0, PeakPagefileUsage: 0

- Working Set Size -
WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0

*** Dump of thread ID 10756 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x751CC42D

- Registers -
eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe
eip=751cc42d esp=1132f9c4 ebp=1132fa14
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000212

- Callstack -
ChildEBP RetAddr  Args to Child
1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 KERNELBASE!RaiseException+0x0 
1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fad8 00408130 1132faec 11484b89 1132faec 11484b11 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff88 76db338a 7efde000 1132ffd4 77629f72 7efde000 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff94 77629f72 7efde000 67bc81d1 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
1132ffd4 77629f45 007bde7f 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
1132ffec 00000000 007bde7f 7efde000 00000000 4b35871b ntdll!RtlInitializeExceptionChain+0x0 

*** Dump of thread ID 9032 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=33d1ff48 edi=00000000
eip=7760fd91 esp=33d1ff04 ebp=33d1ff6c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
33d1ff6c 751d44a5 00000064 00000000 33d1ff88 00749924 ntdll!ZwDelayExecution+0x0 
33d1ff7c 00749924 00000064 33d1ff94 76db338a 00000000 KERNELBASE!Sleep+0x0 
33d1ff88 76db338a 00000000 33d1ffd4 77629f72 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 
33d1ff94 77629f72 00000000 455f81d1 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
33d1ffd4 77629f45 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
33d1ffec 00000000 00749910 00000000 00000000 34230000 ntdll!RtlInitializeExceptionChain+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6728, iMonCtr=1
Model crash detected, will try to restart...
diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 22
WARNING: BOINC Windows Runtime Debugger has been disabled.
04:15:28 (9460): start_timer_thread(): CreateThread() failed, errno 0


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x751CC42D

BOINC Windows Runtime Debugger not configured, terminating application...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6728, iMonCtr=1
Model crash detected, will try to restart...
Could not launch model process. Last Error=1455
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
13 Jun 2014 22:23:10 926174 16658209 hadcm3n_768y_1980_40_008419381_3 51,840 100,116 1.9313
13 Jun 2014 06:41:41 926174 16658209 hadcm3n_768y_1980_40_008419381_3 25,920 50,015 1.9296


©2024 climateprediction.net