Hide Forgot
Description of problem: C++ example "client.exe" for Windows sometimes crashes with error message: "The instruction at "0x7c91168b" referenced memory at "0x12000113". The memory could not be "read"." This issue occurred on Windows XP-x86 and broker on Rhel5-x86_64. Version-Release number of selected component (if applicable): qpid-cpp-win-3.0.0.4-1 How reproducible: less than 10% Steps to Reproduce: 1. Unpack the package. 2. Build and run the C++ examples. Actual results: The "client.exe" example sometimes crashes. Expected results: All example applications should run without exceptions. Additional info: I saw this issue on packages qpid-cpp-win-3.0.0.3-1 too and more often. Except the "client.exe" also "map_sender.exe", "map_receiver.exe" and "drain.exe" crashed with the same problem.
I saw the issue only on Windows XP-x86. On packages "qpid-cpp-win-3.0.0.3-1" I saw this issue very often on "map_sender.exe" and "map_receiver.exe" - in about 70% cases. "drain.exe" and "client.exe"crashed on the older packages only once.
(In reply to comment #29) > Thank you for the additional details. > > Is it the case that: > > A) client.exe always fails before sending any messages > B) client.exe always fails after the last response has been received. > C) not consistently A or B > > Also, were your tests run in a VM or on bare metal? > > Thanks. Hi Cliff, all the tests run in Virtual Machines. I will answer your second question as soon as possible; I am working on it.
Today I saw this issue once on "drain.exe" and twice on "map_sender.exe" examples on WindowsXP-x86 machine. Your option "C" is right: - "drain.exe" crashed after receiving messages - "map_sender.exe" crashed once before sending messages and once after sending messages Will you look at output from WinDBG tool for case when the "map_sender.exe" crashed before sending messages, please? *** wait with pending attach DBGHELP: Symbol Search Path: http://msdl.microsoft.com/download/symbols;c:\qpid_vs2010\bin\debug\map_sender.pdb WARNING: Non-directory path: 'C:\qpid_VS2010\bin\Debug\map_sender.pdb' DBGHELP: Symbol Search Path: http://msdl.microsoft.com/download/symbols;c:\qpid_vs2010\bin\debug\map_sender.pdb DBGHELP: Symbol Search Path: http://msdl.microsoft.com/download/symbols;c:\qpid_vs2010\bin\debug\map_sender.pdb Symbol search path is: http://msdl.microsoft.com/download/symbols;C:\qpid_VS2010\bin\Debug\map_sender.pdb Executable search path is: DBGHELP: SharedUserData - virtual symbol module ModLoad: 00400000 0041b000 c:\qpid_VS2010\bin\Debug\map_sender.exe ModLoad: 7c900000 7c9b2000 C:\WINDOWS\system32\ntdll.dll ModLoad: 7c800000 7c8f6000 C:\WINDOWS\system32\kernel32.dll ModLoad: 10000000 10121000 c:\qpid_VS2010\bin\Debug\qpidmessagingd.dll ModLoad: 00250000 003ce000 c:\qpid_VS2010\bin\Debug\qpidclientd.dll ModLoad: 00db0000 01106000 c:\qpid_VS2010\bin\Debug\qpidcommond.dll ModLoad: 77dd0000 77e6b000 C:\WINDOWS\system32\ADVAPI32.dll ModLoad: 77e70000 77f03000 C:\WINDOWS\system32\RPCRT4.dll ModLoad: 77fe0000 77ff1000 C:\WINDOWS\system32\Secur32.dll ModLoad: 01110000 01164000 c:\qpid_VS2010\bin\Debug\qpidtypesd.dll ModLoad: 10480000 10537000 C:\WINDOWS\system32\MSVCP100D.dll ModLoad: 10200000 10372000 C:\WINDOWS\system32\MSVCR100D.dll ModLoad: 01170000 01179000 c:\qpid_VS2010\bin\Debug\boost_system-vc100-mt-gd-1_47.dll ModLoad: 71ab0000 71ac7000 C:\WINDOWS\system32\WS2_32.dll ModLoad: 77c10000 77c68000 C:\WINDOWS\system32\msvcrt.dll ModLoad: 71aa0000 71aa8000 C:\WINDOWS\system32\WS2HELP.dll ModLoad: 01180000 0120c000 c:\qpid_VS2010\bin\Debug\boost_program_options-vc100-mt-gd-1_47.dll ModLoad: 01210000 01223000 c:\qpid_VS2010\bin\Debug\boost_thread-vc100-mt-gd-1_47.dll ModLoad: 01230000 01246000 c:\qpid_VS2010\bin\Debug\boost_date_time-vc100-mt-gd-1_47.dll ModLoad: 01250000 01285000 c:\qpid_VS2010\bin\Debug\boost_filesystem-vc100-mt-gd-1_47.dll ModLoad: 71a50000 71a8f000 C:\WINDOWS\system32\mswsock.dll ModLoad: 662b0000 66308000 C:\WINDOWS\system32\hnetcfg.dll ModLoad: 77f10000 77f59000 C:\WINDOWS\system32\GDI32.dll ModLoad: 7e410000 7e4a1000 C:\WINDOWS\system32\USER32.dll ModLoad: 76390000 763ad000 C:\WINDOWS\system32\IMM32.DLL ModLoad: 58d40000 58d47000 C:\WINDOWS\System32\wship6.dll ModLoad: 71a90000 71a98000 C:\WINDOWS\System32\wshtcpip.dll Break-in sent, waiting 30 seconds... WARNING: Break-in timed out, suspending. This is usually caused by another thread holding the loader lock (30a0.3040): Wake debugger - code 80000007 (first chance) eax=004156a8 ebx=00000000 ecx=0040b480 edx=00daffb8 esi=7c885780 edi=00000000 eip=7c90e514 esp=00daf4a0 ebp=00daf724 iopl=0 nv up ei pl zr na pe nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246 DBGHELP: ntdll - public symbols C:\Program Files\Debugging Tools for Windows (x86)\sym\ntdll.pdb\CEFC0863B1F84130A11E0F54180CD21A2\ntdll.pdb ntdll!KiFastSystemCallRet: 7c90e514 c3 ret DBGHELP: kernel32 - public symbols C:\Program Files\Debugging Tools for Windows (x86)\sym\kernel32.pdb\072FF0EB54D24DFAAE9D13885486EE092\kernel32.pdb DBGHELP: MSVCR100D - private symbols & lines C:\Program Files\Debugging Tools for Windows (x86)\sym\msvcr100d.i386.pdb\9E26F0DD2C2543B99754DB8B9BA5F8681\msvcr100d.i386.pdb DBGHELP: MSVCP100D - private symbols & lines C:\Program Files\Debugging Tools for Windows (x86)\sym\msvcp100d.i386.pdb\27DBD64D923C447594980A3F48708A7D1\msvcp100d.i386.pdb # ChildEBP RetAddr 00 00daf49c 7c90d9ca ntdll!KiFastSystemCallRet (FPO: [0,0,0]) 01 00daf4a0 7c8645fd ntdll!NtRaiseHardError+0xc (FPO: [6,0,0]) 02 00daf724 7c8438fa kernel32!UnhandledExceptionFilter+0x628 (FPO: [Non-Fpo]) 03 00daf72c 7c839b39 kernel32!BaseProcessStart+0x39 (FPO: [Non-Fpo]) 04 00daf754 7c9032a8 kernel32!_except_handler3+0x61 (FPO: [Uses EBP] [3,0,7]) 05 00daf778 7c90327a ntdll!ExecuteHandler2+0x26 06 00daf828 7c90e48a ntdll!ExecuteHandler+0x24 07 00daf828 7c910a19 ntdll!KiUserExceptionDispatcher+0xe (FPO: [2,0,0]) (CONTEXT @ 00daf85c) 08 00dafb34 7c91084c ntdll!RtlpCoalesceFreeBlocks+0x36e (FPO: [Non-Fpo]) 09 00dafc08 10307359 ntdll!RtlFreeHeap+0x2e9 (FPO: [Non-Fpo]) 0a 00dafc24 10316e5e MSVCR100D!_free_base(void * pBlock = 0x01341d58)+0x29 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\free.c @ 50] 0b 00dafc34 10316970 MSVCR100D!_free_dbg_nolock(void * pUserData = 0x01341d78, int nBlockUse = 0n2)+0x4ae (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\dbgheap.c @ 1431] 0c 00dafc6c 10319090 MSVCR100D!_free_dbg(void * pUserData = 0x01341d78, int nBlockUse = 0n1)+0x50 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\dbgheap.c @ 1265] 0d 00dafc7c 104e2b0f MSVCR100D!free(void * pUserData = 0x01341d78)+0x10 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\dbgfree.c @ 49] 0e 00dafc88 104e2abd MSVCP100D!std::_DebugHeapDelete<std::_Fac_node>(struct std::_Fac_node * _Ptr = 0x01341d78)+0x1f (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\xdebug @ 62] 0f 00dafcac 104e2a4e MSVCP100D!_Fac_tidy(void)+0x5d (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\locale0.cpp @ 41] 10 00dafcb8 1050119f MSVCP100D!std::_Fac_tidy_reg_t::~_Fac_tidy_reg_t(void)+0xe (FPO: [Non-Fpo]) (CONV: thiscall) [f:\dd\vctools\crt_bld\self_x86\crt\src\locale0.cpp @ 48] 11 00dafcc0 104f5354 MSVCP100D!std::`dynamic atexit destructor for '_Fac_tidy_reg''(void)+0xf (FPO: [Non-Fpo]) (CONV: cdecl) 12 00dafcfc 104f5580 MSVCP100D!_CRT_INIT(void * hDllHandle = 0x10480000, unsigned long dwReason = 0, void * lpreserved = 0x00000001)+0x254 (FPO: [Non-Fpo]) (CONV: stdcall) [f:\dd\vctools\crt_bld\self_x86\crt\src\crtdll.c @ 415] 13 00dafd40 104f5441 MSVCP100D!__DllMainCRTStartup(void * hDllHandle = 0x10480000, unsigned long dwReason = 0, void * lpreserved = 0x00000001)+0x130 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\crtdll.c @ 526] 14 00dafd54 7c90118a MSVCP100D!_DllMainCRTStartup(void * hDllHandle = 0x10480000, unsigned long dwReason = 0, void * lpreserved = 0x00000001)+0x21 (FPO: [Non-Fpo]) (CONV: stdcall) [f:\dd\vctools\crt_bld\self_x86\crt\src\crtdll.c @ 476] 15 00dafd74 7c9224ca ntdll!LdrpCallInitRoutine+0x14 16 00dafdf8 7c81caae ntdll!LdrShutdownProcess+0x14f (FPO: [Non-Fpo]) 17 00dafeec 7c81cb26 kernel32!_ExitProcess+0x42 (FPO: [Non-Fpo]) 18 00daff00 102485bb kernel32!ExitProcess+0x14 19 00daff0c 10248467 MSVCR100D!__crtExitProcess(int status = 0n0)+0x1b (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\crt0dat.c @ 709] 1a 00daff5c 102480b2 MSVCR100D!doexit(int code = 0n0, int quick = 0n0, int retcaller = 0n0)+0x1c7 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\crt0dat.c @ 621] 1b 00daff70 0040ba7c MSVCR100D!exit(int code = 0n0)+0x12 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\crt0dat.c @ 393] 1c 00daffb8 0040b88f map_sender!__tmainCRTStartup(void)+0x1dc (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c @ 568] 1d 00daffc0 7c817077 map_sender!mainCRTStartup(void)+0xf (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c @ 371]
Another crash: "client.exe" on the same machine XP-x86 output from WinDBG tool: *** wait with pending attach Symbol search path is: *** Invalid *** **************************************************************************** * Symbol loading may be unreliable without a symbol search path. * * Use .symfix to have the debugger choose a symbol path. * * After setting your symbol path, use .reload to refresh symbol locations. * **************************************************************************** Executable search path is: ModLoad: 00400000 00411000 c:\qpid_VS2010\bin\Debug\client.exe ModLoad: 7c900000 7c9b2000 C:\WINDOWS\system32\ntdll.dll ModLoad: 7c800000 7c8f6000 C:\WINDOWS\system32\kernel32.dll ModLoad: 10000000 10121000 c:\qpid_VS2010\bin\Debug\qpidmessagingd.dll ModLoad: 00250000 003ce000 c:\qpid_VS2010\bin\Debug\qpidclientd.dll ModLoad: 00db0000 01106000 c:\qpid_VS2010\bin\Debug\qpidcommond.dll ModLoad: 77dd0000 77e6b000 C:\WINDOWS\system32\ADVAPI32.dll ModLoad: 77e70000 77f03000 C:\WINDOWS\system32\RPCRT4.dll ModLoad: 77fe0000 77ff1000 C:\WINDOWS\system32\Secur32.dll ModLoad: 01110000 01164000 c:\qpid_VS2010\bin\Debug\qpidtypesd.dll ModLoad: 10480000 10537000 C:\WINDOWS\system32\MSVCP100D.dll ModLoad: 10200000 10372000 C:\WINDOWS\system32\MSVCR100D.dll ModLoad: 01170000 01179000 c:\qpid_VS2010\bin\Debug\boost_system-vc100-mt-gd-1_47.dll ModLoad: 71ab0000 71ac7000 C:\WINDOWS\system32\WS2_32.dll ModLoad: 77c10000 77c68000 C:\WINDOWS\system32\msvcrt.dll ModLoad: 71aa0000 71aa8000 C:\WINDOWS\system32\WS2HELP.dll ModLoad: 01180000 0120c000 c:\qpid_VS2010\bin\Debug\boost_program_options-vc100-mt-gd-1_47.dll ModLoad: 01210000 01223000 c:\qpid_VS2010\bin\Debug\boost_thread-vc100-mt-gd-1_47.dll ModLoad: 01230000 01246000 c:\qpid_VS2010\bin\Debug\boost_date_time-vc100-mt-gd-1_47.dll ModLoad: 01250000 01285000 c:\qpid_VS2010\bin\Debug\boost_filesystem-vc100-mt-gd-1_47.dll ModLoad: 71a50000 71a8f000 C:\WINDOWS\system32\mswsock.dll ModLoad: 662b0000 66308000 C:\WINDOWS\system32\hnetcfg.dll ModLoad: 77f10000 77f59000 C:\WINDOWS\system32\GDI32.dll ModLoad: 7e410000 7e4a1000 C:\WINDOWS\system32\USER32.dll ModLoad: 76390000 763ad000 C:\WINDOWS\system32\IMM32.DLL ModLoad: 58d40000 58d47000 C:\WINDOWS\System32\wship6.dll ModLoad: 71a90000 71a98000 C:\WINDOWS\System32\wshtcpip.dll Break-in sent, waiting 30 seconds... WARNING: Break-in timed out, suspending. This is usually caused by another thread holding the loader lock (4818.466c): Wake debugger - code 80000007 (first chance) eax=00daf614 ebx=00405100 ecx=7c839ad8 edx=7c9032bc esi=7c885780 edi=00000000 eip=7c90e514 esp=00daf368 ebp=00daf5ec iopl=0 nv up ei pl zr na pe nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246 *** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\system32\ntdll.dll - ntdll!KiFastSystemCallRet: 7c90e514 c3 ret 0:000> WARNING: Non-directory path: 'C:\qpid_VS2010\bin\Debug\client.pdb' DBGHELP: Symbol Search Path: http://msdl.microsoft.com/download/symbols;c:\qpid_vs2010\bin\debug\client.pdb DBGHELP: Symbol Search Path: http://msdl.microsoft.com/download/symbols;c:\qpid_vs2010\bin\debug\client.pdb 0:002> .reload Reloading current modules ........................... DBGHELP: ntdll - public symbols C:\Program Files\Debugging Tools for Windows (x86)\sym\ntdll.pdb\CEFC0863B1F84130A11E0F54180CD21A2\ntdll.pdb DBGHELP: kernel32 - public symbols C:\Program Files\Debugging Tools for Windows (x86)\sym\kernel32.pdb\072FF0EB54D24DFAAE9D13885486EE092\kernel32.pdb DBGHELP: MSVCR100D - private symbols & lines C:\Program Files\Debugging Tools for Windows (x86)\sym\msvcr100d.i386.pdb\9E26F0DD2C2543B99754DB8B9BA5F8681\msvcr100d.i386.pdb SYMSRV: C:\Program Files\Debugging Tools for Windows (x86)\sym\qpidcommond.pdb\BEB30FAFDC4245BD8D95B6AD44E7087214\qpidcommond.pdb not found SYMSRV: http://msdl.microsoft.com/download/symbols/qpidcommond.pdb/BEB30FAFDC4245BD8D95B6AD44E7087214/qpidcommond.pdb not found DBGHELP: c:\qpid_vs2010\bin\debug\client.pdb\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\client.pdb\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\client.pdb\symbols\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_VS2010\bin\Debug\qpidcommond.pdb - file not found DBGHELP: C:\cygwin\tmp\build\source\mrg-qpid-cpp-winsdk\v2.2\source\x86-VS2010\src\Debug\qpidcommond.pdb - file not found *** WARNING: Unable to verify checksum for c:\qpid_VS2010\bin\Debug\qpidcommond.dll *** ERROR: Symbol file could not be found. Defaulted to export symbols for c:\qpid_VS2010\bin\Debug\qpidcommond.dll - DBGHELP: qpidcommond - export symbols WARNING: Non-directory path: 'C:\qpid_VS2010\bin\Debug\client.pdb' WARNING: Non-directory path: 'C:\qpid_VS2010\bin\Debug\qpidcommond.pdb' DBGHELP: Symbol Search Path: http://msdl.microsoft.com/download/symbols;c:\qpid_vs2010\bin\debug\client.pdb;c:\qpid_vs2010\bin\debug\qpidcommond.pdb DBGHELP: Symbol Search Path: http://msdl.microsoft.com/download/symbols;c:\qpid_vs2010\bin\debug\client.pdb;c:\qpid_vs2010\bin\debug\qpidcommond.pdb 0:000> .reload Reloading current modules .......................... DBGHELP: ntdll - public symbols C:\Program Files\Debugging Tools for Windows (x86)\sym\ntdll.pdb\CEFC0863B1F84130A11E0F54180CD21A2\ntdll.pdb . DBGHELP: kernel32 - public symbols C:\Program Files\Debugging Tools for Windows (x86)\sym\kernel32.pdb\072FF0EB54D24DFAAE9D13885486EE092\kernel32.pdb DBGHELP: MSVCR100D - private symbols & lines C:\Program Files\Debugging Tools for Windows (x86)\sym\msvcr100d.i386.pdb\9E26F0DD2C2543B99754DB8B9BA5F8681\msvcr100d.i386.pdb SYMSRV: C:\Program Files\Debugging Tools for Windows (x86)\sym\qpidcommond.pdb\BEB30FAFDC4245BD8D95B6AD44E7087214\qpidcommond.pdb not found SYMSRV: http://msdl.microsoft.com/download/symbols/qpidcommond.pdb/BEB30FAFDC4245BD8D95B6AD44E7087214/qpidcommond.pdb not found DBGHELP: c:\qpid_vs2010\bin\debug\client.pdb\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\client.pdb\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\client.pdb\symbols\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidcommond.pdb\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidcommond.pdb\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidcommond.pdb\symbols\dll\qpidcommond.pdb - file not found *** WARNING: Unable to verify checksum for c:\qpid_VS2010\bin\Debug\qpidcommond.dll DBGHELP: qpidcommond - private symbols & lines c:\qpid_VS2010\bin\Debug\qpidcommond.pdb WARNING: Non-directory path: 'C:\qpid_VS2010\bin\Debug\client.pdb' WARNING: Non-directory path: 'C:\qpid_VS2010\bin\Debug\qpidcommond.pdb' WARNING: Non-directory path: 'C:\qpid_VS2010\bin\Debug\qpidmessagingd.pdb' WARNING: Non-directory path: 'C:\qpid_VS2010\bin\Debug\qpidclientd.pdb' DBGHELP: Symbol Search Path: http://msdl.microsoft.com/download/symbols;c:\qpid_vs2010\bin\debug\client.pdb;c:\qpid_vs2010\bin\debug\qpidcommond.pdb;c:\qpid_vs2010\bin\debug\qpidmessagingd.pdb;c:\qpid_vs2010\bin\debug\qpidclientd.pdb DBGHELP: Symbol Search Path: http://msdl.microsoft.com/download/symbols;c:\qpid_vs2010\bin\debug\client.pdb;c:\qpid_vs2010\bin\debug\qpidcommond.pdb;c:\qpid_vs2010\bin\debug\qpidmessagingd.pdb;c:\qpid_vs2010\bin\debug\qpidclientd.pdb 0:000> .reload Reloading current modules .......................... DBGHELP: ntdll - public symbols C:\Program Files\Debugging Tools for Windows (x86)\sym\ntdll.pdb\CEFC0863B1F84130A11E0F54180CD21A2\ntdll.pdb . DBGHELP: kernel32 - public symbols C:\Program Files\Debugging Tools for Windows (x86)\sym\kernel32.pdb\072FF0EB54D24DFAAE9D13885486EE092\kernel32.pdb DBGHELP: MSVCR100D - private symbols & lines C:\Program Files\Debugging Tools for Windows (x86)\sym\msvcr100d.i386.pdb\9E26F0DD2C2543B99754DB8B9BA5F8681\msvcr100d.i386.pdb SYMSRV: C:\Program Files\Debugging Tools for Windows (x86)\sym\qpidcommond.pdb\BEB30FAFDC4245BD8D95B6AD44E7087214\qpidcommond.pdb not found SYMSRV: http://msdl.microsoft.com/download/symbols/qpidcommond.pdb/BEB30FAFDC4245BD8D95B6AD44E7087214/qpidcommond.pdb not found DBGHELP: c:\qpid_vs2010\bin\debug\client.pdb\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\client.pdb\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\client.pdb\symbols\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidcommond.pdb\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidcommond.pdb\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidcommond.pdb\symbols\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidmessagingd.pdb\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidmessagingd.pdb\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidmessagingd.pdb\symbols\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidclientd.pdb\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidclientd.pdb\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidclientd.pdb\symbols\dll\qpidcommond.pdb - file not found *** WARNING: Unable to verify checksum for c:\qpid_VS2010\bin\Debug\qpidcommond.dll DBGHELP: qpidcommond - private symbols & lines c:\qpid_VS2010\bin\Debug\qpidcommond.pdb WARNING: Non-directory path: 'C:\qpid_VS2010\bin\Debug\client.pdb' WARNING: Non-directory path: 'C:\qpid_VS2010\bin\Debug\qpidcommond.pdb' WARNING: Non-directory path: 'C:\qpid_VS2010\bin\Debug\qpidmessagingd.pdb' WARNING: Non-directory path: 'C:\qpid_VS2010\bin\Debug\qpidclientd.pdb' WARNING: Non-directory path: 'C:\qpid_VS2010\bin\Debug\qpidtypesd.pdb' DBGHELP: Symbol Search Path: http://msdl.microsoft.com/download/symbols;c:\qpid_vs2010\bin\debug\client.pdb;c:\qpid_vs2010\bin\debug\qpidcommond.pdb;c:\qpid_vs2010\bin\debug\qpidmessagingd.pdb;c:\qpid_vs2010\bin\debug\qpidclientd.pdb;c:\qpid_vs2010\bin\debug\qpidtypesd.pdb DBGHELP: Symbol Search Path: http://msdl.microsoft.com/download/symbols;c:\qpid_vs2010\bin\debug\client.pdb;c:\qpid_vs2010\bin\debug\qpidcommond.pdb;c:\qpid_vs2010\bin\debug\qpidmessagingd.pdb;c:\qpid_vs2010\bin\debug\qpidclientd.pdb;c:\qpid_vs2010\bin\debug\qpidtypesd.pdb 0:000> .reload Reloading current modules .......................... DBGHELP: ntdll - public symbols C:\Program Files\Debugging Tools for Windows (x86)\sym\ntdll.pdb\CEFC0863B1F84130A11E0F54180CD21A2\ntdll.pdb . DBGHELP: kernel32 - public symbols C:\Program Files\Debugging Tools for Windows (x86)\sym\kernel32.pdb\072FF0EB54D24DFAAE9D13885486EE092\kernel32.pdb DBGHELP: MSVCR100D - private symbols & lines C:\Program Files\Debugging Tools for Windows (x86)\sym\msvcr100d.i386.pdb\9E26F0DD2C2543B99754DB8B9BA5F8681\msvcr100d.i386.pdb SYMSRV: C:\Program Files\Debugging Tools for Windows (x86)\sym\qpidcommond.pdb\BEB30FAFDC4245BD8D95B6AD44E7087214\qpidcommond.pdb not found SYMSRV: http://msdl.microsoft.com/download/symbols/qpidcommond.pdb/BEB30FAFDC4245BD8D95B6AD44E7087214/qpidcommond.pdb not found DBGHELP: c:\qpid_vs2010\bin\debug\client.pdb\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\client.pdb\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\client.pdb\symbols\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidcommond.pdb\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidcommond.pdb\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidcommond.pdb\symbols\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidmessagingd.pdb\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidmessagingd.pdb\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidmessagingd.pdb\symbols\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidclientd.pdb\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidclientd.pdb\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidclientd.pdb\symbols\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidtypesd.pdb\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidtypesd.pdb\dll\qpidcommond.pdb - file not found DBGHELP: c:\qpid_vs2010\bin\debug\qpidtypesd.pdb\symbols\dll\qpidcommond.pdb - file not found *** WARNING: Unable to verify checksum for c:\qpid_VS2010\bin\Debug\qpidcommond.dll DBGHELP: qpidcommond - private symbols & lines c:\qpid_VS2010\bin\Debug\qpidcommond.pdb # ChildEBP RetAddr 00 00daf364 7c90d9ca ntdll!KiFastSystemCallRet (FPO: [0,0,0]) 01 00daf368 7c8645fd ntdll!NtRaiseHardError+0xc (FPO: [6,0,0]) 02 00daf5ec 7c8438fa kernel32!UnhandledExceptionFilter+0x628 (FPO: [Non-Fpo]) 03 00daf5f4 7c839b39 kernel32!BaseProcessStart+0x39 (FPO: [Non-Fpo]) 04 00daf61c 7c9032a8 kernel32!_except_handler3+0x61 (FPO: [Uses EBP] [3,0,7]) 05 00daf640 7c90327a ntdll!ExecuteHandler2+0x26 06 00daf6f0 7c90e48a ntdll!ExecuteHandler+0x24 07 00daf6f0 7c911689 ntdll!KiUserExceptionDispatcher+0xe (FPO: [2,0,0]) (CONTEXT @ 00daf724) 08 00daf9fc 7c91084c ntdll!RtlpCoalesceFreeBlocks+0x128 (FPO: [Non-Fpo]) 09 00dafad0 10307359 ntdll!RtlFreeHeap+0x2e9 (FPO: [Non-Fpo]) 0a 00dafaec 10316e5e MSVCR100D!_free_base(void * pBlock = 0x01300cd0)+0x29 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\free.c @ 50] 0b 00dafafc 10316970 MSVCR100D!_free_dbg_nolock(void * pUserData = 0x01300cf0, int nBlockUse = 0n1)+0x4ae (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\dbgheap.c @ 1431] 0c 00dafb34 10315a39 MSVCR100D!_free_dbg(void * pUserData = 0x01300cf0, int nBlockUse = 0n1)+0x50 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\dbgheap.c @ 1265] 0d 00dafb70 00f471c7 MSVCR100D!operator delete(void * pUserData = 0x01300cf0)+0xb9 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt_bld\self_x86\crt\src\dbgdel.cpp @ 54] 0e 00dafb80 00f491e6 qpidcommond!std::allocator<std::_Tree_nod<std::_Tset_traits<qpid::log::Statement *,std::less<qpid::log::Statement *>,std::allocator<qpid::log::Statement *>,0> >::_Node>::deallocate(struct std::_Tree_nod<std::_Tset_traits<qpid::log::Statement *,std::less<qpid::log::Statement *>,std::allocator<qpid::log::Statement *>,0> >::_Node * _Ptr = 0x01300cf0, unsigned int __formal = 1)+0x17 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\program files (x86)\microsoft visual studio 10.0\vc\include\xmemory @ 182] 0f 00dafb98 00f4852f qpidcommond!std::_Tree<std::_Tset_traits<qpid::log::Statement *,std::less<qpid::log::Statement *>,std::allocator<qpid::log::Statement *>,0> >::_Erase(struct std::_Tree_nod<std::_Tset_traits<qpid::log::Statement *,std::less<qpid::log::Statement *>,std::allocator<qpid::log::Statement *>,0> >::_Node * _Rootnode = 0x01300cf0)+0x96 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\program files (x86)\microsoft visual studio 10.0\vc\include\xtree @ 1620] 10 00dafba8 00f46da1 qpidcommond!std::_Tree<std::_Tset_traits<qpid::log::Statement *,std::less<qpid::log::Statement *>,std::allocator<qpid::log::Statement *>,0> >::clear(void)+0x2f (FPO: [Non-Fpo]) (CONV: thiscall) [c:\program files (x86)\microsoft visual studio 10.0\vc\include\xtree @ 1416] 11 00dafc14 00f461e2 qpidcommond!std::_Tree<std::_Tset_traits<qpid::log::Statement *,std::less<qpid::log::Statement *>,std::allocator<qpid::log::Statement *>,0> >::erase(class std::_Tree_const_iterator<std::_Tree_val<std::_Tset_traits<qpid::log::Statement *,std::less<qpid::log::Statement *>,std::allocator<qpid::log::Statement *>,0> > > _First = class std::_Tree_const_iterator<std::_Tree_val<std::_Tset_traits<qpid::log::Statement *,std::less<qpid::log::Statement *>,std::allocator<qpid::log::Statement *>,0> > >, class std::_Tree_const_iterator<std::_Tree_val<std::_Tset_traits<qpid::log::Statement *,std::less<qpid::log::Statement *>,std::allocator<qpid::log::Statement *>,0> > > _Last = class std::_Tree_const_iterator<std::_Tree_val<std::_Tset_traits<qpid::log::Statement *,std::less<qpid::log::Statement *>,std::allocator<qpid::log::Statement *>,0> > >)+0x111 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\program files (x86)\microsoft visual studio 10.0\vc\include\xtree @ 1383] 12 00dafc74 00f459ac qpidcommond!std::_Tree<std::_Tset_traits<qpid::log::Statement *,std::less<qpid::log::Statement *>,std::allocator<qpid::log::Statement *>,0> >::_Tidy(void)+0x82 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\program files (x86)\microsoft visual studio 10.0\vc\include\xtree @ 1866] 13 00dafc90 00f45236 qpidcommond!std::_Tree<std::_Tset_traits<qpid::log::Statement *,std::less<qpid::log::Statement *>,std::allocator<qpid::log::Statement *>,0> >::~_Tree<std::_Tset_traits<qpid::log::Statement *,std::less<qpid::log::Statement *>,std::allocator<qpid::log::Statement *>,0> >(void)+0x3c (FPO: [Non-Fpo]) (CONV: thiscall) [c:\program files (x86)\microsoft visual studio 10.0\vc\include\xtree @ 792]
Created attachment 578448 [details] user.dmp, drwtsn32.log, some analysis Attached is a debug file set from a bare XP machine. Client.exe eventually fails after 1000s of passes but with identical "instruction at 7c910717 referenced 01f30005" with a read error.
Created attachment 578531 [details] "spout.exe" Crash - Output from WinDBG
The "spout.exe" application crashed before sending messages on WindowsXP-x86 too.
I run our smoke test script on the scratch version "qpid-cpp-x*-VS*-3759.0.0.3.zip" packages on machines: - WindowsXP-x86 and Rhel5-x86_64 (packages for VS2008) 1 crash[es] of 70 client runs - 1.42% - Windows Server2003-x86 and Rhel6-i686 (packages for VS2010) 0 crash[es] of X client runs - 0% - Windows Server2008-x64 and Rhel5-i386 (packages for VS2008) 0 crash[es] of X client runs - 0% - Windows Server2008 R2 and Rhel6-x86_64 (packages for VS2010) 0 crash[es] of X client runs - 0% In total 1 crash[es] of Y runs - P % This issue appeared once on WindowsXP only; the "map_receiver.exe" crashed (after receiving messages)- one case from 70 cases.
Oh, I am sorry; the "client.exe" application runs 70 times on every machine and crashed only once - this is in 0.3% cases ;-)
I am sorry, but I saw this issue on qpid-cpp-win-3.1.0.1 too, but only on WindowsXP-x86 (in about 1% cases). I ran C++ examples built on MS Visual Studio 2008 (server.exe, client.exe, spout.exe, drain.exe, map_sender.exe and map_receiver.exe) 488 times and the same examples built on MS Visual Studio 2010 488 times: - 122 times against broker on Rhel5-i368 - 122 times against broker on Rhel5-x86_64 - 122 times against broker on Rhel6-i686 - 122 times against broker on Rhel6-x86_64) Results: 4 crashes of C++ examples built on MS Visual Studio 2008 (from 488 runs): - map_sender.exe against broker on Rhel5-x86_64 - drain.exe - three times: against broker on Rhel5-x86_64, Rhel6-i686 and Rhel6-x86_64 4 crashes of C++ examples built on MS Visual Studio 2010 (from 488 runs): - drain.exe - two times: against broker on Rhel5-x86_64 and Rhel6-x86_64 - spout.exe against broker on Rhel5-x86_64 - map_sender.exe against broker on Rhel6-x86_64 I have not seen this bug on another Windows machine yet (with the qpid-cpp-win-3.1.0.1). --> ASSIGNED
It seems that this issue occurs on WindowsXP-x86 if the machine has at least two processors only. I did not see the bug after removing a processor (on the same virtual machine WindowsXP-x86 with only one processor).
Cliff, is this issue affected by the recent windows fixes?
I do not have enough stress testing in various Windows setups to be sure. The fixes are certainly consistent with seeing the error more frequently on one windows version than another.
New fix upstream: QPID-4330. Still seeing similar faults on exit in other areas of the code (static destructors, malloc).
See https://issues.apache.org/jira/browse/QPID-4354 for followup on this, including steps to resolve and additional stack traces.
This issue still appears on WindowsXP-x86 machine, but only on 0.02% cases (3 fails from 12640 cases). I tested it on two different instances of the machine: the first machine is my origin virtual machine running on Rhel5 and the second one is a new installed virtual machine on another virtualisation server with Rhel6. --> ASSIGNED
Okay, we'll try to address it some more for the next release.