Description of problem: During automated testing we've seen coredump from messaging library used by collector QMF plugin. Unfortunately coredump file is not available and some debug info packages were not installed. /var/log/condor/core.1809: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'condor_collector -f' [09:54:07] Core file: /var/log/condor/core.1809 generated by condor_collector ----------------------1/1- -rw-------. 1 root root 23584768 Jan 5 09:51 /var/log/condor/core.1809 /var/log/condor/core.1809: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'condor_collector -f' GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see: warning: core file may not match specified executable file. [New Thread 1814] [New Thread 1809] [New Thread 1813] [Thread debugging using libthread_db enabled] Core was generated by `condor_collector -f'. Program terminated with signal 11, Segmentation fault. #0 0x00275424 in __kernel_vsyscall () Missing separate debuginfos, use: debuginfo-install condor-7.8.10-0.2.el6.i686 (gdb) eax 0x0 0 ecx 0x716 1814 edx 0xb 11 ebx 0x711 1809 esp 0xb640d200 0xb640d200 ebp 0xb640d218 0xb640d218 esi 0xb 11 edi 0x4ceff4 5042164 eip 0x275424 0x275424 <__kernel_vsyscall+16> eflags 0x206 [ PF IF ] cs 0x73 115 ss 0x7b 123 ds 0x7b 123 es 0x7b 123 fs 0x0 0 gs 0x33 51 (gdb) Using memory regions provided by the target. There are no memory regions defined. (gdb) 32 AT_SYSINFO Special system info/entry points 0x275414 33 AT_SYSINFO_EHDR System-supplied DSO's ELF header 0x275000 16 AT_HWCAP Machine-dependent CPU capability hints 0x78afbfd 6 AT_PAGESZ System page size 4096 17 AT_CLKTCK Frequency of times() 100 3 AT_PHDR Program headers for program 0x6b7034 4 AT_PHENT Size of program header entry 32 5 AT_PHNUM Number of program headers 9 7 AT_BASE Base address of interpreter 0xffbf9000 8 AT_FLAGS Flags 0x0 9 AT_ENTRY Entry point of program 0x6c2590 11 AT_UID Real user ID 0 12 AT_EUID Effective user ID 0 13 AT_GID Real group ID 0 14 AT_EGID Effective group ID 0 23 AT_SECURE Boolean, was exec setuid-like? 0 25 AT_RANDOM Address of 16 random bytes 0xbfc5dfdb 31 AT_EXECFN File name of executable 0xbfc5efe1 "/usr/sbin/condor_collector" 15 AT_PLATFORM String identifying platform 0xbfc5dfeb "i686" 0 AT_NULL End of vector 0x0 (gdb) Stack level 0, frame at 0xb640d210: eip = 0x275424 in __kernel_vsyscall; saved eip 0x4c5910 called by frame at 0xb640d220 Arglist at 0xb640d208, args: Locals at 0xb640d208, Previous frame's sp is 0xb640d210 Saved registers: ebp at 0xb640d200, eip at 0xb640d20c (gdb) From To Syms Read Shared Object Library 0x004dca60 0x004dda88 Yes (*) /lib/libdl.so.2 0x0079dd20 0x0094d1a8 Yes (*) /usr/lib/libcondor_utils_7_8_10.so 0x00375ab0 0x003c4558 Yes (*) /usr/lib/libclassad.so.4 0x0020d150 0x002260e8 Yes (*) /lib/libexpat.so.1 0x00233f90 0x00250588 Yes (*) /lib/libpcre.so.0 0x00b832d0 0x00bc4d88 Yes (*) /usr/lib/libssl.so.10 0x0051f000 0x0062b118 Yes (*) /usr/lib/libcrypto.so.10 0x002869a0 0x003059d8 Yes (*) /lib/libkrb5.so.3 0x00c94e50 0x00c95b88 Yes (*) /lib/libcom_err.so.2 0x003dcae0 0x003f6ef8 Yes (*) /lib/libk5crypto.so.3 0x00264ef0 0x0026a6f8 Yes (*) /lib/libkrb5support.so.0 0x0040be30 0x0043a8b8 Yes (*) /lib/libgssapi_krb5.so.2 0x00a1f7f0 0x00a2a8f8 Yes (*) /lib/libz.so.1 0x00e37610 0x00ea9fb8 Yes (*) /usr/lib/libstdc++.so.6 0x00448470 0x00462968 Yes (*) /lib/libm.so.6 0x0049afa0 0x004b3468 Yes (*) /lib/libgcc_s.so.1 0x004bb670 0x004c76e8 Yes (*) /lib/libpthread.so.0 0xb75b4b10 0xb76dea04 Yes (*) /lib/libc.so.6 0x00479830 0x0049147f Yes (*) /lib/ld-linux.so.2 0x00c216a0 0x00c30b18 Yes (*) /lib/libresolv.so.2 0x00ff8970 0x00ff91f8 Yes (*) /lib/libkeyutils.so.1 0x00a363f0 0x00a48008 Yes (*) /lib/libselinux.so.1 0x00d25aa0 0x00d2de98 Yes (*) /lib/libnss_files.so.2 0x00353c00 0x00357118 Yes (*) /lib/libnss_dns.so.2 0x00f4ce10 0x00f94cc8 Yes (*) /usr/lib/condor/plugins/MgmtCollectorPlugin-plugin.so 0xb72e11c0 0xb7400f98 Yes (*) /usr/lib/libqpidcommon.so.8 0x00a8f840 0x00aec9d8 Yes (*) /usr/lib/libqpidclient.so.8 0x00b1d560 0x00b381e8 Yes (*) /usr/lib/libqmf.so.6 0xb71934f0 0xb71d7b78 Yes (*) /usr/lib/libqmfengine.so.6 0x00b48d70 0x00b57338 Yes (*) /usr/lib/libqpidtypes.so.1 0x00bebc70 0x00c0c4d8 Yes (*) /usr/lib/libqmfconsole.so.7 0xb71153d0 0xb7156388 Yes (*) /usr/lib/libqpidmessaging.so.3 0xb708dc40 0xb70d9aa8 Yes (*) /usr/lib/libqmf2.so.1 0x00c55320 0x00c726b8 Yes (*) /usr/lib/libboost_program_options.so.5 0x00d10fc0 0x00d12be8 Yes (*) /lib/libuuid.so.1 0x00470880 0x00474618 Yes (*) /lib/librt.so.1 0x00c9c1d0 0x00cacad8 Yes (*) /usr/lib/libsasl2.so.2 0xb70448b0 0xb7049a18 Yes (*) /lib/libcrypt.so.1 0xb6fe2e60 0xb70253b8 Yes (*) /lib/libfreebl3.so 0x00cbf810 0x00ccb898 Yes (*) /usr/lib/qpid/client/sslconnector.so 0xb6fc18b0 0xb6fd81e8 Yes (*) /usr/lib/libsslcommon.so.8 0xb6e86210 0xb6f73858 Yes (*) /usr/lib/libnss3.so 0xb6e3b880 0xb6e63d18 Yes (*) /usr/lib/libssl3.so 0x00d5e8b0 0x00d82a78 Yes (*) /lib/libnspr4.so 0xb6e167e0 0xb6e264a8 Yes (*) /usr/lib/libnssutil3.so 0x00d9ae70 0x00d9c838 Yes (*) /lib/libplc4.so 0x0026fb10 0x00270b68 Yes (*) /lib/libplds4.so (*): Shared library is missing debugging information. (gdb) 3 Thread 0xb6e0fb70 (LWP 1813) 0x00275424 in __kernel_vsyscall () 2 Thread 0xb759b750 (LWP 1809) 0x00275424 in __kernel_vsyscall () * 1 Thread 0xb640eb70 (LWP 1814) 0x00275424 in __kernel_vsyscall () Thread 3 (Thread 0xb6e0fb70 (LWP 1813)): #0 0x00275424 in __kernel_vsyscall () #1 0xb7678f71 in select () from /lib/libc.so.6 #2 0x00846aeb in sleep () from /usr/lib/libcondor_utils_7_8_10.so #3 0xb72f706e in qpid::sys::sleep(int) () from /usr/lib/libqpidcommon.so.8 #4 0x00b2bb42 in qpid::management::ManagementAgentImpl::ConnectionThread::run() () from /usr/lib/libqmf.so.6 #5 0xb72f74f2 in ?? () from /usr/lib/libqpidcommon.so.8 #6 0x004bdb39 in start_thread () from /lib/libpthread.so.0 #7 0xb7680c1e in clone () from /lib/libc.so.6 Thread 2 (Thread 0xb759b750 (LWP 1809)): #0 0x00275424 in __kernel_vsyscall () #1 0x004c4a88 in send () from /lib/libpthread.so.0 #2 0x008b3172 in condor_write(char const*, int, char const*, int, int, int) () from /usr/lib/libcondor_utils_7_8_10.so #3 0x008e7115 in Buf::write(char const*, int, int, int) () from /usr/lib/libcondor_utils_7_8_10.so #4 0x008e71c3 in Buf::flush(char const*, int, void*, int, int) () from /usr/lib/libcondor_utils_7_8_10.so #5 0x008bcf2b in ReliSock::SndMsg::snd_packet(char const*, int, int, int) () from /usr/lib/libcondor_utils_7_8_10.so #6 0x008bdc16 in ReliSock::end_of_message() () from /usr/lib/libcondor_utils_7_8_10.so #7 0x00935c5e in DaemonCommandProtocol::ReadCommand() () from /usr/lib/libcondor_utils_7_8_10.so #8 0x00936440 in DaemonCommandProtocol::doProtocol() () from /usr/lib/libcondor_utils_7_8_10.so #9 0x00916a68 in DaemonCore::HandleReq(Stream*, Stream*) () from /usr/lib/libcondor_utils_7_8_10.so #10 0x00920233 in DaemonCore::HandleReq(int, Stream*) () from /usr/lib/libcondor_utils_7_8_10.so #11 0x009229db in DaemonCore::CallSocketHandler_worker(int, bool, Stream*) () from /usr/lib/libcondor_utils_7_8_10.so #12 0x00922a40 in DaemonCore::CallSocketHandler_worker_demarshall(void*) () from /usr/lib/libcondor_utils_7_8_10.so #13 0x008409b7 in CondorThreads::pool_add(void (*)(void*), void*, int*, char const*) () from /usr/lib/libcondor_utils_7_8_10.so #14 0x00916de0 in DaemonCore::CallSocketHandler(int&, bool) () from /usr/lib/libcondor_utils_7_8_10.so #15 0x0091a19e in DaemonCore::Driver() () from /usr/lib/libcondor_utils_7_8_10.so #16 0x00929d27 in dc_main(int, char**) () from /usr/lib/libcondor_utils_7_8_10.so #17 0x006c2746 in main () Thread 1 (Thread 0xb640eb70 (LWP 1814)): #0 0x00275424 in __kernel_vsyscall () #1 0x004c5910 in raise () from /lib/libpthread.so.0 #2 0x007c9dab in ?? () from /usr/lib/libcondor_utils_7_8_10.so #3 <signal handler called> #4 0x00e81886 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib/libstdc++.so.6 #5 0x00b50d5f in qpid::types::VariantImpl::asString() const () from /usr/lib/libqpidtypes.so.1 #6 0x00b511c0 in qpid::types::Variant::asString() const () from /usr/lib/libqpidtypes.so.1 #7 0x00b53ae7 in qpid::types::VariantImpl::create(qpid::types::Variant const&) () from /usr/lib/libqpidtypes.so.1 #8 0x00b53c7e in qpid::types::Variant::Variant(qpid::types::Variant const&) () from /usr/lib/libqpidtypes.so.1 #9 0x00f94482 in std::_Rb_tree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant>, std::_Select1st<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> > >::_M_copy(std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> > const*, std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> >*) () from /usr/lib/condor/plugins/MgmtCollectorPlugin-plugin.so #10 0x00f94549 in std::_Rb_tree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant>, std::_Select1st<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> > >::_M_copy(std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> > const*, std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> >*) () from /usr/lib/condor/plugins/MgmtCollectorPlugin-plugin.so #11 0x00b53ee0 in qpid::types::VariantImpl::VariantImpl(std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, qpid::types::Variant, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> > > const&) () from /usr/lib/libqpidtypes.so.1 #12 0x00b53fa1 in qpid::types::Variant::operator=(std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, qpid::types::Variant, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> > > const&) () from /usr/lib/libqpidtypes.so.1 #13 0x00b1e255 in qpid::management::ManagementAgentImpl::getHeartbeatContent(std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, qpid::types::Variant, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> > >&) () from /usr/lib/libqmf.so.6 #14 0x00b28526 in qpid::management::ManagementAgentImpl::sendHeartbeat() () from /usr/lib/libqmf.so.6 #15 0x00b28d73 in qpid::management::ManagementAgentImpl::periodicProcessing() () from /usr/lib/libqmf.so.6 #16 0x00b2aceb in qpid::management::ManagementAgentImpl::PublishThread::run() () from /usr/lib/libqmf.so.6 #17 0xb72f74f2 in ?? () from /usr/lib/libqpidcommon.so.8 #18 0x004bdb39 in start_thread () from /lib/libpthread.so.0 #19 0xb7680c1e in clone () from /lib/libc.so.6 (gdb) quit Version-Release number of selected component (if applicable): PyYAML-3.10-3.1.el6.i686 boost-1.41.0-25.el6.i686 boost-date-time-1.41.0-25.el6.i686 boost-devel-1.41.0-25.el6.i686 boost-filesystem-1.41.0-25.el6.i686 boost-graph-1.41.0-25.el6.i686 boost-iostreams-1.41.0-25.el6.i686 boost-math-1.41.0-25.el6.i686 boost-program-options-1.41.0-25.el6.i686 boost-python-1.41.0-25.el6.i686 boost-regex-1.41.0-25.el6.i686 boost-serialization-1.41.0-25.el6.i686 boost-signals-1.41.0-25.el6.i686 boost-system-1.41.0-25.el6.i686 boost-test-1.41.0-25.el6.i686 boost-thread-1.41.0-25.el6.i686 boost-wave-1.41.0-25.el6.i686 clusterlib-3.0.12.1-68.el6.i686 compat-readline5-5.2-17.1.el6.i686 condor-7.8.10-0.2.el6.i686 condor-classads-7.8.10-0.2.el6.i686 condor-kbdd-7.8.10-0.2.el6.i686 condor-qmf-7.8.10-0.2.el6.i686 corosync-1.4.7-1.el6.i686 corosynclib-1.4.7-1.el6.i686 cumin-0.1.5797-5.el6.noarch java-1.5.0-gcj-1.5.0.0-29.1.el6.i686 java-1.6.0-openjdk-1.6.0.33-1.13.5.1.el6_6.i686 java-1.6.0-openjdk-devel-1.6.0.33-1.13.5.1.el6_6.i686 java_cup-0.10k-5.el6.i686 jline-0.9.94-0.8.el6.noarch libcgroup-0.40.rc1-15.el6_6.i686 libibverbs-1.1.8-3.el6.i686 libicu-4.2.1-9.1.el6_2.i686 librdmacm-1.0.18.1-1.el6.i686 libuuid-devel-2.17.2-12.18.el6.i686 libyaml-0.1.3-1.4.el6.i686 lm_sensors-libs-3.1.1-17.el6.i686 log4j-1.2.14-6.4.el6.i686 lynx-2.8.6-27.el6.i686 mrg-release-2.5.0-1.el6.noarch net-snmp-libs-5.5-50.el6_6.1.i686 perl-Date-Manip-6.24-1.el6.noarch perl-YAML-Syck-1.07-4.el6.i686 postgresql-8.4.20-1.el6_5.i686 postgresql-libs-8.4.20-1.el6_5.i686 postgresql-server-8.4.20-1.el6_5.i686 python-condorutils-1.5-6.el6.noarch python-devel-2.6.6-52.el6.i686 python-kerberos-1.1-7.el6.i686 python-ldap-2.3.10-1.el6.i686 python-pip-1.3.1-4.el6.noarch python-psycopg2-2.0.14-2.el6.i686 python-qpid-0.18-13.el6.noarch python-qpid-qmf-0.18-29.el6.i686 python-saslwrapper-0.18-1.el6_3.i686 python-suds-0.4.1-3.el6.noarch python-wallaby-0.16.3-1.el6.noarch python-wallabyclient-5.0.5-2.el6.noarch qpid-cpp-client-0.18-36.el6.i686 qpid-cpp-client-devel-0.18-36.el6.i686 qpid-cpp-client-devel-docs-0.18-36.el6.noarch qpid-cpp-client-ssl-0.18-36.el6.i686 qpid-cpp-server-0.18-36.el6.i686 qpid-cpp-server-cluster-0.18-36.el6.i686 qpid-cpp-server-devel-0.18-36.el6.i686 qpid-cpp-server-ssl-0.18-36.el6.i686 qpid-cpp-server-store-0.18-36.el6.i686 qpid-cpp-server-xml-0.18-36.el6.i686 qpid-java-client-0.18-9.el6.noarch qpid-java-common-0.18-9.el6.noarch qpid-java-example-0.18-9.el6.noarch qpid-jca-0.18-8.el6.noarch qpid-jca-xarecovery-0.18-8.el6.noarch qpid-qmf-0.18-29.el6.i686 qpid-tools-0.18-10.el6_4.noarch rhino-1.7-0.7.r2.2.el6.noarch rhm-docs-0.18-2.el6.noarch ruby-1.8.7.374-3.el6_6.i686 ruby-libs-1.8.7.374-3.el6_6.i686 saslwrapper-0.18-1.el6_3.i686 sesame-1.0-8.el6.i686 sinjdoc-0.5-9.1.el6.i686 xerces-c-3.0.1-20.el6.i686 xml-commons-apis-1.3.04-3.6.el6.i686 xqilla-2.2.3-8.el6.i686 How reproducible: rare Steps to Reproduce: 1. run automated tests
MRG-Grid is in maintenance and only customer escalations will be considered. This issue can be reopened if a customer escalation associated with it occurs.