Bug 459261

Summary:	Qpid SIGABRT after "too many files open" error has occured (RHEL4)
Product:	Red Hat Enterprise MRG	Reporter:	Gordon Sim <gsim>
Component:	qpid-cpp	Assignee:	messaging-bugs <messaging-bugs>
Status:	CLOSED ERRATA	QA Contact:	Kim van der Riet <kim.vdriet>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	1.0	CC:	freznice
Target Milestone:	1.0.1
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2008-10-06 18:59:11 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	456931
Bug Blocks:

Description Gordon Sim 2008-08-15 13:32:01 UTC

+++ This bug was initially created as a clone of Bug #456931 +++

qpid broker do not stop or die when the storage have used up all available file
descriptors.  But when a new client tries to connect to the broker, it dies with
the SIGABRT signal.

To reproduce:

  $> for id in `seq 10000 10080`; do (qpid-config --durable add queue queue_$id
&); done

This step should produce too many open files error.  By doing the following
command the broker dies:

  $> qpid-config 


Backtrace:
Core was generated by `/usr/sbin/qpidd --port 42557 --num-jfile 8
--jfile-size-pgs 128'.
Program terminated with signal 6, Aborted.
#0  0x000000323b830155 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x000000323b830155 in raise () from /lib64/libc.so.6
#1  0x000000323b831bf0 in abort () from /lib64/libc.so.6
#2  0x000000323e0becc4 in __gnu_cxx::__verbose_terminate_handler () from
/usr/lib64/libstdc++.so.6
#3  0x000000323e0bce36 in std::set_unexpected () from /usr/lib64/libstdc++.so.6
#4  0x000000323e0bce63 in std::terminate () from /usr/lib64/libstdc++.so.6
#5  0x000000323e0bcf4a in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x000000323f131e54 in qpid::sys::Socket::accept () from
/usr/lib64/libqpidcommon.so.0
#7  0x000000323f135bf6 in qpid::sys::AsynchAcceptor::readable () from
/usr/lib64/libqpidcommon.so.0
#8  0x000000323f16cf77 in boost::function1<void, qpid::sys::DispatchHandle&,
std::allocator<boost::function_base> >::operator() () from
/usr/lib64/libqpidcommon.so.0
#9  0x000000323f16b2c7 in qpid::sys::DispatchHandle::dispatchCallbacks () from
/usr/lib64/libqpidcommon.so.0
#10 0x000000323f16b36b in qpid::sys::Dispatcher::run () from
/usr/lib64/libqpidcommon.so.0
#11 0x000000323f13acda in qpid::sys::Thread::runRunnable () from
/usr/lib64/libqpidcommon.so.0
#12 0x000000323c0062f7 in start_thread () from /lib64/libpthread.so.0
#13 0x000000323b8d1b6d in clone () from /lib64/libc.so.6
(gdb)

--- Additional comment from gsim on 2008-07-30 04:40:44 EDT ---

Added exception handling to avoid abort. Checked in as r680937 on qpid.0-10,
r680939 on trunk.

Comment 2 Frantisek Reznicek 2008-09-04 13:06:41 UTC

RHTS automated test MRG/qpid_broker_die_bz456931 proves that qpidd is no longer dying after all file descriptors are used.

Qpidd produces tons of lines like this one (when run out of file descriptors):
'2008-sep-04 14:33:08 error Could not accept socket: Too many open files (qpid/sys/posix/Socket.cpp:224)' this indicate that logging improvements are needed.

Comment 4 errata-xmlrpc 2008-10-06 18:59:11 UTC

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2008-0867.html