Hide Forgot
Description of problem: Follow-up of https://bugzilla.redhat.com/show_bug.cgi?id=1367735 that addrsses mem.leak in qdrouterd (that caused mem.increase in qpidd indirectly), same reproducers trigger a memory leak in qpidd as well. In particular: - python script from https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c21 (does not rely on Sat bits at all, worth for qpid engineering) - restarting goferd - installing a package via katello-agent Repeating either scenario many times, RSS of qpidd process slowly grows. Even on proton 0.9-16 and router 0.4-19. Version-Release number of selected component (if applicable): qpid-dispatch-router-0.4-19.el7sat.x86_64 qpid-proton-c-0.9-16.el7.x86_64 qpid-cpp-server-0.30-11.el7sat.x86_64 How reproducible: 100% Steps to Reproduce: 1) Use either reproducer: a) https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c21 b) "service goferd restart" on a Content Host c) install a package or errata via katello-agent 2) Repeat 1 many times and observe RSS of qpidd Actual results: Continuous memory growth. Expected results: No memory growth after a while. Additional info:
Smaller reproducer: - /etc/qpid/qpidd.conf: log-enable=error+ log-to-syslog=yes auth=no #require-encryption=yes #ssl-require-client-authentication=yes ssl-port=5671 ssl-cert-db=/etc/pki/katello/nssdb ssl-cert-password-file=/etc/pki/katello/nssdb/nss_db_password-file ssl-cert-name=broker (the ssl-* options are mandatory, disabling ssl completely prevents the leak) - Satellite services stopped, just qpidd and qdrouterd running where qdrouterd is connected via 5672 nonSSL port (sic!). - create pulp.bz1272758 queue in qpidd - run 10 (or 20 or 30) instances of the script from https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c21 - wait some time - running qpidd under valgrind shows SSL related leaks (while no SSL connection made? and the only one connection from qdrouterd was to 5672 and unencrypted?
Created attachment 1218977 [details] valgrind report See esp.: ==21883== 121,648 (96 direct, 121,552 indirect) bytes in 2 blocks are definitely lost in loss record 307 of 308 ==21883== at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==21883== by 0x7951F69: _PR_Getfd (prfdcach.c:103) ==21883== by 0x796BC36: pt_SetMethods.isra.11 (ptio.c:3303) ==21883== by 0x796C03E: PR_Socket (ptio.c:3505) ==21883== by 0x5628C90: qpid::sys::ssl::SslSocket::SslSocket(std::string const&, bool) (SslSocket.cpp:90) ==21883== by 0x50840D5: qpid::sys::(anonymous namespace)::createServerSSLSocket(qpid::sys::SslServerOptions const&) (SslPlugin.cpp:62) ==21883== by 0x51DF7CE: operator() (function_template.hpp:767) ==21883== by 0x51DF7CE: qpid::sys::SocketAcceptor::listen(std::vector<std::string, std::allocator<std::string> > const&, unsigned short, int, boost::function0<qpid::sys::Socket*> const&) (SocketTransport.cpp:160) ==21883== by 0x50863FA: qpid::sys::SslPlugin::initialize(qpid::Plugin::Target&) (SslPlugin.cpp:126) ==21883== by 0x563E8FE: operator() (mem_fn_template.hpp:165) ==21883== by 0x563E8FE: operator()<boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>, boost::_bi::list1<qpid::Plugin* const&> > (bind.hpp:313) ==21883== by 0x563E8FE: operator()<qpid::Plugin*> (bind_template.hpp:47) ==21883== by 0x563E8FE: for_each<__gnu_cxx::__normal_iterator<qpid::Plugin* const*, std::vector<qpid::Plugin*> >, boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>, boost::_bi::list2<boost::arg<1>, boost::reference_wrapper<qpid::Plugin::Target> > > > (stl_algo.h:4417) ==21883== by 0x563E8FE: void qpid::(anonymous namespace)::each_plugin<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>, boost::_bi::list2<boost::arg<1>, boost::reference_wrapper<qpid::Plugin::Target> > > >(boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>, boost::_bi::list2<boost::arg<1>, boost::reference_wrapper<qpid::Plugin::Target> > > const&) [clone .constprop.57] (Plugin.cpp:73) ==21883== by 0x563E9F1: qpid::Plugin::initializeAll(qpid::Plugin::Target&) (Plugin.cpp:91) ==21883== by 0x50A3559: qpid::broker::Broker::Broker(qpid::broker::BrokerOptions const&) (Broker.cpp:366) ==21883== by 0x405C81: qpid::broker::QpiddBroker::execute(qpid::broker::QpiddOptions*) (QpiddBroker.cpp:229)
Another reproducer "outside Sat6": creating AMQP links to qpidd directly and detaching them (and doing so in a tight loop): 1) Create some queues: for i in $(seq 0 30); do qpid-config add queue queue.$i ; done 2) Call python reactor script: #!/usr/bin/python from time import sleep from uuid import uuid4 from proton import ConnectionException, Timeout from proton import SSLDomain, SSLException #from proton import Message from proton.utils import BlockingConnection import threading import traceback import os ROUTER_ADDRESS = "proton+amqp://127.0.0.1:5672" ADDRESS = "queue" HEARTBEAT = 2 SLEEP = 0.2 THREADS = 30 class ReceiverThread(threading.Thread): def __init__(self, _id, address=ADDRESS, domain=None): super(ReceiverThread, self).__init__() self._id = _id self.address = address print self.address self.domain = domain self.running = True self.conn = None def connect(self): try: self.conn = BlockingConnection(ROUTER_ADDRESS, ssl_domain=self.domain, heartbeat=HEARTBEAT) except Exception, e: print e self.conn = None def run(self): while self.running: while self.conn == None: self.connect() sleep(SLEEP) #connection = self.conn #self.conn = None try: recv = self.conn.create_receiver(self.address, name=str(uuid4()), dynamic=False, options=None) sleep(SLEEP) recv.close() except Exception, e: print e pass def stop(self): self.running = False threads = [] for i in range(THREADS): threads.append(ReceiverThread(i, '%s.%s' %(ADDRESS, i), None)) sleep(SLEEP/THREADS) threads[i].start() while True: sleep(10) _in = raw_input("Press Enter to exit:") for i in range(THREADS): threads[i].stop() for i in range(THREADS): threads[i].join()
btw testing same scenarios (outside Sat6) with below packages does _not_ show the mem.leak. So rebase of qpid bits in Sat6 is a solution. qpid-cpp-client-0.34-17.el6.x86_64 qpid-cpp-server-0.34-17.el6.x86_64 qpid-proton-c-devel-0.10-2.el6.x86_64 qpid-proton-c-0.10-2.el6.x86_64
(In reply to Pavel Moravec from comment #0) > Description of problem: > Follow-up of https://bugzilla.redhat.com/show_bug.cgi?id=1367735 that > addrsses mem.leak in qdrouterd (that caused mem.increase in qpidd > indirectly), same reproducers trigger a memory leak in qpidd as well. In > particular: > > - python script from https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c21 > (does not rely on Sat bits at all, worth for qpid engineering) > - restarting goferd > - installing a package via katello-agent > > Repeating either scenario many times, RSS of qpidd process slowly grows. > Even on proton 0.9-16 and router 0.4-19. > > > Version-Release number of selected component (if applicable): > qpid-dispatch-router-0.4-19.el7sat.x86_64 > qpid-proton-c-0.9-16.el7.x86_64 > qpid-cpp-server-0.30-11.el7sat.x86_64 > > > How reproducible: > 100% > > > Steps to Reproduce: > 1) Use either reproducer: > a) https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c21 > b) "service goferd restart" on a Content Host > c) install a package or errata via katello-agent > 2) Repeat 1 many times and observe RSS of qpidd > > > Actual results: > Continuous memory growth. > > > Expected results: > No memory growth after a while. > > > Additional info: I verified all the scenarios with qpid-cpp 0.34 (see https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c51) and no mem.leak is present. So this BZ is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1367735 that can be resolved by qpid-cpp upgrade to 0.34 (independently to proton-c version). *** This bug has been marked as a duplicate of bug 1367735 ***