Bug 1392768

Smaller reproducer:

- /etc/qpid/qpidd.conf:
log-enable=error+
log-to-syslog=yes
auth=no
#require-encryption=yes
#ssl-require-client-authentication=yes
ssl-port=5671
ssl-cert-db=/etc/pki/katello/nssdb
ssl-cert-password-file=/etc/pki/katello/nssdb/nss_db_password-file
ssl-cert-name=broker

(the ssl-* options are mandatory, disabling ssl completely prevents the leak)

- Satellite services stopped, just qpidd and qdrouterd running where qdrouterd is connected via 5672 nonSSL port (sic!).

- create pulp.bz1272758 queue in qpidd

- run 10 (or 20 or 30) instances of the script from https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c21

- wait some time

- running qpidd under valgrind shows SSL related leaks (while no SSL connection made? and the only one connection from qdrouterd was to 5672 and unencrypted?

Created attachment 1218977 [details]
valgrind report

See esp.:

==21883== 121,648 (96 direct, 121,552 indirect) bytes in 2 blocks are definitely lost in loss record 307 of 308
==21883==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==21883==    by 0x7951F69: _PR_Getfd (prfdcach.c:103)
==21883==    by 0x796BC36: pt_SetMethods.isra.11 (ptio.c:3303)
==21883==    by 0x796C03E: PR_Socket (ptio.c:3505)
==21883==    by 0x5628C90: qpid::sys::ssl::SslSocket::SslSocket(std::string const&, bool) (SslSocket.cpp:90)
==21883==    by 0x50840D5: qpid::sys::(anonymous namespace)::createServerSSLSocket(qpid::sys::SslServerOptions const&) (SslPlugin.cpp:62)
==21883==    by 0x51DF7CE: operator() (function_template.hpp:767)
==21883==    by 0x51DF7CE: qpid::sys::SocketAcceptor::listen(std::vector<std::string, std::allocator<std::string> > const&, unsigned short, int, boost::function0<qpid::sys::Socket*> const&) (SocketTransport.cpp:160)
==21883==    by 0x50863FA: qpid::sys::SslPlugin::initialize(qpid::Plugin::Target&) (SslPlugin.cpp:126)
==21883==    by 0x563E8FE: operator() (mem_fn_template.hpp:165)
==21883==    by 0x563E8FE: operator()<boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>, boost::_bi::list1<qpid::Plugin* const&> > (bind.hpp:313)
==21883==    by 0x563E8FE: operator()<qpid::Plugin*> (bind_template.hpp:47)
==21883==    by 0x563E8FE: for_each<__gnu_cxx::__normal_iterator<qpid::Plugin* const*, std::vector<qpid::Plugin*> >, boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>, boost::_bi::list2<boost::arg<1>, boost::reference_wrapper<qpid::Plugin::Target> > > > (stl_algo.h:4417)
==21883==    by 0x563E8FE: void qpid::(anonymous namespace)::each_plugin<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>, boost::_bi::list2<boost::arg<1>, boost::reference_wrapper<qpid::Plugin::Target> > > >(boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>, boost::_bi::list2<boost::arg<1>, boost::reference_wrapper<qpid::Plugin::Target> > > const&) [clone .constprop.57] (Plugin.cpp:73)
==21883==    by 0x563E9F1: qpid::Plugin::initializeAll(qpid::Plugin::Target&) (Plugin.cpp:91)
==21883==    by 0x50A3559: qpid::broker::Broker::Broker(qpid::broker::BrokerOptions const&) (Broker.cpp:366)
==21883==    by 0x405C81: qpid::broker::QpiddBroker::execute(qpid::broker::QpiddOptions*) (QpiddBroker.cpp:229)

Another reproducer "outside Sat6": creating AMQP links to qpidd directly and detaching them (and doing so in a tight loop):

1) Create some queues:

for i in $(seq 0 30); do qpid-config add queue queue.$i ; done

2) Call python reactor script:

#!/usr/bin/python

from time import sleep
from uuid import uuid4

from proton import ConnectionException, Timeout
from proton import SSLDomain, SSLException
#from proton import Message

from proton.utils import BlockingConnection

import threading
import traceback
import os

ROUTER_ADDRESS = "proton+amqp://127.0.0.1:5672"
ADDRESS = "queue"
HEARTBEAT = 2
SLEEP = 0.2
THREADS = 30

class ReceiverThread(threading.Thread):
    def __init__(self, _id, address=ADDRESS, domain=None):
        super(ReceiverThread, self).__init__()
        self._id = _id
        self.address = address
        print self.address
        self.domain = domain
        self.running = True
        self.conn = None

    def connect(self):
        try:
            self.conn = BlockingConnection(ROUTER_ADDRESS, ssl_domain=self.domain, heartbeat=HEARTBEAT)
        except Exception, e:
            print e
            self.conn = None

    def run(self):
        while self.running:
            while self.conn == None:
                self.connect()
            sleep(SLEEP)
            #connection = self.conn
            #self.conn = None
            try:
                recv = self.conn.create_receiver(self.address, name=str(uuid4()), dynamic=False, options=None)
                sleep(SLEEP)
                recv.close()
            except Exception, e:
                print e
                pass

    def stop(self):
        self.running = False

threads = []
for i in range(THREADS):
  threads.append(ReceiverThread(i, '%s.%s' %(ADDRESS, i), None))
  sleep(SLEEP/THREADS)
  threads[i].start()

while True:
  sleep(10)
_in = raw_input("Press Enter to exit:")

for i in range(THREADS):
    threads[i].stop()
for i in range(THREADS):
    threads[i].join()

btw testing same scenarios (outside Sat6) with below packages does _not_ show the mem.leak. So rebase of qpid bits in Sat6 is a solution.

qpid-cpp-client-0.34-17.el6.x86_64
qpid-cpp-server-0.34-17.el6.x86_64
qpid-proton-c-devel-0.10-2.el6.x86_64
qpid-proton-c-0.10-2.el6.x86_64

(In reply to Pavel Moravec from comment #0)
> Description of problem:
> Follow-up of https://bugzilla.redhat.com/show_bug.cgi?id=1367735 that
> addrsses mem.leak in qdrouterd (that caused mem.increase in qpidd
> indirectly), same reproducers trigger a memory leak in qpidd as well. In
> particular:
> 
> - python script from https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c21
> (does not rely on Sat bits at all, worth for qpid engineering)
> - restarting goferd
> - installing a package via katello-agent
> 
> Repeating either scenario many times, RSS of qpidd process slowly grows.
> Even on proton 0.9-16 and router 0.4-19.
> 
> 
> Version-Release number of selected component (if applicable):
> qpid-dispatch-router-0.4-19.el7sat.x86_64
> qpid-proton-c-0.9-16.el7.x86_64
> qpid-cpp-server-0.30-11.el7sat.x86_64
> 
> 
> How reproducible:
> 100%
> 
> 
> Steps to Reproduce:
> 1) Use either reproducer:
>   a) https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c21
>   b) "service goferd restart" on a Content Host
>   c) install a package or errata via katello-agent
> 2) Repeat 1 many times and observe RSS of qpidd
> 
> 
> Actual results:
> Continuous memory growth.
> 
> 
> Expected results:
> No memory growth after a while.
> 
> 
> Additional info:

I verified all the scenarios with qpid-cpp 0.34 (see https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c51) and no mem.leak is present. So this BZ is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1367735 that can be resolved by qpid-cpp upgrade to 0.34 (independently to proton-c version).

*** This bug has been marked as a duplicate of bug 1367735 ***