Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1392768 - qpidd memory increase when installing a package or restarting goferd
Summary: qpidd memory increase when installing a package or restarting goferd
Keywords:
Status: CLOSED DUPLICATE of bug 1367735
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: katello-agent
Version: 6.2.4
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: Unspecified
Assignee: Mike Cressman
QA Contact: Katello QA List
URL: https://bugzilla.redhat.com/show_bug....
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-08 08:29 UTC by Pavel Moravec
Modified: 2020-01-17 16:08 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-06 14:43:39 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
valgrind report (22.48 KB, application/x-gzip)
2016-11-09 15:16 UTC, Pavel Moravec
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2720301 0 None None None 2016-11-08 13:14:54 UTC

Description Pavel Moravec 2016-11-08 08:29:37 UTC
Description of problem:
Follow-up of https://bugzilla.redhat.com/show_bug.cgi?id=1367735 that addrsses mem.leak in qdrouterd (that caused mem.increase in qpidd indirectly), same reproducers trigger a memory leak in qpidd as well. In particular:

- python script from https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c21 (does not rely on Sat bits at all, worth for qpid engineering)
- restarting goferd
- installing a package via katello-agent

Repeating either scenario many times, RSS of qpidd process slowly grows. Even on proton 0.9-16 and router 0.4-19.


Version-Release number of selected component (if applicable):
qpid-dispatch-router-0.4-19.el7sat.x86_64
qpid-proton-c-0.9-16.el7.x86_64
qpid-cpp-server-0.30-11.el7sat.x86_64


How reproducible:
100%


Steps to Reproduce:
1) Use either reproducer:
  a) https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c21
  b) "service goferd restart" on a Content Host
  c) install a package or errata via katello-agent
2) Repeat 1 many times and observe RSS of qpidd


Actual results:
Continuous memory growth.


Expected results:
No memory growth after a while.


Additional info:

Comment 1 Pavel Moravec 2016-11-09 15:12:23 UTC
Smaller reproducer:

- /etc/qpid/qpidd.conf:
log-enable=error+
log-to-syslog=yes
auth=no
#require-encryption=yes
#ssl-require-client-authentication=yes
ssl-port=5671
ssl-cert-db=/etc/pki/katello/nssdb
ssl-cert-password-file=/etc/pki/katello/nssdb/nss_db_password-file
ssl-cert-name=broker

(the ssl-* options are mandatory, disabling ssl completely prevents the leak)

- Satellite services stopped, just qpidd and qdrouterd running where qdrouterd is connected via 5672 nonSSL port (sic!).

- create pulp.bz1272758 queue in qpidd

- run 10 (or 20 or 30) instances of the script from https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c21

- wait some time

- running qpidd under valgrind shows SSL related leaks (while no SSL connection made? and the only one connection from qdrouterd was to 5672 and unencrypted?

Comment 2 Pavel Moravec 2016-11-09 15:16:45 UTC
Created attachment 1218977 [details]
valgrind report

See esp.:

==21883== 121,648 (96 direct, 121,552 indirect) bytes in 2 blocks are definitely lost in loss record 307 of 308
==21883==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==21883==    by 0x7951F69: _PR_Getfd (prfdcach.c:103)
==21883==    by 0x796BC36: pt_SetMethods.isra.11 (ptio.c:3303)
==21883==    by 0x796C03E: PR_Socket (ptio.c:3505)
==21883==    by 0x5628C90: qpid::sys::ssl::SslSocket::SslSocket(std::string const&, bool) (SslSocket.cpp:90)
==21883==    by 0x50840D5: qpid::sys::(anonymous namespace)::createServerSSLSocket(qpid::sys::SslServerOptions const&) (SslPlugin.cpp:62)
==21883==    by 0x51DF7CE: operator() (function_template.hpp:767)
==21883==    by 0x51DF7CE: qpid::sys::SocketAcceptor::listen(std::vector<std::string, std::allocator<std::string> > const&, unsigned short, int, boost::function0<qpid::sys::Socket*> const&) (SocketTransport.cpp:160)
==21883==    by 0x50863FA: qpid::sys::SslPlugin::initialize(qpid::Plugin::Target&) (SslPlugin.cpp:126)
==21883==    by 0x563E8FE: operator() (mem_fn_template.hpp:165)
==21883==    by 0x563E8FE: operator()<boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>, boost::_bi::list1<qpid::Plugin* const&> > (bind.hpp:313)
==21883==    by 0x563E8FE: operator()<qpid::Plugin*> (bind_template.hpp:47)
==21883==    by 0x563E8FE: for_each<__gnu_cxx::__normal_iterator<qpid::Plugin* const*, std::vector<qpid::Plugin*> >, boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>, boost::_bi::list2<boost::arg<1>, boost::reference_wrapper<qpid::Plugin::Target> > > > (stl_algo.h:4417)
==21883==    by 0x563E8FE: void qpid::(anonymous namespace)::each_plugin<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>, boost::_bi::list2<boost::arg<1>, boost::reference_wrapper<qpid::Plugin::Target> > > >(boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>, boost::_bi::list2<boost::arg<1>, boost::reference_wrapper<qpid::Plugin::Target> > > const&) [clone .constprop.57] (Plugin.cpp:73)
==21883==    by 0x563E9F1: qpid::Plugin::initializeAll(qpid::Plugin::Target&) (Plugin.cpp:91)
==21883==    by 0x50A3559: qpid::broker::Broker::Broker(qpid::broker::BrokerOptions const&) (Broker.cpp:366)
==21883==    by 0x405C81: qpid::broker::QpiddBroker::execute(qpid::broker::QpiddOptions*) (QpiddBroker.cpp:229)

Comment 4 Pavel Moravec 2016-11-24 11:58:36 UTC
Another reproducer "outside Sat6": creating AMQP links to qpidd directly and detaching them (and doing so in a tight loop):

1) Create some queues:

for i in $(seq 0 30); do qpid-config add queue queue.$i ; done

2) Call python reactor script:

#!/usr/bin/python

from time import sleep
from uuid import uuid4

from proton import ConnectionException, Timeout
from proton import SSLDomain, SSLException
#from proton import Message

from proton.utils import BlockingConnection

import threading
import traceback
import os

ROUTER_ADDRESS = "proton+amqp://127.0.0.1:5672"
ADDRESS = "queue"
HEARTBEAT = 2
SLEEP = 0.2
THREADS = 30

class ReceiverThread(threading.Thread):
    def __init__(self, _id, address=ADDRESS, domain=None):
        super(ReceiverThread, self).__init__()
        self._id = _id
        self.address = address
        print self.address
        self.domain = domain
        self.running = True
        self.conn = None

    def connect(self):
        try:
            self.conn = BlockingConnection(ROUTER_ADDRESS, ssl_domain=self.domain, heartbeat=HEARTBEAT)
        except Exception, e:
            print e
            self.conn = None

    def run(self):
        while self.running:
            while self.conn == None:
                self.connect()
            sleep(SLEEP)
            #connection = self.conn
            #self.conn = None
            try:
                recv = self.conn.create_receiver(self.address, name=str(uuid4()), dynamic=False, options=None)
                sleep(SLEEP)
                recv.close()
            except Exception, e:
                print e
                pass

    def stop(self):
        self.running = False

threads = []
for i in range(THREADS):
  threads.append(ReceiverThread(i, '%s.%s' %(ADDRESS, i), None))
  sleep(SLEEP/THREADS)
  threads[i].start()

while True:
  sleep(10)
_in = raw_input("Press Enter to exit:")

for i in range(THREADS):
    threads[i].stop()
for i in range(THREADS):
    threads[i].join()

Comment 5 Pavel Moravec 2016-11-24 12:38:15 UTC
btw testing same scenarios (outside Sat6) with below packages does _not_ show the mem.leak. So rebase of qpid bits in Sat6 is a solution.

qpid-cpp-client-0.34-17.el6.x86_64
qpid-cpp-server-0.34-17.el6.x86_64
qpid-proton-c-devel-0.10-2.el6.x86_64
qpid-proton-c-0.10-2.el6.x86_64

Comment 7 Pavel Moravec 2016-12-06 14:43:39 UTC
(In reply to Pavel Moravec from comment #0)
> Description of problem:
> Follow-up of https://bugzilla.redhat.com/show_bug.cgi?id=1367735 that
> addrsses mem.leak in qdrouterd (that caused mem.increase in qpidd
> indirectly), same reproducers trigger a memory leak in qpidd as well. In
> particular:
> 
> - python script from https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c21
> (does not rely on Sat bits at all, worth for qpid engineering)
> - restarting goferd
> - installing a package via katello-agent
> 
> Repeating either scenario many times, RSS of qpidd process slowly grows.
> Even on proton 0.9-16 and router 0.4-19.
> 
> 
> Version-Release number of selected component (if applicable):
> qpid-dispatch-router-0.4-19.el7sat.x86_64
> qpid-proton-c-0.9-16.el7.x86_64
> qpid-cpp-server-0.30-11.el7sat.x86_64
> 
> 
> How reproducible:
> 100%
> 
> 
> Steps to Reproduce:
> 1) Use either reproducer:
>   a) https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c21
>   b) "service goferd restart" on a Content Host
>   c) install a package or errata via katello-agent
> 2) Repeat 1 many times and observe RSS of qpidd
> 
> 
> Actual results:
> Continuous memory growth.
> 
> 
> Expected results:
> No memory growth after a while.
> 
> 
> Additional info:

I verified all the scenarios with qpid-cpp 0.34 (see https://bugzilla.redhat.com/show_bug.cgi?id=1367735#c51) and no mem.leak is present. So this BZ is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1367735 that can be resolved by qpid-cpp upgrade to 0.34 (independently to proton-c version).

*** This bug has been marked as a duplicate of bug 1367735 ***


Note You need to log in before you can comment on or make changes to this bug.