Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1518008 - goferd memory leak when installing/removing a package via katello-agent in a loop
Summary: goferd memory leak when installing/removing a package via katello-agent in a ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: katello-agent
Version: Unspecified
Hardware: x86_64
OS: Linux
high
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Jan Hutař
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-27 21:48 UTC by Pavel Moravec
Modified: 2021-06-10 13:43 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-02 17:53:13 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Pavel Moravec 2017-11-27 21:48:26 UTC 
Description of problem:
When installing and removing a package via goferd in a loop, goferd grows in memory usage.


Version-Release number of selected component (if applicable):
(everything up-to-date plus proton 0.9-21)
Sat6.2.12 / goferd from 6.2.12 sat tools on RHEL7.4 / plus qpid-proton 0.9-21

katello-agent-3.0.2-1.el7sat.noarch
katello-ca-consumer-pmoravec-sat62-rhel7.gsslab.brq2.redhat.com-1.0-1.noarch
katello-host-tools-3.0.2-1.el7sat.noarch
katello-host-tools-fact-plugin-3.0.2-1.el7sat.noarch
python-gofer-proton-2.7.7-2.el7sat.noarch
python-qpid-proton-0.9-21.el7.x86_64
python-rhsm-1.19.9-1.el7.x86_64
python-rhsm-certificates-1.19.9-1.el7.x86_64
qpid-proton-c-0.9-21.el7.x86_64
qpid-proton-debuginfo-0.9-20.el7.x86_64
yum-3.4.3-150.el7.noarch
yum-langpacks-0.4.2-7.el7.noarch
yum-metadata-parser-1.1.4-10.el7.x86_64
yum-rhn-plugin-2.0.1-6.1.el7_3.noarch
yum-utils-1.1.31-42.el7.noarch


How reproducible:
100%


Steps to Reproduce:
1) Running on Satellite:

host=fqdn.of.your.content.host
package=sos

action=install
while true; do
	hammer -u admin -p redhat host package ${action} --host ${host} --packages ${package}
	if [ "${action}" == "install" ]; then
		action=remove
	else
		action=install
	fi
	date
	sleep 1
done

2) monitor goferd memory usage on the Content Host


Actual results:
2) shows this memory growth ("ps aux" taken once per minute):
root      8091 19.8  7.0 1162256 127532 ?      Ssl  19:36   0:12 python /usr/bin/goferd --foreground
root      8091 23.3  7.0 1157100 128420 ?      Ssl  19:36   0:29 python /usr/bin/goferd --foreground
root      8091 23.6  7.2 1160196 131396 ?      Ssl  19:36   0:43 python /usr/bin/goferd --foreground
root      8091 24.2  7.6 1173648 137992 ?      Ssl  19:36   0:59 python /usr/bin/goferd --foreground
root      8091 24.4  7.5 1164696 136224 ?      Ssl  19:36   1:14 python /usr/bin/goferd --foreground
root      8091 24.4  7.6 1168176 139588 ?      Ssl  19:36   1:29 python /usr/bin/goferd --foreground
root      8091 24.2  7.8 1170660 142312 ?      Ssl  19:36   1:43 python /usr/bin/goferd --foreground
root      8091 24.4  8.7 1194448 158972 ?      Ssl  19:36   1:58 python /usr/bin/goferd --foreground
root      8091 24.5  8.3 1187180 151828 ?      Ssl  19:36   2:14 python /usr/bin/goferd --foreground
root      8091 24.6  8.2 1178736 150568 ?      Ssl  19:36   2:29 python /usr/bin/goferd --foreground
root      8091 24.7  8.4 1180976 152788 ?      Ssl  19:36   2:44 python /usr/bin/goferd --foreground
root      8091 24.7  8.9 1196776 162664 ?      Ssl  19:36   2:59 python /usr/bin/goferd --foreground
root      8091 24.8  9.2 1201504 167912 ?      Ssl  19:36   3:15 python /usr/bin/goferd --foreground
root      8091 24.9  8.8 1188676 160752 ?      Ssl  19:36   3:30 python /usr/bin/goferd --foreground
root      8091 24.8  9.0 1191336 163472 ?      Ssl  19:36   3:45 python /usr/bin/goferd --foreground
root      8091 24.8 10.0 1218388 183332 ?      Ssl  19:36   4:00 python /usr/bin/goferd --foreground
root      8091 24.8  9.1 1193952 166208 ?      Ssl  19:36   4:14 python /usr/bin/goferd --foreground
root      8091 24.8  9.1 1193936 166260 ?      Ssl  19:36   4:29 python /usr/bin/goferd --foreground
root      8091 24.8  9.5 1200328 172712 ?      Ssl  19:36   4:44 python /usr/bin/goferd --foreground
root      8091 24.8  9.7 1203896 176336 ?      Ssl  19:36   4:59 python /usr/bin/goferd --foreground
root      8091 24.9  9.7 1206376 176384 ?      Ssl  19:36   5:15 python /usr/bin/goferd --foreground
root      8091 25.0 10.0 1209076 181632 ?      Ssl  19:36   5:32 python /usr/bin/goferd --foreground
root      8091 25.1 10.0 1210100 182716 ?      Ssl  19:36   5:48 python /usr/bin/goferd --foreground


Expected results:
No such memory growth


Additional info:
- this might be relevant / regression from qpid-proton since new version is testing (cf. https://bugzilla.redhat.com/show_bug.cgi?id=1318015#c96)
- we should check if a dummy / skipped yum operation triggers the same mem.leak
Comment 3 Pavel Moravec 2017-11-28 08:25:56 UTC 
https://issues.apache.org/jira/browse/PROTON-1120 has been backported into 0.9-21 (or some lder version already).

The reproducer is in these lines of code only:

            sender = self.connection.create_sender(ADDRESS, name="someSender")
            sender.close()

being called in a loop, since:
- if I comment out receiver in above reproducer, leak is present
- if I comment out sending the message, leak is present
- if I make the sender stable (create it once, send msgs in a loop, close at the end), no leak


The most simplified reproducer (requires pulp.test queue and qpidd accepting non-SSL connections on 5672):

from time import sleep
from uuid import uuid4

from proton.utils import BlockingConnection


ADDRESS = 'pulp.test.3'
URL = 'amqp://pmoravec-sat62-rhel7.gsslab.brq2.redhat.com:5672'
DELAY = 0.1

connection = BlockingConnection(url=URL, ssl_domain=None)
for n in range(10000):
    print n
    sender = connection.create_sender(ADDRESS, name=str(uuid4()))
    sleep(DELAY)
    sender.close()
    sleep(DELAY)
Comment 4 Pavel Moravec 2017-11-28 08:32:30 UTC 
(the most simplified reproducer would be even:

while True:
  connection.create_sender(ADDRESS, name=str(uuid4())).close()

but that could be blamed as over-artificial reproducer)
Comment 7 Bryan Kearney 2019-07-02 17:53:13 UTC 
Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this, please do not reopen. Instead, feel free to contact Red Hat Technical Support. Thank you.
Comment 8 Bryan Kearney 2019-12-03 19:36:40 UTC 
Update. This was due to https://issues.apache.org/jira/browse/PROTON-1120 which was addressed in proton 0.12. The satellite version of qpid-proton contained this fix in 6.4 and beyond.
Note You need to log in before you can comment on or make changes to this bug.