Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1418026

Summary: goferd errors with "[...] Condition('amqp:resource-limit-exceeded', 'local-idle-timeout expired')" when pushing Errata from Satellite
Product: Red Hat Satellite Reporter: Ben <ben.argyle>
Component: QpidAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: Jan Hutaƙ <jhutar>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.2.7CC: afarley, anerurka, ben.argyle, bkearney, cdonnell, egolov, hmore, jalviso, janarula, jbhatia, jcallaha, jhutar, jortel, jsherril, ktordeur, mlele, mmccune, mmello, mvanderw, pmoravec, rajukuma, ralph.william, sbobylev, stbenjam
Target Milestone: 6.5.0Keywords: PrioBumpGSS, Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: gofer-2.12.3, gofer-2.11.7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1646736 (view as bug list) Environment:
Last Closed: 2019-05-14 12:36:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ben 2017-01-31 16:19:37 UTC
Description of problem:
When executing Errata installation from Satellite, whilest the RPM installations take place, the following error is seen in the Content Host's /var/log/messages:

Jan 31 15:40:51 client1 goferd: [ERROR][worker-0] gofer.messaging.adapter.proton.reliability:53 - Connection amqps://satellite1:5647 disconnected: Condition('amqp:resource-limit-exceeded', 'local-idle-timeout expired')

Version-Release number of selected component (if applicable):

Content Host:
gofer-2.7.6-1.el7sat.noarch
python-gofer-proton-2.7.6-1.el7sat.noarch
python-gofer-2.7.6-1.el7sat.noarch
katello-agent-2.5.0-5.el7sat.noarch

Satellite Host:
qpid-proton-c-0.9-16.el7.x86_64
libqpid-dispatch-0.4-21.el7sat.x86_64
qpid-cpp-server-linearstore-0.30-11.el7sat.x86_64
python-qpid-0.30-9.el7sat.noarch
tfm-rubygem-qpid_messaging-0.30.0-7.el7sat.x86_64
python-qpid-qmf-0.30-5.el7.x86_64
qpid-dispatch-router-0.4-21.el7sat.x86_64
qpid-cpp-client-devel-0.30-11.el7sat.x86_64
qpid-cpp-client-0.30-11.el7sat.x86_64
qpid-qmf-0.30-5.el7.x86_64
satellite1-qpid-router-client-1.0-1.noarch
satellite1-qpid-client-cert-1.0-1.noarch
satellite1-qpid-router-server-1.0-1.noarch
python-gofer-qpid-2.7.6-1.el7sat.noarch
qpid-cpp-server-0.30-11.el7sat.x86_64
qpid-tools-0.30-4.el7.noarch
satellite1-qpid-broker-1.0-1.noarch

How reproducible:

Every Errata installation action initiated in the Satellite GUI.


Steps to Reproduce:
1. Publish Content View with new RPMs
2. Execute a Collection Action -> Errata Installation against a Content Host
3. Watch /var/log/messages on a Content Host

Actual results:

Errata are installed, and Satellite claims task completed successfully, but "ERROR" in /var/log/messages on Content Host.


Expected results:

Well, no error in /var/log/messages on Content Host.


Additional info:

Messages on the Satellite host at around the same time associated with the consumer "client1":

Jan 31 15:40:19 satellite1 pulp: pulp.plugins.pulp_rpm.plugins.profilers.yum:INFO: Rpms: <[{u'src': u'firefox-45.7.0-1.el7_3.src.rpm', u'name': u'firefox', u'sum': [u'sha256', u'cd21ad7f5a7a75449df379832e08f513424a1f87d6e9d7e977c2c58e7edb0e7d'], u'filename': u'firefox-45.7.0-1.el7_3.x86_64.rpm', u'epoch': u'0', u'version': u'45.7.0', u'release': u'1.el7_3', u'arch': u'x86_64'}, {u'src': u'firefox-45.7.0-1.el7_3.src.rpm', u'name': u'firefox', u'sum': [u'sha256', u'cd21ad7f5a7a75449df379832e08f513424a1f87d6e9d7e977c2c58e7edb0e7d'], u'filename': u'firefox-45.7.0-1.el7_3.x86_64.rpm', u'epoch': u'0', u'version': u'45.7.0', u'release': u'1.el7_3', u'arch': u'x86_64'}]> were found to be related to errata <Unit [key={'errata_id': u'RHSA-2017:0190'}] [type=erratum] [id=d29b9249-b2d2-458b-8e62-d98e7b8276f4]> and applicable to consumer <927de638-7bf9-4c26-9716-6ed42ca2e2cb>
Jan 31 15:41:02 satellite1 pulp: pulp.server.agent.direct.services:INFO: (43494-08352)   user data : {'task_id': 'af54453a-52fd-4264-ba05-d89d1f0a905e', 'consumer_id': '927de638-7bf9-4c26-9716-6ed42ca2e2cb'}

It _looks_ like the issue in the attached Foreman bug.  On my Satellite host I have "idle-timeout-seconds: 0" in /etc/qpid-dispatch/qdrouterd.conf...  I haven't tried commenting that out, yet.

Comment 1 Satellite Program 2017-01-31 17:09:53 UTC
Upstream bug assigned to jsherril

Comment 2 Satellite Program 2017-01-31 17:09:56 UTC
Upstream bug assigned to jsherril

Comment 5 Satellite Program 2017-09-13 20:10:25 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16897 has been resolved.

Comment 9 Pavel Moravec 2018-04-24 07:27:02 UTC
Reproducer:

Have some RHEL system with many updates applicable (extreme example: RHEL7.0 (RHEL7.4 GA suffices) system subscribed to rhel-7-server-rpms repo) and run:

hammer host package upgrade-all --host <hostname>


Then logs have:

Apr 24 09:17:20 pmoravec-rhel74.gsslab.brq2.redhat.com goferd[30091]: [INFO][worker-0] gofer.rmi.dispatcher:603 - call: Content.update() sn=b4ebaef8-4836-4a09-9d4a-f3b5ce01d91c data={u'task_id': u'05a4efdf-8194-499f-ab88-6f84212b254b', u'consumer_id': u'507d2144-b02c-4189-97b8-cd7c45411528'}
..
Apr 24 09:22:45 pmoravec-rhel74.gsslab.brq2.redhat.com goferd[30091]: [INFO][worker-0] gofer.agent.rmi:193 - Request: b4ebaef8-4836-4a09-9d4a-f3b5ce01d91c, committed
Apr 24 09:22:45 pmoravec-rhel74.gsslab.brq2.redhat.com goferd[30091]: [INFO][worker-0] gofer.agent.rmi:147 - Request: b4ebaef8-4836-4a09-9d4a-f3b5ce01d91c processed in: 5.455 (minutes)
Apr 24 09:22:45 pmoravec-rhel74.gsslab.brq2.redhat.com goferd[30091]: [ERROR][worker-0] gofer.messaging.adapter.proton.reliability:53 - Connection amqps://pmoravec-sat62-rhel7.gsslab.brq2.redhat.com:5647 disconnected: Condition('amqp:resource-limit-exceeded', 'local-idle-timeout expired')
Apr 24 09:22:55 pmoravec-rhel74.gsslab.brq2.redhat.com goferd[30091]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:131 - closed: proton+amqps://pmoravec-sat62-rhel7.gsslab.brq2.redhat.com:5647
Apr 24 09:22:55 pmoravec-rhel74.gsslab.brq2.redhat.com goferd[30091]: [INFO][worker-0] gofer.messaging.adapter.connect:28 - connecting: proton+amqps://pmoravec-sat62-rhel7.gsslab.brq2.redhat.com:5647
Apr 24 09:22:55 pmoravec-rhel74.gsslab.brq2.redhat.com goferd[30091]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:87 - open: URL: amqps://pmoravec-sat62-rhel7.gsslab.brq2.redhat.com:5647|SSL: ca: /etc/rhsm/ca/katello-default-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
Apr 24 09:22:56 pmoravec-rhel74.gsslab.brq2.redhat.com goferd[30091]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:92 - opened: proton+amqps://pmoravec-sat62-rhel7.gsslab.brq2.redhat.com:5647
Apr 24 09:22:56 pmoravec-rhel74.gsslab.brq2.redhat.com goferd[30091]: [INFO][worker-0] gofer.messaging.adapter.connect:30 - connected: proton+amqps://pmoravec-sat62-rhel7.gsslab.brq2.redhat.com:5647
Apr 24 09:22:57 pmoravec-rhel74.gsslab.brq2.redhat.com goferd[30091]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:131 - closed: proton+amqps://pmoravec-sat62-rhel7.gsslab.brq2.redhat.com:5647
..

Anyway the error in client's /var/log/messages is worth ignoring as the errata/update request itself succeeded well. Just with some connection bounce between goferd and qdrouterd.


Is this correct, please, that the errata apply succeeded well?

Comment 10 Ben 2018-05-08 08:52:45 UTC
I _think_ my errata applied successfully.  As you say, I think there's just some connection bounce, and then things eventually work.

Comment 16 Pavel Moravec 2018-07-25 06:46:17 UTC
(In reply to Pavel Moravec from comment #15)
> I second this. Some connection bounces might happen, these are not erroneous
> itself, rather a heads up / warning. An error would be if subsequent
> connection attempt would fail.


A bit relevant bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1608217

Comment 32 errata-xmlrpc 2019-05-14 12:36:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222