Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1385890 - qdrouterd on capsule routinely throws exception and has to be restarted
Summary: qdrouterd on capsule routinely throws exception and has to be restarted
Keywords:
Status: CLOSED DUPLICATE of bug 1366232
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Capsule - Content
Version: 6.2.1
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-17 20:59 UTC by Calvin Hartwell
Modified: 2023-09-14 03:32 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-05 14:26:34 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Comment 1 Calvin Hartwell 2016-10-17 21:03:53 UTC 
Description of problem:

qdrouterd throws exception on the capsule/satellite server periodically and requires a restart in order to function. 

Sep 29 09:00:29 gbl15219 qdrouterd: *** Error in `/usr/sbin/qdrouterd': double free or corruption (!prev): 0x00007efcac6d9650 ***
Sep 29 09:00:29 gbl15219 qdrouterd: ======= Backtrace: =========
Sep 29 09:00:29 gbl15219 qdrouterd: /lib64/libc.so.6(+0x7d053)[0x7efcbf57b053]
Sep 29 09:00:29 gbl15219 qdrouterd: /lib64/libqpid-proton.so.2(pn_class_decref+0x56)[0x7efcc02c7806]
Sep 29 09:00:29 gbl15219 qdrouterd: /lib64/libqpid-proton.so.2(+0x27580)[0x7efcc02d5580]
Sep 29 09:00:29 gbl15219 qdrouterd: /lib64/libqpid-proton.so.2(pn_class_decref+0x38)[0x7efcc02c77e8]
Sep 29 09:00:29 gbl15219 qdrouterd: /lib64/libqpid-proton.so.2(pn_collector_pop+0x22)[0x7efcc02d5722]
Sep 29 09:00:29 gbl15219 qdrouterd: /lib64/libqpid-dispatch.so.0(+0x16f00)[0x7efcc0514f00]
Sep 29 09:00:29 gbl15219 qdrouterd: /lib64/libqpid-dispatch.so.0(+0x29b9c)[0x7efcc0527b9c]
Sep 29 09:00:29 gbl15219 qdrouterd: /lib64/libqpid-dispatch.so.0(qd_server_run+0xb0)[0x7efcc05287b0]
Sep 29 09:00:29 gbl15219 qdrouterd: /usr/sbin/qdrouterd[0x401cd8]
Sep 29 09:00:29 gbl15219 qdrouterd: /usr/sbin/qdrouterd(main+0x140)[0x401950]
Sep 29 09:00:29 gbl15219 qdrouterd: /lib64/libc.so.6(__libc_start_main+0xf5)[0x7efcbf51fb15]
Sep 29 09:00:29 gbl15219 qdrouterd: /usr/sbin/qdrouterd[0x4019a1]
Sep 29 09:00:29 gbl15219 qdrouterd: ======= Memory map: ========
Sep 29 09:00:29 gbl15219 qdrouterd: 00400000-00403000 r-xp 00000000 fd:00 10740512 /usr/sbin/qdrouterd
Sep 29 09:00:29 gbl15219 qdrouterd: 00602000-00603000 r--p 00002000 fd:00 10740512 /usr/sbin/qdrouterd
Sep 29 09:00:29 gbl15219 qdrouterd: 00603000-00604000 rw-p 00003000 fd:00 10740512 /usr/sbin/qdrouterd
Sep 29 09:00:29 gbl15219 qdrouterd: 02264000-036b6000 rw-p 00000000 00:00 0 [heap]

This then causes disconnects on the client: 

[root@cutekittens~]# tail -f /var/log/messages
Oct 17 10:39:56 cutekittens adclient[7856]: INFO <fd:29 PAMVerifyPassword > audit User 'd3228273' authenticated based on Kerberos exchange to AD
Oct 17 10:39:56 cutekittens adclient[7856]: INFO <fd:30 PAMIsUserAllowedAccess2 > audit User 'd3228273' is authorized
Oct 17 10:39:56 cutekittens sshd[2003]: Accepted keyboard-interactive/pam for 43228273 from 10.199.226.102 port 51189 ssh2
Oct 17 10:40:04 cutekittens goferd: [INFO][worker-0] gofer.messaging.adapter.connect:28 - connecting: proton+amqps://capsulefqdn:5647
Oct 17 10:40:04 cutekittens goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:87 - open: URL: amqps://capsulefqdn:5647|SSL: ca: /etc/rhsm/ca/katello-default-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
Oct 17 10:40:04 cutekittens goferd: [INFO][worker-0] root:490 - connecting to capsulefqdn:5647...
Oct 17 10:40:04 cutekittens goferd: [INFO][worker-0] root:532 - Disconnected
Oct 17 10:40:04 cutekittens goferd: [ERROR][worker-0] gofer.messaging.adapter.connect:33 - connect: proton+amqps://capsulefqdn:5647, failed: Connection amqps://capsulefqdn:5647 disconnected
Oct 17 10:40:04 cutekittens goferd: [INFO][worker-0] gofer.messaging.adapter.connect:35 - retry in 106 seconds
Oct 17 10:40:05 cutekittens dzdo[2049]: INFO dz.dzdo Starting dzdo

This means the customer has to create a cron job to periodically restart the goferd agent

Version-Release number of selected component (if applicable):

6.2.1

How reproducible:

100% reproducible happening on all capsules

Steps to Reproduce:
1. Register machines to capsules, do some usual stuff. 
2. qdrouterd stops working, breaks, needs to be restarted.

Actual results:

qdrouterd is unstable, causes goferd to break. 

Expected results:

qdrouterd +goferd should work and be stable. 

Additional info:
Comment 6 Michael Hrivnak 2016-10-21 17:32:47 UTC 
Pulp upstream does not use qdrouterd, so we aren't much help with these bugs.

I'm adding Justin Ross who can advise from the messaging team's perspective.
Comment 10 Andrew Kofink 2017-01-05 14:26:34 UTC 

*** This bug has been marked as a duplicate of bug 1366232 ***
Comment 11 Red Hat Bugzilla 2023-09-14 03:32:41 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days
Note You need to log in before you can comment on or make changes to this bug.