Bug 1452183

Summary: goferd disconnects with no route to destination node on client
Product: Red Hat Satellite Reporter: Satellite Program <pm-sat>
Component: QpidAssignee: Mike Cressman <mcressma>
Status: CLOSED ERRATA QA Contact: Lukas Pramuk <lpramuk>
Severity: medium Docs Contact:
Priority: high    
Version: 6.2.7CC: adprice, akofink, alan, aperotti, bbuckingham, bkearney, cbirk, daniele, dnagarat, egolov, ehelms, ggatward, gmurthy, hmore, janarula, jcallaha, jentrena, justin.matlock, ktordeur, lpramuk, mkalyat, mlele, mmccune, mvanderw, pdwyer, pmoravec, sghai, swadeley
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qpid-dispatch-0.4-22 Doc Type: Bug Fix
Doc Text:
When pausing a Satellite in a VM, any goferd client on a machine registered to a Capsule failed to connect to the Capsule and logged “qd:no-route-to-dest” error. The error persisted after qdrouterd on the Satellite resumed. The qpid dispatch router has been improved to unmap all addresses in a more reliable way.
Story Points: ---
Clone Of: 1426242 Environment:
Last Closed: 2018-02-21 12:57:54 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1353215    

Comment 2 Bryan Kearney 2017-06-15 13:31:20 UTC
PLease pull in at least qpid-dispatch-0.4-22

Comment 7 Lukas Pramuk 2018-01-17 10:53:44 UTC
VERIFIED.

@satellite-capsule-6.3.0-23.0.el7sat.noarch
qpid-dispatch-router-0.8.0-16.el7sat.x86_64

using manual reproducer described in https://bugzilla.redhat.com/show_bug.cgi?id=1426242#c40

5) service goferd restart

Jan 17 05:36:58 host.example.com systemd[1]: Stopping Gofer Agent...
Jan 17 05:36:58 host.example.com systemd[1]: Started Gofer Agent.
Jan 17 05:36:58 host.example.com systemd[1]: Starting Gofer Agent...
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][Thread-1] gofer.rmi.store:114 - Using: /var/lib/gofer/messaging/pending/demo
Jan 17 05:36:58 host.example.com goferd[29656]: [WARNING][MainThread] gofer.agent.plugin:639 - plugin:demo, DISABLED
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][Thread-2] gofer.rmi.store:114 - Using: /var/lib/gofer/messaging/pending/katelloplugin
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][Thread-3] gofer.rmi.store:114 - Using: /var/lib/gofer/messaging/pending/katelloplugin
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][MainThread] gofer.agent.plugin:682 - plugin:katelloplugin loaded using: /usr/lib/gofer/plugins/katelloplugin.py
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][MainThread] rhsm.connection:821 - Connection built: host=cap.example.com port=8443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][MainThread] rhsm.connection:551 - Response: status=200, request="GET /rhsm/consumers/1f59d249-4042-4885-aa8f-8b6b071695fd"
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][MainThread] katelloplugin:164 - Using /etc/rhsm/ca/katello-default-ca.pem as the ca cert for qpid connection
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][worker-0] gofer.messaging.adapter.connect:28 - connecting: proton+amqps://cap.example.com:5647
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:87 - open: URL: amqps://cap.example.com:5647|SSL: ca: /etc/rhsm/ca/katello-default-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][MainThread] gofer.agent.main:87 - agent started.
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:92 - opened: proton+amqps://cap.example.com:5647
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][worker-0] gofer.messaging.adapter.connect:30 - connected: proton+amqps://cap.example.com:5647
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:131 - closed: proton+amqps://cap.example.com:5647
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][pulp.agent.1f59d249-4042-4885-aa8f-8b6b071695fd] gofer.messaging.adapter.connect:28 - connecting: proton+amqps://cap.example.com:5647
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][pulp.agent.1f59d249-4042-4885-aa8f-8b6b071695fd] gofer.messaging.adapter.proton.connection:87 - open: URL: amqps://cap.example.com:5647|SSL: ca: /etc/rhsm/ca/katello-default-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
Jan 17 05:36:58 host.example.com goferd[29656]: [INFO][worker-0] gofer.agent.plugin:368 - plugin:katelloplugin, attached => pulp.agent.1f59d249-4042-4885-aa8f-8b6b071695fd
Jan 17 05:36:59 host.example.com goferd[29656]: [INFO][pulp.agent.1f59d249-4042-4885-aa8f-8b6b071695fd] gofer.messaging.adapter.proton.connection:92 - opened: proton+amqps://cap.example.com:5647
Jan 17 05:36:59 host.example.com goferd[29656]: [INFO][pulp.agent.1f59d249-4042-4885-aa8f-8b6b071695fd] gofer.messaging.adapter.connect:30 - connected: proton+amqps://cap.example.com:5647

>>> the error Condition('qd:no-route-to-dest', 'No route to the destination node') no longer appears after the Capsule<->Satellite connection is re-established

Comment 10 Pavel Moravec 2018-01-18 08:35:41 UTC
It was discovered that very similar symptoms (qdrouterd on Caps rejecting goferd connection with 'qd:no-route-to-dest' error despite everything is properly inter-connected) can appear even on fixed qdrouterd 0.4-22 or newer.

Since the underlying bug behind this BZ was properly fixed, I filed new:


https://bugzilla.redhat.com/show_bug.cgi?id=1535891


Please link cases from qdrouterd 0.4-22 or newer to _that_ BZ, to properly distinguish scenarios leading to either (separate) bug in code behind these two BZs.

Comment 16 errata-xmlrpc 2018-02-21 12:57:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0338