Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1164324

Summary:

workers cannot connect to qpid broker

Product:

[Retired] Pulp

Reporter:

Irina Gulina <igulina>

Component:

async/tasks

Assignee:

pulp-bugs

Status:

CLOSED NOTABUG

QA Contact:

pulp-qe-list

Severity:

unspecified

Docs Contact:

Priority:

unspecified

Version:

2.5

CC:

bmbouter, cduryee, igulina, jon, mhrivnak

Target Milestone:

---

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2014-11-19 14:19:07 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
cat /etc/pulp/server.conf	none

Description Irina Gulina 2014-11-14 16:36:48 UTC

Created attachment 957670 [details]
cat /etc/pulp/server.conf

Description of problem:
On new ec2 RHEL7 instance with fresh installed pulp 2.5 workers cannot connect to qpid broker, even though it's not over ssl

Version-Release number of selected component (if applicable):
>> uname -r
3.10.0-123.8.1.el7.x86_64
>> rpm -qa | grep pulp-server
pulp-server-2.5.0-0.18.rc.el7.noarch
>>rpm -qa | grep qpid
qpid-proton-c-0.7-3.el7.x86_64
qpid-qmf-0.24-21.el7.x86_64
qpid-cpp-client-0.28-5.el7.x86_64
qpid-cpp-server-store-0.28-5.el7.x86_64
python-qpid-common-0.28-4.el7.noarch
python-qpid-qmf-0.24-21.el7.x86_64
python-gofer-qpid-1.3.1-1.el7.noarch
qpid-cpp-server-0.28-5.el7.x86_64
python-qpid-0.28-4.el7.noarch


How reproducible:
always

Steps to Reproduce:
>> pulp-admin rpm repo create --repo-id=foo --feed=https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/Successfully created repository [foo]

>> pulp-admin -u admin -p admin rpm repo sync run --repo-id foo
+----------------------------------------------------------------------+
                     Synchronizing Repository [foo]
+----------------------------------------------------------------------+

An internal error occurred on the Pulp server:


RequestException: POST request
on /pulp/api/v2/repositories/foo/actions/sync/ failed with 500 - sasl
negotiation failed: no mechanism agreed

>>journalctl -a
 pulp.server.webservices.middleware.exception:ERROR:     reply_to=reply_to or self.oid, **options
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/celery/app/amqp.py", line 305, in publish_task
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     **kwargs
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 168, in publish
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     routing_key, mandatory, immediate, exchange, declare)
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 461, in _ensured
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     interval_max)
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 373, in ensure_conne
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     interval_start, interval_step, interval_max, callback)
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 243, in retry_ov
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     return fun(*args, **kwargs)
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 241, in connect
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     return self.connection
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 758, in connection
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     self._connection = self._establish_connection()
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 713, in _establish_c
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     conn = self.transport.establish_connection()
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1673, in establi
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     conn = self.Connection(**opts)
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1365, in __init_
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     self._qpid_conn = establish(**self.connection_options)
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 68, in estab
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     conn.open(timeout=timeout)
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "<string>", line 6, in open
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 268, in open
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     self.attach(timeout=timeout)
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "<string>", line 6, in attach
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 286, in atta
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeo
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 219, in _ewa
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     self.check_error()
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 212, in chec
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR:     raise e
Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: AuthenticationFailure: sasl negotiation failed: no mechanism agreed
Nov 14 11:26:35 ip-10-75-35-247.eu-west-1.compute.internal pulp[1444]: celery.worker.consumer:ERROR: consumer: Cannot connect to qpid://guest.0.1:5672//: sasl negotiation failed: no mechanism agreed.
Nov 14 11:26:35 ip-10-75-35-247.eu-west-1.compute.internal pulp[1444]: celery.worker.consumer:ERROR: Trying again in 32.00 seconds...
Nov 14 11:26:35 ip-10-75-35-247.eu-west-1.compute.internal pulp[1444]: celery.worker.consumer:ERROR:
Nov 14 11:26:38 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: pulp.server.async.scheduler:ERROR: sasl negotiation failed: no mechanism agreed
Nov 14 11:26:48 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: pulp.server.async.scheduler:ERROR: sasl negotiation failed: no mechanism agreed
Nov 14 11:26:48 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: celery.beat:ERROR: beat: Connection error: sasl negotiation failed: no mechanism agreed. Trying again in 32.0 seconds...
Nov 14 11:26:58 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: pulp.server.async.scheduler:ERROR: sasl negotiation failed: no mechanism agreed
Nov 14 11:26:58 ip-10-75-35-247.eu-west-1.compute.internal pulp[1389]: celery.worker.consumer:ERROR: consumer: Cannot connect to qpid://guest.0.1:5672//: sasl negotiation failed: no mechanism agreed.
Nov 14 11:26:58 ip-10-75-35-247.eu-west-1.compute.internal pulp[1389]: celery.worker.consumer:ERROR: Trying again in 32.00 seconds...
Nov 14 11:26:58 ip-10-75-35-247.eu-west-1.compute.internal pulp[1389]: celery.worker.consumer:ERROR:
Nov 14 11:27:07 ip-10-75-35-247.eu-west-1.compute.internal pulp[1444]: celery.worker.consumer:ERROR: consumer: Cannot connect to qpid://guest.0.1:5672//: sasl negotiation failed: no mechanism agreed.
Nov 14 11:27:07 ip-10-75-35-247.eu-west-1.compute.internal pulp[1444]: celery.worker.consumer:ERROR: Trying again in 32.00 seconds...
Nov 14 11:27:07 ip-10-75-35-247.eu-west-1.compute.internal pulp[1444]: celery.worker.consumer:ERROR:
Nov 14 11:27:08 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: pulp.server.async.scheduler:ERROR: sasl negotiation failed: no mechanism agreed
Nov 14 11:27:18 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: pulp.server.async.scheduler:ERROR: sasl negotiation failed: no mechanism agreed
Nov 14 11:27:20 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: celery.beat:ERROR: beat: Connection error: sasl negotiation failed: no mechanism agreed. Trying again in 32.0 seconds...
Nov 14 11:27:28 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: pulp.server.async.scheduler:ERROR: sasl negotiation failed: no mechanism agreed
Nov 14 11:27:30 ip-10-75-35-247.eu-west-1.compute.internal pulp[1389]: celery.worker.consumer:ERROR: consumer: Cannot connect to qpid://guest.0.1:5672//: sasl negotiation failed: no mechanism agreed.
Nov 14 11:27:30 ip-10-75-35-247.eu-west-1.compute.internal pulp[1389]: celery.worker.consumer:ERROR: Trying again in 32.00 seconds...
Nov 14 11:27:30 ip-10-75-35-247.eu-west-1.compute.internal pulp[1389]: celery.worker.consumer:ERROR:

Comment 1 Chris Duryee 2014-11-14 22:15:20 UTC

Can you attach your /etc/qpid/qpidd.conf?

Comment 2 Brian Bouterse 2014-11-17 12:57:03 UTC

Also I think your machine may require the cyrus-sasl-plain. This was fixed in 2.5.1 so that you don't have to disable authentication to use Pulp, but for 2.5.0 I think you still require this package.

See:  https://bugzilla.redhat.com/show_bug.cgi?id=1099954

Comment 3 Irina Gulina 2014-11-19 13:37:06 UTC

right, adding auth=no to qpidd.conf fixed that. Thanks!

Comment 4 Brian Bouterse 2014-11-19 14:19:07 UTC

This issue looks resolved so I'm closing it as not a bug. Irina if you think its still a bug please reopen.

Comment 5 Jonathan Mainguy 2015-01-03 03:42:29 UTC

This should be a bug, the default behavior is broken, it should default to auth=no if you don't believe this is a bug.

Comment 6 Michael Hrivnak 2015-01-05 13:31:36 UTC

The "auth=no" setting is in a qpid config file, so if there's an argument to be made over what its default should be, please take it up with the qpid project.

Comment 7 Brian Bouterse 2015-01-05 15:16:45 UTC

Also, the upcoming 2.6.0 release will allow Pulp out-of-the-box to work with Qpid out-of-the-box using the ANONYMOUS SASL mechanism. By out-of-the-box I mean running a software without making any changes to a config file.