Created attachment 957670 [details] cat /etc/pulp/server.conf Description of problem: On new ec2 RHEL7 instance with fresh installed pulp 2.5 workers cannot connect to qpid broker, even though it's not over ssl Version-Release number of selected component (if applicable): >> uname -r 3.10.0-123.8.1.el7.x86_64 >> rpm -qa | grep pulp-server pulp-server-2.5.0-0.18.rc.el7.noarch >>rpm -qa | grep qpid qpid-proton-c-0.7-3.el7.x86_64 qpid-qmf-0.24-21.el7.x86_64 qpid-cpp-client-0.28-5.el7.x86_64 qpid-cpp-server-store-0.28-5.el7.x86_64 python-qpid-common-0.28-4.el7.noarch python-qpid-qmf-0.24-21.el7.x86_64 python-gofer-qpid-1.3.1-1.el7.noarch qpid-cpp-server-0.28-5.el7.x86_64 python-qpid-0.28-4.el7.noarch How reproducible: always Steps to Reproduce: >> pulp-admin rpm repo create --repo-id=foo --feed=https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/Successfully created repository [foo] >> pulp-admin -u admin -p admin rpm repo sync run --repo-id foo +----------------------------------------------------------------------+ Synchronizing Repository [foo] +----------------------------------------------------------------------+ An internal error occurred on the Pulp server: RequestException: POST request on /pulp/api/v2/repositories/foo/actions/sync/ failed with 500 - sasl negotiation failed: no mechanism agreed >>journalctl -a pulp.server.webservices.middleware.exception:ERROR: reply_to=reply_to or self.oid, **options Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/celery/app/amqp.py", line 305, in publish_task Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: **kwargs Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 168, in publish Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: routing_key, mandatory, immediate, exchange, declare) Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 461, in _ensured Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: interval_max) Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 373, in ensure_conne Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: interval_start, interval_step, interval_max, callback) Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 243, in retry_ov Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: return fun(*args, **kwargs) Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 241, in connect Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: return self.connection Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 758, in connection Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: self._connection = self._establish_connection() Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 713, in _establish_c Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: conn = self.transport.establish_connection() Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1673, in establi Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: conn = self.Connection(**opts) Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1365, in __init_ Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: self._qpid_conn = establish(**self.connection_options) Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 68, in estab Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: conn.open(timeout=timeout) Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "<string>", line 6, in open Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 268, in open Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: self.attach(timeout=timeout) Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "<string>", line 6, in attach Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 286, in atta Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeo Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 219, in _ewa Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: self.check_error() Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 212, in chec Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: raise e Nov 14 11:26:34 ip-10-75-35-247.eu-west-1.compute.internal pulp[1576]: pulp.server.webservices.middleware.exception:ERROR: AuthenticationFailure: sasl negotiation failed: no mechanism agreed Nov 14 11:26:35 ip-10-75-35-247.eu-west-1.compute.internal pulp[1444]: celery.worker.consumer:ERROR: consumer: Cannot connect to qpid://guest.0.1:5672//: sasl negotiation failed: no mechanism agreed. Nov 14 11:26:35 ip-10-75-35-247.eu-west-1.compute.internal pulp[1444]: celery.worker.consumer:ERROR: Trying again in 32.00 seconds... Nov 14 11:26:35 ip-10-75-35-247.eu-west-1.compute.internal pulp[1444]: celery.worker.consumer:ERROR: Nov 14 11:26:38 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: pulp.server.async.scheduler:ERROR: sasl negotiation failed: no mechanism agreed Nov 14 11:26:48 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: pulp.server.async.scheduler:ERROR: sasl negotiation failed: no mechanism agreed Nov 14 11:26:48 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: celery.beat:ERROR: beat: Connection error: sasl negotiation failed: no mechanism agreed. Trying again in 32.0 seconds... Nov 14 11:26:58 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: pulp.server.async.scheduler:ERROR: sasl negotiation failed: no mechanism agreed Nov 14 11:26:58 ip-10-75-35-247.eu-west-1.compute.internal pulp[1389]: celery.worker.consumer:ERROR: consumer: Cannot connect to qpid://guest.0.1:5672//: sasl negotiation failed: no mechanism agreed. Nov 14 11:26:58 ip-10-75-35-247.eu-west-1.compute.internal pulp[1389]: celery.worker.consumer:ERROR: Trying again in 32.00 seconds... Nov 14 11:26:58 ip-10-75-35-247.eu-west-1.compute.internal pulp[1389]: celery.worker.consumer:ERROR: Nov 14 11:27:07 ip-10-75-35-247.eu-west-1.compute.internal pulp[1444]: celery.worker.consumer:ERROR: consumer: Cannot connect to qpid://guest.0.1:5672//: sasl negotiation failed: no mechanism agreed. Nov 14 11:27:07 ip-10-75-35-247.eu-west-1.compute.internal pulp[1444]: celery.worker.consumer:ERROR: Trying again in 32.00 seconds... Nov 14 11:27:07 ip-10-75-35-247.eu-west-1.compute.internal pulp[1444]: celery.worker.consumer:ERROR: Nov 14 11:27:08 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: pulp.server.async.scheduler:ERROR: sasl negotiation failed: no mechanism agreed Nov 14 11:27:18 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: pulp.server.async.scheduler:ERROR: sasl negotiation failed: no mechanism agreed Nov 14 11:27:20 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: celery.beat:ERROR: beat: Connection error: sasl negotiation failed: no mechanism agreed. Trying again in 32.0 seconds... Nov 14 11:27:28 ip-10-75-35-247.eu-west-1.compute.internal pulp[1419]: pulp.server.async.scheduler:ERROR: sasl negotiation failed: no mechanism agreed Nov 14 11:27:30 ip-10-75-35-247.eu-west-1.compute.internal pulp[1389]: celery.worker.consumer:ERROR: consumer: Cannot connect to qpid://guest.0.1:5672//: sasl negotiation failed: no mechanism agreed. Nov 14 11:27:30 ip-10-75-35-247.eu-west-1.compute.internal pulp[1389]: celery.worker.consumer:ERROR: Trying again in 32.00 seconds... Nov 14 11:27:30 ip-10-75-35-247.eu-west-1.compute.internal pulp[1389]: celery.worker.consumer:ERROR:
Can you attach your /etc/qpid/qpidd.conf?
Also I think your machine may require the cyrus-sasl-plain. This was fixed in 2.5.1 so that you don't have to disable authentication to use Pulp, but for 2.5.0 I think you still require this package. See: https://bugzilla.redhat.com/show_bug.cgi?id=1099954
right, adding auth=no to qpidd.conf fixed that. Thanks!
This issue looks resolved so I'm closing it as not a bug. Irina if you think its still a bug please reopen.
This should be a bug, the default behavior is broken, it should default to auth=no if you don't believe this is a bug.
The "auth=no" setting is in a qpid config file, so if there's an argument to be made over what its default should be, please take it up with the qpid project.
Also, the upcoming 2.6.0 release will allow Pulp out-of-the-box to work with Qpid out-of-the-box using the ANONYMOUS SASL mechanism. By out-of-the-box I mean running a software without making any changes to a config file.