1141336 – "PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - parameters for queue 'reserved_resource_worker-0@fqdn.dq' in vhost '/' not equivalent" when using RabbitMQ

Bug 1141336 - "PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - parameters for queue 'reserved_resource_worker-0' in vhost '/' not equivalent" when using RabbitMQ

Summary: "PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - parameters fo...

Keywords:
Status:	CLOSED UPSTREAM
Alias:	None
Product:	Pulp
Classification:	Retired
Component:	z_other
Sub Component:
Version:	2.4.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	high
Target Milestone:	---
Target Release:	2.6.0
Assignee:	Brian Bouterse
QA Contact:	Irina Gulina
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2014-09-12 17:59 UTC by neoice
Modified:	2015-02-28 22:19 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2015-02-28 22:19:46 UTC
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Pulp Redmine	525	0	None	None	None	Never

Description neoice 2014-09-12 17:59:54 UTC

Description of problem:

when submitting jobs via `pulp-admin`, application 500s with "Unhandled Exception". syslog shows:

pulp: pulp.server.webservices.middleware.exception:ERROR: PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - parameters for queue 'reserved_resource_worker-0' in vhost '/' not equivalent

Version-Release number of selected component (if applicable):

Name        : pulp-server
Arch        : noarch
Version     : 2.4.0
Release     : 1.el6
Size        : 2.7 M


Steps to Reproduce:
1. using the Puppet module https://forge.puppetlabs.com/jcline/pulp:

  class { '::pulp::server':
    default_login    => 'admin',
    default_password => 'admin',
    msg_transport    => 'amqplib',
    msg_url          => 'tcp://127.0.0.1:5672',
    tasks_broker_url => 'amqp://127.0.0.1:5672',
    require          => [
                          Package['python-qpid-qmf'],
                          Package['python-amqplib'],
                          Package['python-gofer-amqplib'],
                          Service['mongod'],
                          Class['profiles::rabbitmq::pulp'],
                        ],  
  }

(there are some additional parts not included)

2. this results in the following being added to /etc/pulp/server.conf:

[messaging]
url: tcp://127.0.0.1:5672
transport: amqplib

[tasks]
broker_url: amqp://127.0.0.1:5672

3. restart all Pulp services

4. submit a job via `pulp-admin`:

pulp-admin rpm repo sync run --repo-id jenkins

Actual results:

An internal error occurred on the Pulp server:

RequestException: POST request
on /pulp/api/v2/repositories/jenkins/actions/sync/ failed with 500 - Unhandled
Exception


Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: Traceback (most recent call last):
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.app(environ, start_response)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/postponed.py", line 42, in __call__
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.app(environ, start_response)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 279, in wsgi
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     result = self.handle_with_processors()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors
1 pulp: pulp.server.webservices.middleware.exception:ERROR:     return process(self.processors)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return handler()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     result = handler()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 28, in process
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.handle()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return self._delegate(fn, self.fvars, args)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 422, in _delegate
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return f()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 430, in <lambda>
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     f = lambda: self._delegate_sub_application(pat, what)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 455, in _delegate_sub_application
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return app.handle_with_processors()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_procesors
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return process(self.processors)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return handler()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     result = handler()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 28, in process
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.handle()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return self._delegate(fn, self.fvars, args)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 420, in _delegate
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return handle_class(cls)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 396, in handle_class
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return tocall(*args)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/decorators.py", line 227, in _auth_decorator
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     value = method(self, *args, **kwargs)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/repositories.py", line 710, in POST
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     repo_id, [repo_id, overrides], {}, tags=task_tags)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 240, in apply_async_with_reservation
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     _queue_release_resource.apply_async((resource_id,), queue=queue)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/celery/app/task.py", line 555, in apply_async
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     **dict(self._get_exec_options(), **options)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/celery/app/base.py", line 351, in send_task
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     reply_to=reply_to or self.oid, **options
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/celery/app/amqp.py", line 305, in publish_task
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     **kwargs
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 168, in publish
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     routing_key, mandatory, immediate, exchange, declare)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 440, in _ensured
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return fun(*args, **kwargs)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 180, in _publish
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     [maybe_declare(entity) for entity in declare]
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 111, in maybe_declare
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return maybe_declare(entity, self.channel, retry, **retry_policy)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/kombu/common.py", line 99, in maybe_declare
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return _maybe_declare(entity)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/kombu/common.py", line 110, in _maybe_declare
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     entity.declare()
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/kombu/entity.py", line 505, in declare
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     self.queue_declare(nowait, passive=False)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/kombu/entity.py", line 531, in queue_declare
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     nowait=nowait)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/amqp/channel.py", line 1258, in queue_declare
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     (50, 11),  # Channel.queue_declare_ok
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/amqp/abstract_channel.py", line 69, in wait
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.dispatch_method(method_sig, args, content)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     return amqp_method(self, args)
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/amqp/channel.py", line 241, in _close
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR:     reply_code, reply_text, (class_id, method_id), ChannelError,
Sep 12 17:57:08 pulp-test pulp: pulp.server.webservices.middleware.exception:ERROR: PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - parameters for queue 'reserved_resource_worker-0.smartsheet.com.dq' in vhost '/' not equivalent

Expected results:

queue should be created properly and job should be submitted and run without error.

Additional info:

using Puppet+TheForeman, I can rebuild my host fully from kickstart to this error state. I intend to test this bug by destroying my Pulp VM and recreating it until it works :)

Comment 1 Randy Barlow 2014-09-19 13:47:44 UTC

Brian and I spent some time trying to figure this out yesterday, and we believe it is a bug in Celery[0].

The good news is that we found a way to work around the issue, and we've included that fix in PR #1158[1].

Thanks to Brian for taking the time to sort this out with me for the price of a can of La Croix!

[0] https://github.com/celery/celery/issues/2269
[1] https://github.com/pulp/pulp/pull/1158

Comment 2 Brian Bouterse 2014-09-19 14:54:04 UTC

Merged to 2.5-dev, master

Comment 3 Chris Duryee 2014-12-23 20:52:38 UTC

fixed in pulp 2.6.0-0.2.beta

Comment 4 Brian Bouterse 2015-02-28 22:19:46 UTC

Moved to https://pulp.plan.io/issues/525

Note You need to log in before you can comment on or make changes to this bug.