Bug 1100610 - pulp-manage-db fails on migration 0009_qpid_queues - Cannot delete queue
Summary: pulp-manage-db fails on migration 0009_qpid_queues - Cannot delete queue
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Pulp
Classification: Retired
Component: rel-eng
Version: 2.4 Beta
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: ---
: 2.4.0
Assignee: Brian Bouterse
QA Contact: Preethi Thomas
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-05-23 08:26 UTC by Paul Urwin
Modified: 2014-08-09 06:56 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-08-09 06:56:55 UTC
Embargoed:


Attachments (Terms of Use)

Description Paul Urwin 2014-05-23 08:26:56 UTC
Description of problem:

Was running Pulp 2.3.1, then updated to 2.4 beta 18 via yum update.

After resolving an initial issue whereby the qpid-tools package was required (but not a dependency of the Pulp packages), running pulp-manage-db results in the following failure:

stdout/stderr output from command:

[root@myhostname ~]# sudo -u apache /usr/bin/pulp-manage-db Loading content types.
/usr/lib/python2.6/site-packages/pulp/server/db/connection.py:135: DeprecationWarning: The safe parameter is deprecated. Please use write concern options instead.
  return method(*args, **kwargs)
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Applying pulp.server.db.migrations version 9 Applying migration pulp.server.db.migrations.0009_qpid_queues failed.  See log for details.
Migration package pulp_puppet.plugins.migrations is up to date at version 2 Migration package pulp_rpm.plugins.migrations is up to date at version 15 Database migrations complete.


/var/log/messages events:

May 22 11:24:50 myhostname pulp: pulp.server.db.manage:INFO: Loading content types.
May 22 11:24:50 myhostname pulp: db:INFO: Loading type descriptors [iso_support.json, rpm_support.json, puppet.json] 
May 22 11:24:50 myhostname pulp: db:INFO: Parsing type descriptors 
May 22 11:24:50 myhostname pulp: db:INFO: Validating type descriptor syntactic integrity 
May 22 11:24:50 myhostname pulp: db:INFO: Validating type descriptor semantic integrity 
May 22 11:24:50 myhostname pulp: db:INFO: Updating the database with types [iso, distribution, drpm, erratum, package_group, package_category, package_environment, rpm, srpm, yum_repo_metadata_file, pup pet_module] 
May 22 11:24:53 myhostname pulp: pulp.server.db.manage:INFO: Content types loaded.
May 22 11:24:53 myhostname pulp: pulp.server.db.manage:INFO: Ensuring the admin role and user are in place.
May 22 11:24:53 myhostname pulp: pulp.server.db.connection:INFO: Attempting Database connection with seeds = localhost:27017 
May 22 11:24:53 myhostname pulp: pulp.server.db.connection:INFO: Connection Arguments: {'max_pool_size': 10} 
May 22 11:24:53 myhostname pulp: pulp.server.db.connection:INFO: Database connection established with: seeds = localhost:27017, name = pulp_database 
May 22 11:24:53 myhostname pulp: pulp.server.db.manage:INFO: Admin role and user are in place.
May 22 11:24:53 myhostname pulp: pulp.server.db.manage:INFO: Beginning database migrations.
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:INFO: Applying pulp.server.db.migrations version 9 
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: Applying migration pulp.server.db.migrations.0009_qpid_queues failed.
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: Cannot delete queue kvm-guest-70-consumer; queue in use 
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: Traceback (most recent call last):
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 86, in migrate_database
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:     update_current_version=not options.test)
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp/server/db/migrate/models.py", line 161, in apply_migration
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:     migration.migrate()
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp/server/db/migrations/0009_qpid_queues.py", line 70, in migrate
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:     _migrate_agent_queues(broker)
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp/server/db/migrations/0009_qpid_queues.py", line 97, in _migrate_agent_queues
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:     _del_agent_queues(broker)
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp/server/db/migrations/0009_qpid_queues.py", line 129, in _del_agent_queues
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:     broker.delQueue(name)
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/qpidtoollibs/broker.py", line 258, in delQueue
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:     self._tmpQueueCheck(name, options)
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/qpidtoollibs/broker.py", line 286, in _tmpQueueCheck
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:     raise Exception(msg)
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: Exception: Cannot delete queue kvm-guest-70-consumer; queue in use May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL:
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:INFO: Migration package pulp_puppet.plugins.migrations is up to date at version 2 
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:INFO: Migration package pulp_rpm.plugins.migrations is up to date at version 15 
May 22 11:24:54 myhostname pulp: pulp.server.db.manage:INFO: Database migrations complete.



I have these qpid related package versions installed:

qpid-cpp-client-0.18-20.el6.x86_64
python-gofer-qpid-1.0.13-1.el6.noarch
qpid-tools-0.18-10.el6_4.noarch
qpid-cpp-server-0.18-20.el6.x86_64
python-qpid-0.18-9.el6.noarch
qpid-qmf-0.18-20.el6.x86_64
python-qpid-qmf-0.18-20.el6.x86_64

I have these pulp package versions installed:

[root@myhostname ~]# rpm -qa | grep pulp
m2crypto-0.21.1.pulp-8.el6.x86_64
python-pulp-bindings-2.4.0-0.18.beta.el6.noarch
pulp-server-2.4.0-0.18.beta.el6.noarch
pulp-admin-client-2.4.0-0.18.beta.el6.noarch
pulp-rpm-plugins-2.4.0-0.18.beta.el6.noarch
python-isodate-0.5.0-1.pulp.el6.noarch
python-pulp-common-2.4.0-0.18.beta.el6.noarch
python-pulp-puppet-common-2.4.0-0.18.beta.el6.noarch
createrepo-0.9.9-21.2.pulp.el6.noarch
pulp-nodes-common-2.4.0-0.18.beta.el6.noarch
python-pulp-client-lib-2.4.0-0.18.beta.el6.noarch
pulp-rpm-admin-extensions-2.4.0-0.18.beta.el6.noarch
pulp-nodes-admin-extensions-2.4.0-0.18.beta.el6.noarch
pulp-nodes-parent-2.4.0-0.18.beta.el6.noarch
pulp-selinux-2.4.0-0.18.beta.el6.noarch
python-kombu-3.0.15-5.pulp.el6.noarch
python-pulp-rpm-common-2.4.0-0.18.beta.el6.noarch
pulp-puppet-admin-extensions-2.4.0-0.18.beta.el6.noarch
pulp-puppet-plugins-2.4.0-0.18.beta.el6.noarch
mod_wsgi-3.4-1.pulp.el6.x86_64



Version-Release number of selected component (if applicable): 2.4 beta 18


How reproducible:



Steps to Reproduce:
1. Run pulp 2.3.1 add repositories, use for an indeterminate period of time with an indeterminate number of repositories, have at least one other host as a consumer (presumably since the failure relates to a consumer queue). 
2. Point yum config at the 2.4 beta 18 repo
3. yum update
4. resolve any dependency issues
5. run pulp-manage-db



Actual results:

pulp-manage-db results in failure message Applying migration pulp.server.db.migrations.0009_qpid_queues failed, syslog contains error details.  pulp does not work.

Expected results:

completes without failure reported, able to restart httpd and use pulp

Additional info:

Comment 1 Randy Barlow 2014-05-23 14:17:52 UTC
We believe this is a documentation and error reporting issue. It turns out that users must upgrade and restart the consumer side of Pulp before upgrading the server side.

We'll keep this bug open so that we can do these two things:

1) Add to the upgrade documentation steps about shutting down or restarting consumers during the server upgrade.

2) Add better error reporting to the migration so that the user is informed that they need to upgrade and stop or restart their consumers before the migration can succeed. This error reporting could also link to the docs from #1.

Comment 2 Brian Bouterse 2014-06-09 20:39:14 UTC
PR available here:

https://github.com/pulp/pulp/pull/1001/files

Comment 3 Brian Bouterse 2014-06-10 19:59:45 UTC
Merged to pulp-2.4 and master

Comment 4 Jeff Ortel 2014-06-11 01:33:04 UTC
build: 2.4.0-0.20.beta

Comment 6 Randy Barlow 2014-08-09 06:56:55 UTC
This has been fixed in Pulp 2.4.0-1.


Note You need to log in before you can comment on or make changes to this bug.