Description of problem: Was running Pulp 2.3.1, then updated to 2.4 beta 18 via yum update. After resolving an initial issue whereby the qpid-tools package was required (but not a dependency of the Pulp packages), running pulp-manage-db results in the following failure: stdout/stderr output from command: [root@myhostname ~]# sudo -u apache /usr/bin/pulp-manage-db Loading content types. /usr/lib/python2.6/site-packages/pulp/server/db/connection.py:135: DeprecationWarning: The safe parameter is deprecated. Please use write concern options instead. return method(*args, **kwargs) Content types loaded. Ensuring the admin role and user are in place. Admin role and user are in place. Beginning database migrations. Applying pulp.server.db.migrations version 9 Applying migration pulp.server.db.migrations.0009_qpid_queues failed. See log for details. Migration package pulp_puppet.plugins.migrations is up to date at version 2 Migration package pulp_rpm.plugins.migrations is up to date at version 15 Database migrations complete. /var/log/messages events: May 22 11:24:50 myhostname pulp: pulp.server.db.manage:INFO: Loading content types. May 22 11:24:50 myhostname pulp: db:INFO: Loading type descriptors [iso_support.json, rpm_support.json, puppet.json] May 22 11:24:50 myhostname pulp: db:INFO: Parsing type descriptors May 22 11:24:50 myhostname pulp: db:INFO: Validating type descriptor syntactic integrity May 22 11:24:50 myhostname pulp: db:INFO: Validating type descriptor semantic integrity May 22 11:24:50 myhostname pulp: db:INFO: Updating the database with types [iso, distribution, drpm, erratum, package_group, package_category, package_environment, rpm, srpm, yum_repo_metadata_file, pup pet_module] May 22 11:24:53 myhostname pulp: pulp.server.db.manage:INFO: Content types loaded. May 22 11:24:53 myhostname pulp: pulp.server.db.manage:INFO: Ensuring the admin role and user are in place. May 22 11:24:53 myhostname pulp: pulp.server.db.connection:INFO: Attempting Database connection with seeds = localhost:27017 May 22 11:24:53 myhostname pulp: pulp.server.db.connection:INFO: Connection Arguments: {'max_pool_size': 10} May 22 11:24:53 myhostname pulp: pulp.server.db.connection:INFO: Database connection established with: seeds = localhost:27017, name = pulp_database May 22 11:24:53 myhostname pulp: pulp.server.db.manage:INFO: Admin role and user are in place. May 22 11:24:53 myhostname pulp: pulp.server.db.manage:INFO: Beginning database migrations. May 22 11:24:54 myhostname pulp: pulp.server.db.manage:INFO: Applying pulp.server.db.migrations version 9 May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: Applying migration pulp.server.db.migrations.0009_qpid_queues failed. May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: Cannot delete queue kvm-guest-70-consumer; queue in use May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: Traceback (most recent call last): May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 86, in migrate_database May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: update_current_version=not options.test) May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp/server/db/migrate/models.py", line 161, in apply_migration May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: migration.migrate() May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp/server/db/migrations/0009_qpid_queues.py", line 70, in migrate May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: _migrate_agent_queues(broker) May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp/server/db/migrations/0009_qpid_queues.py", line 97, in _migrate_agent_queues May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: _del_agent_queues(broker) May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp/server/db/migrations/0009_qpid_queues.py", line 129, in _del_agent_queues May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: broker.delQueue(name) May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/qpidtoollibs/broker.py", line 258, in delQueue May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: self._tmpQueueCheck(name, options) May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/qpidtoollibs/broker.py", line 286, in _tmpQueueCheck May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: raise Exception(msg) May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: Exception: Cannot delete queue kvm-guest-70-consumer; queue in use May 22 11:24:54 myhostname pulp: pulp.server.db.manage:CRITICAL: May 22 11:24:54 myhostname pulp: pulp.server.db.manage:INFO: Migration package pulp_puppet.plugins.migrations is up to date at version 2 May 22 11:24:54 myhostname pulp: pulp.server.db.manage:INFO: Migration package pulp_rpm.plugins.migrations is up to date at version 15 May 22 11:24:54 myhostname pulp: pulp.server.db.manage:INFO: Database migrations complete. I have these qpid related package versions installed: qpid-cpp-client-0.18-20.el6.x86_64 python-gofer-qpid-1.0.13-1.el6.noarch qpid-tools-0.18-10.el6_4.noarch qpid-cpp-server-0.18-20.el6.x86_64 python-qpid-0.18-9.el6.noarch qpid-qmf-0.18-20.el6.x86_64 python-qpid-qmf-0.18-20.el6.x86_64 I have these pulp package versions installed: [root@myhostname ~]# rpm -qa | grep pulp m2crypto-0.21.1.pulp-8.el6.x86_64 python-pulp-bindings-2.4.0-0.18.beta.el6.noarch pulp-server-2.4.0-0.18.beta.el6.noarch pulp-admin-client-2.4.0-0.18.beta.el6.noarch pulp-rpm-plugins-2.4.0-0.18.beta.el6.noarch python-isodate-0.5.0-1.pulp.el6.noarch python-pulp-common-2.4.0-0.18.beta.el6.noarch python-pulp-puppet-common-2.4.0-0.18.beta.el6.noarch createrepo-0.9.9-21.2.pulp.el6.noarch pulp-nodes-common-2.4.0-0.18.beta.el6.noarch python-pulp-client-lib-2.4.0-0.18.beta.el6.noarch pulp-rpm-admin-extensions-2.4.0-0.18.beta.el6.noarch pulp-nodes-admin-extensions-2.4.0-0.18.beta.el6.noarch pulp-nodes-parent-2.4.0-0.18.beta.el6.noarch pulp-selinux-2.4.0-0.18.beta.el6.noarch python-kombu-3.0.15-5.pulp.el6.noarch python-pulp-rpm-common-2.4.0-0.18.beta.el6.noarch pulp-puppet-admin-extensions-2.4.0-0.18.beta.el6.noarch pulp-puppet-plugins-2.4.0-0.18.beta.el6.noarch mod_wsgi-3.4-1.pulp.el6.x86_64 Version-Release number of selected component (if applicable): 2.4 beta 18 How reproducible: Steps to Reproduce: 1. Run pulp 2.3.1 add repositories, use for an indeterminate period of time with an indeterminate number of repositories, have at least one other host as a consumer (presumably since the failure relates to a consumer queue). 2. Point yum config at the 2.4 beta 18 repo 3. yum update 4. resolve any dependency issues 5. run pulp-manage-db Actual results: pulp-manage-db results in failure message Applying migration pulp.server.db.migrations.0009_qpid_queues failed, syslog contains error details. pulp does not work. Expected results: completes without failure reported, able to restart httpd and use pulp Additional info:
We believe this is a documentation and error reporting issue. It turns out that users must upgrade and restart the consumer side of Pulp before upgrading the server side. We'll keep this bug open so that we can do these two things: 1) Add to the upgrade documentation steps about shutting down or restarting consumers during the server upgrade. 2) Add better error reporting to the migration so that the user is informed that they need to upgrade and stop or restart their consumers before the migration can succeed. This error reporting could also link to the docs from #1.
PR available here: https://github.com/pulp/pulp/pull/1001/files
Merged to pulp-2.4 and master
build: 2.4.0-0.20.beta
verified https://pulp-user-guide.readthedocs.org/en/latest/release-notes/2.4.x.html#upgrade-instructions-for-2-3-x-2-4-0
This has been fixed in Pulp 2.4.0-1.