Created attachment 891907 [details] screen log Description of problem: not stopping on a failure buries possible error logs and might render the database state undefined Version-Release number of selected component (if applicable): pulp-server-2.4.0-0.10.beta.fc20.noarch How reproducible: Always Steps to Reproduce: 1. mongo pulp_database --eval "db.dropDatabase()" 2. systemctl stop qpidd.service 3. sudo -u apache pulp-manage-db Actual results: all migrations applied no matter one failed Expected results: migration process stops at first failure Additional info: # screen log attached
I think we should retriage this one and consider medium priority.
I also think we should re-triage this to medium for 3 reasons: 1) We've had two developers say "the migrations work", but it still gives the error message: "pulp.server.db.migrate.models:ERROR: pulp.server.db.migrations hasn't been updated to the latest available migration". The repetition from multiple developers means this is truly hard to miss, and it wastes time when we need to help them debug it. 2) It won't be hard to add a do-not-continue behavior to the migration runner. 3) It won't be hard to QE this change because running the migrations on a fresh install with qpidd not running will cause migration 9 to fail.
https://github.com/pulp/pulp/pull/1116
https://github.com/pulp/pulp/pull/1117 PR #1116 was closed and reopened against the new 2.4.1-dev branch.
To verify: 1. Follow the reproduction steps 2. The migrations should now halt at migration 9 and make it clear there was an error. The logs should include the traceback.
This was fixed in 2.4.1-0.2.alpha, available in Pulp's testing repository.
[root@qe-blade-08 module]# mongo pulp_database --eval "db.dropDatabase()" MongoDB shell version: 2.4.6 connecting to: pulp_database [object Object] [root@qe-blade-08 module]# systemctl stop qpidd.service -bash: systemctl: command not found [root@qe-blade-08 module]# [root@qe-blade-08 module]# [root@qe-blade-08 module]# service qpidd stop Stopping Qpid AMQP daemon: [ OK ] [root@qe-blade-08 module]# [root@qe-blade-08 module]# [root@qe-blade-08 module]# sudo -u apache pulp-manage-db Loading content types. /usr/lib/python2.6/site-packages/pulp/server/db/connection.py:133: DeprecationWarning: The safe parameter is deprecated. Please use write concern options instead. return method(*args, **kwargs) Content types loaded. Ensuring the admin role and user are in place. Admin role and user are in place. Beginning database migrations. Applying pulp.server.db.migrations version 1 Migration to pulp.server.db.migrations version 1 complete. Applying pulp.server.db.migrations version 2 Migration to pulp.server.db.migrations version 2 complete. Applying pulp.server.db.migrations version 3 Migration to pulp.server.db.migrations version 3 complete. Applying pulp.server.db.migrations version 4 Migration to pulp.server.db.migrations version 4 complete. Applying pulp.server.db.migrations version 5 Migration to pulp.server.db.migrations version 5 complete. Applying pulp.server.db.migrations version 6 Migration to pulp.server.db.migrations version 6 complete. Applying pulp.server.db.migrations version 7 Migration to pulp.server.db.migrations version 7 complete. Applying pulp.server.db.migrations version 8 Migration to pulp.server.db.migrations version 8 complete. Applying pulp.server.db.migrations version 9 Applying migration pulp.server.db.migrations.0009_qpid_queues failed. Halting migrations due to a migration failure. See log for details. [Errno 111] Connection refused [root@qe-blade-08 module]#
verified [root@qe-blade-08 module]# rpm -qa pulp-server pulp-server-2.4.1-0.2.alpha.el6.noarch [root@qe-blade-08 module]# [root@qe-blade-13 module]# rpm -qa pulp-server pulp-server-2.4.1-0.2.alpha.el7.noarch [root@qe-blade-13 module]# also on Rhel7 [root@qe-blade-13 ~]# sudo -u apache pulp-manage-db Loading content types. Content types loaded. Ensuring the admin role and user are in place. Admin role and user are in place. Beginning database migrations. Applying pulp.server.db.migrations version 1 Migration to pulp.server.db.migrations version 1 complete. Applying pulp.server.db.migrations version 2 Migration to pulp.server.db.migrations version 2 complete. Applying pulp.server.db.migrations version 3 Migration to pulp.server.db.migrations version 3 complete. Applying pulp.server.db.migrations version 4 Migration to pulp.server.db.migrations version 4 complete. Applying pulp.server.db.migrations version 5 Migration to pulp.server.db.migrations version 5 complete. Applying pulp.server.db.migrations version 6 Migration to pulp.server.db.migrations version 6 complete. Applying pulp.server.db.migrations version 7 Migration to pulp.server.db.migrations version 7 complete. Applying pulp.server.db.migrations version 8 Migration to pulp.server.db.migrations version 8 complete. Applying pulp.server.db.migrations version 9 Applying migration pulp.server.db.migrations.0009_qpid_queues failed. Halting migrations due to a migration failure. See log for details. [Errno 111] Connection refused [root@qe-blade-13 ~]# service qpidd start Redirecting to /bin/systemctl start qpidd.service [root@qe-blade-13 ~]# [root@qe-blade-13 ~]# [root@qe-blade-13 ~]# sudo -u apache pulp-manage-db Loading content types. Content types loaded. Ensuring the admin role and user are in place. Admin role and user are in place. Beginning database migrations. Applying pulp.server.db.migrations version 9 Migration to pulp.server.db.migrations version 9 complete. Applying pulp_puppet.plugins.migrations version 1 Migration to pulp_puppet.plugins.migrations version 1 complete. Applying pulp_puppet.plugins.migrations version 2 Migration to pulp_puppet.plugins.migrations version 2 complete. Applying pulp_rpm.plugins.migrations version 1 Migration to pulp_rpm.plugins.migrations version 1 complete. Applying pulp_rpm.plugins.migrations version 2 Migration to pulp_rpm.plugins.migrations version 2 complete. Applying pulp_rpm.plugins.migrations version 3 Migration to pulp_rpm.plugins.migrations version 3 complete. Applying pulp_rpm.plugins.migrations version 4 Migration to pulp_rpm.plugins.migrations version 4 complete. Applying pulp_rpm.plugins.migrations version 5 Migration to pulp_rpm.plugins.migrations version 5 complete. Applying pulp_rpm.plugins.migrations version 6 Migration to pulp_rpm.plugins.migrations version 6 complete. Applying pulp_rpm.plugins.migrations version 7 Migration to pulp_rpm.plugins.migrations version 7 complete. Applying pulp_rpm.plugins.migrations version 8 Migration to pulp_rpm.plugins.migrations version 8 complete. Applying pulp_rpm.plugins.migrations version 9 Migration to pulp_rpm.plugins.migrations version 9 complete. Applying pulp_rpm.plugins.migrations version 10 Migration to pulp_rpm.plugins.migrations version 10 complete. Applying pulp_rpm.plugins.migrations version 11 Migration to pulp_rpm.plugins.migrations version 11 complete. Applying pulp_rpm.plugins.migrations version 12 Migration to pulp_rpm.plugins.migrations version 12 complete. Applying pulp_rpm.plugins.migrations version 13 Migration to pulp_rpm.plugins.migrations version 13 complete. Applying pulp_rpm.plugins.migrations version 14 Migration to pulp_rpm.plugins.migrations version 14 complete. Applying pulp_rpm.plugins.migrations version 15 Migration to pulp_rpm.plugins.migrations version 15 complete. Applying pulp_rpm.plugins.migrations version 16 Migration to pulp_rpm.plugins.migrations version 16 complete. Database migrations complete. [root@qe-blade-13 ~]#
This is fixed in Pulp-2.4.1-1.