When trying to install OpenStack Icehouse (rhos-5.0), using packstack, it fails on keystone puppet, based on the keystone.log it failed when applying DB MIGRATIONS. It seems that python-sqlalchemy got updated to version 0.9.8 in RHEL-7.2/Server, and is used instead of the one shipped with RHOS-5.0 which is 0.8.4. Based on old Debian OpenStack packaging mail archives it could be the reason why migration is failing (and so keystone.pp and whole installation), as they mentioned that at that time used python-migrate (among others) is not supporting newer sqlalchemy version. https://lists.debian.org/debian-release/2014/02/msg00477.html Snippet from keystone.log: > 2015-09-18 05:11:33.025 8475 INFO migrate.versioning.api [-] done > 2015-09-18 05:11:33.025 8475 INFO migrate.versioning.api [-] 10 -> 11... > 2015-09-18 05:11:33.031 8475 CRITICAL keystone [-] AttributeError: 'unicode' object has no attribute 'quote' > 2015-09-18 05:11:33.031 8475 TRACE keystone Traceback (most recent call last): > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/bin/keystone-manage", line 51, in <module> > 2015-09-18 05:11:33.031 8475 TRACE keystone cli.main(argv=sys.argv, config_files=config_files) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/keystone/cli.py", line 190, in main > 2015-09-18 05:11:33.031 8475 TRACE keystone CONF.command.cmd_class.main() > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/keystone/cli.py", line 66, in main > 2015-09-18 05:11:33.031 8475 TRACE keystone migration_helpers.sync_database_to_version(extension, version) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 160, in sync_database_to_version > 2015-09-18 05:11:33.031 8475 TRACE keystone migration.db_sync(engine, abs_path, version=version) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/keystone/openstack/common/db/sqlalchemy/migration.py", line 197, in db_sync > 2015-09-18 05:11:33.031 8475 TRACE keystone return versioning_api.upgrade(engine, repository, version) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade > 2015-09-18 05:11:33.031 8475 TRACE keystone return _migrate(url, repository, version, upgrade=True, err=err, **opts) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "<string>", line 2, in _migrate > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 159, in with_engine > 2015-09-18 05:11:33.031 8475 TRACE keystone return f(*a, **kw) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate > 2015-09-18 05:11:33.031 8475 TRACE keystone schema.runchange(ver, change, changeset.step) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 91, in runchange > 2015-09-18 05:11:33.031 8475 TRACE keystone change.run(self.engine, step) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 145, in run > 2015-09-18 05:11:33.031 8475 TRACE keystone script_func(engine) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/keystone/common/sql/migrate_repo/versions/011_endpoints_v3.py", line 33, in upgrade > 2015-09-18 05:11:33.031 8475 TRACE keystone migrate_engine) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 99, in rename_tables_with_constraints > 2015-09-18 05:11:33.031 8475 TRACE keystone remove_constraints(constraints) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 61, in remove_constraints > 2015-09-18 05:11:33.031 8475 TRACE keystone name=constraint_name).drop() > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/changeset/constraint.py", line 59, in drop > 2015-09-18 05:11:33.031 8475 TRACE keystone self.__do_imports('constraintdropper', *a, **kw) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/changeset/constraint.py", line 32, in __do_imports > 2015-09-18 05:11:33.031 8475 TRACE keystone run_single_visitor(engine, visitorcallable, self, *a, **kw) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/changeset/databases/visitor.py", line 75, in run_single_visitor > 2015-09-18 05:11:33.031 8475 TRACE keystone fn(element, **kwargs) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 284, in visit_migrate_foreign_key_constraint > 2015-09-18 05:11:33.031 8475 TRACE keystone self._visit_constraint(*p, **k) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 300, in _visit_constraint > 2015-09-18 05:11:33.031 8475 TRACE keystone constraint.name = self.get_constraint_name(constraint) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 278, in get_constraint_name > 2015-09-18 05:11:33.031 8475 TRACE keystone return self.preparer.quote(ret, cons.quote) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "<string>", line 2, in quote > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/deprecations.py", line 106, in warned > 2015-09-18 05:11:33.031 8475 TRACE keystone return fn(*args, **kwargs) > 2015-09-18 05:11:33.031 8475 TRACE keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/schema.py", line 90, in quote > 2015-09-18 05:11:33.031 8475 TRACE keystone return self.name.quote > 2015-09-18 05:11:33.031 8475 TRACE keystone AttributeError: 'unicode' object has no attribute 'quote' yum info python-{migrate,sqlalchemy} > Loaded plugins: search-disabled-repos > Installed Packages > Name : python-migrate > Arch : noarch > Version : 0.8.2 > Release : 2.el7ost > Size : 770 k > Repo : installed > From repo : rhelosp-5.0-el7-puddle > Summary : Schema migration tools for SQLAlchemy > URL : https://github.com/stackforge/sqlalchemy-migrate > License : MIT > Description : Schema migration tools for SQLAlchemy designed to support an agile approach > : to database design and make it easier to keep development and production > : databases in sync as schema changes are required. It allows you to manage > : database change sets and database repository versioning. > > Name : python-sqlalchemy > Arch : x86_64 > Version : 0.9.8 > Release : 1.el7 > Size : 20 M > Repo : installed > From repo : rhelosp-rhel-7-server > Summary : Modular and flexible ORM library for python > URL : http://www.sqlalchemy.org/ > License : MIT > Description : SQLAlchemy is an Object Relational Mappper (ORM) that provides a flexible, > : high-level interface to SQL databases. Database and domain concepts are > : decoupled, allowing both sides maximum flexibility and power. SQLAlchemy > : provides a powerful mapping layer that can work as automatically or as manually > : as you choose, determining relationships based on foreign keys or letting you > : define the join conditions explicitly, to bridge the gap between database and > : domain. > : > : This package includes the python 2 version of the module. yum list --showduplicates python-sqlalchemy > Loaded plugins: search-disabled-repos > Installed Packages > python-sqlalchemy.x86_64 0.9.8-1.el7 @rhelosp-rhel-7-server > Available Packages > python-sqlalchemy.x86_64 0.8.4-2.el7ost rhelosp-5.0-el7-puddle > python-sqlalchemy.x86_64 0.9.8-1.el7 rhelosp-rhel-7-server yum repolist -v # cutted > Repo-id : rhelosp-rhel-7-server/x86_64 > Repo-name : Red Hat Enterprise Linux 7Server - x86_64 - Server > Repo-revision: 1440085913 > Repo-updated : Thu Aug 20 11:51:54 2015 > Repo-pkgs : 4,610 > Repo-size : 3.4 G > Repo-baseurl : http://..snip../released/RHEL-7/7.2-Beta/Server/x86_64/os/ > Repo-expire : 21,600 second(s) (last: Fri Sep 18 07:19:12 2015) > Repo-filename: /etc/yum.repos.d/rhos-release-5-rhel-7.2.repo > ... > Repo-id : rhelosp-5.0-el7-puddle/x86_64 > Repo-name : RHOS-5.0 el7 > Repo-revision: 1441975196 > Repo-updated : Fri Sep 11 08:40:15 2015 > Repo-pkgs : 302 > Repo-size : 164 M > Repo-baseurl : http://..snip../rel-eng/OpenStack/5.0-RHEL-7/latest/RH7-RHOS-5.0/x86_64/os > Repo-expire : 21,600 second(s) (last: Fri Sep 18 07:19:12 2015) > Repo-filename: /etc/yum.repos.d/rhos-release-5-rhel-7.2.repo Simplified reproducer calling keystone db sync directly (without packstack/puppet): > # > /var/log/keystone/keystone.log > # keystone-manage db_version > 10 > # keystone-manage db_sync > # tail /var/log/keystone/keystone.log > 2015-09-22 11:26:40.258 20327 TRACE keystone constraint.name = self.get_constraint_name(constraint) > 2015-09-22 11:26:40.258 20327 TRACE keystone File "/usr/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 278, in get_constraint_name > 2015-09-22 11:26:40.258 20327 TRACE keystone return self.preparer.quote(ret, cons.quote) > 2015-09-22 11:26:40.258 20327 TRACE keystone File "<string>", line 2, in quote > 2015-09-22 11:26:40.258 20327 TRACE keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/deprecations.py", line 106, in warned > 2015-09-22 11:26:40.258 20327 TRACE keystone return fn(*args, **kwargs) > 2015-09-22 11:26:40.258 20327 TRACE keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/schema.py", line 90, in quote > 2015-09-22 11:26:40.258 20327 TRACE keystone return self.name.quote > 2015-09-22 11:26:40.258 20327 TRACE keystone AttributeError: 'unicode' object has no attribute 'quote' > 2015-09-22 11:26:40.258 20327 TRACE keystone > # keystone-manage db_version > 10 then downgrade to 'rhos-5' version of sqlalchemy > # yum downgrade python-sqlalchemy-0.8.4-2.el7ost > .. > Installing : python-sqlalchemy-0.8.4-2.el7ost.x86_64 1/2 > Cleanup : python-sqlalchemy-0.9.8-1.el7.x86_64 2/2 > .. drop all keystone tables (/etc/keystone/keystone.conf has connection info) > # mysql -ukeystone_admin -p keystone > DROP TABLE `credential` CASCADE; > DROP TABLE `domain` CASCADE; > DROP TABLE `ec2_credential` CASCADE; > DROP TABLE `endpoint` CASCADE; > DROP TABLE `group` CASCADE; > DROP TABLE `group_domain_metadata` CASCADE; > DROP TABLE `group_project_metadata` CASCADE; > DROP TABLE `metadata` CASCADE; > DROP TABLE `migrate_version` CASCADE; > DROP TABLE `policy` CASCADE; > DROP TABLE `project` CASCADE; > DROP TABLE `role` CASCADE; > DROP TABLE `service` CASCADE; > DROP TABLE `token` CASCADE; > DROP TABLE `user` CASCADE; > DROP TABLE `user_domain_metadata` CASCADE; > DROP TABLE `user_group_membership` CASCADE; > DROP TABLE `user_project_membership` CASCADE; > DROP TABLE `user_project_metadata` CASCADE; and retrying again, it works: > # > /var/log/keystone.log > # keystone-manage db_sync > # keystone-manage db_version > 44 > # tail /var/log/keystone.log > 2015-09-22 11:41:11.377 3601 INFO migrate.versioning.api [-] 39 -> 40... > 2015-09-22 11:41:11.397 3601 INFO migrate.versioning.api [-] done > 2015-09-22 11:41:11.398 3601 INFO migrate.versioning.api [-] 40 -> 41... > 2015-09-22 11:41:11.405 3601 INFO migrate.versioning.api [-] done > 2015-09-22 11:41:11.406 3601 INFO migrate.versioning.api [-] 41 -> 42... > 2015-09-22 11:41:11.418 3601 INFO migrate.versioning.api [-] done > 2015-09-22 11:41:11.418 3601 INFO migrate.versioning.api [-] 42 -> 43... > 2015-09-22 11:41:11.423 3601 INFO migrate.versioning.api [-] done > 2015-09-22 11:41:11.423 3601 INFO migrate.versioning.api [-] 43 -> 44... > 2015-09-22 11:41:11.433 3601 INFO migrate.versioning.api [-] done
> It seems that python-sqlalchemy got updated to version 0.9.8 in RHEL-7.2/Server, and is used instead of the one shipped with RHOS-5.0 which is 0.8.4. for this part of it, I'm not sure what should be done; it should not be the case that the package is installed from both places, so I don't know the rationale for this. As far as fixes, the fixes can include: 1. Rely on rhel-7.2/server python-sqlalchemy, rebase python-migrate to 0.9.1 2. Rely on rhos-5.0 python-sqlalchemy, make sure rhel-7.2/server is not installed 3. rebase rhos-5.0-rhel-7 python-sqlalchemy and python-migrate to 0.9.8 and 0.9.1, respectively, matching what we'd see from rhel-7.2/server if it happens to be there
have a Q out to lon on IRC for what we should do here.
issue in question was in https://review.openstack.org/#/c/77387/
I rebased the package in git but there's already a python-migrate-0.9.1-1 package on brew, so ultimately I just tagged that.
*** Bug 1275777 has been marked as a duplicate of this bug. ***
https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RHOS/view/RHOS5/job/rhos-jenkins-rhos-5.0-puddle-rhel-7.1-multi-node-packstack-neutron-ml2-vxlan-rabbitmq-tempest-git-all/ automation passed
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-2494.html