Bug 1265306 - update to python-sqlalchemy-0.9.8-1 breaks OpenStack db migrations - AttributeError 'unicode' object has no attribute 'quota'
Summary: update to python-sqlalchemy-0.9.8-1 breaks OpenStack db migrations - Attribut...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-migrate
Version: 5.0 (RHEL 7)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: async
: 5.0 (RHEL 7)
Assignee: Michael Bayer
QA Contact: nlevinki
URL:
Whiteboard:
: 1275777 (view as bug list)
Depends On:
Blocks: 1266192 1266193 1266195
TreeView+ depends on / blocked
 
Reported: 2015-09-22 15:47 UTC by Pavel Sedlák
Modified: 2016-04-27 04:33 UTC (History)
6 users (show)

Fixed In Version: python-migrate-0.9.1-1.el7ost
Doc Type: Rebase: Bug Fixes and Enhancements
Doc Text:
Clone Of:
: 1266192 1266193 (view as bug list)
Environment:
Last Closed: 2015-11-19 23:31:12 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2494 0 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform 5 Bug Fix and Enhancement Advisory 2015-11-20 03:34:44 UTC

Description Pavel Sedlák 2015-09-22 15:47:29 UTC
When trying to install OpenStack Icehouse (rhos-5.0),
using packstack, it fails on keystone puppet,
based on the keystone.log it failed when applying DB MIGRATIONS.

It seems that python-sqlalchemy got updated to version 0.9.8 in RHEL-7.2/Server,
 and is used instead of the one shipped with RHOS-5.0 which is 0.8.4.

Based on old Debian OpenStack packaging mail archives it could
be the reason why migration is failing (and so keystone.pp and whole installation),
as they mentioned that at that time used python-migrate (among others) is not supporting
newer sqlalchemy version.

https://lists.debian.org/debian-release/2014/02/msg00477.html

Snippet from keystone.log:
> 2015-09-18 05:11:33.025 8475 INFO migrate.versioning.api [-] done
> 2015-09-18 05:11:33.025 8475 INFO migrate.versioning.api [-] 10 -> 11... 
> 2015-09-18 05:11:33.031 8475 CRITICAL keystone [-] AttributeError: 'unicode' object has no attribute 'quote'
> 2015-09-18 05:11:33.031 8475 TRACE keystone Traceback (most recent call last):
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/bin/keystone-manage", line 51, in <module>
> 2015-09-18 05:11:33.031 8475 TRACE keystone     cli.main(argv=sys.argv, config_files=config_files)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/cli.py", line 190, in main
> 2015-09-18 05:11:33.031 8475 TRACE keystone     CONF.command.cmd_class.main()
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/cli.py", line 66, in main
> 2015-09-18 05:11:33.031 8475 TRACE keystone     migration_helpers.sync_database_to_version(extension, version)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 160, in sync_database_to_version
> 2015-09-18 05:11:33.031 8475 TRACE keystone     migration.db_sync(engine, abs_path, version=version)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/openstack/common/db/sqlalchemy/migration.py", line 197, in db_sync
> 2015-09-18 05:11:33.031 8475 TRACE keystone     return versioning_api.upgrade(engine, repository, version)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
> 2015-09-18 05:11:33.031 8475 TRACE keystone     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "<string>", line 2, in _migrate
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 159, in with_engine
> 2015-09-18 05:11:33.031 8475 TRACE keystone     return f(*a, **kw)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
> 2015-09-18 05:11:33.031 8475 TRACE keystone     schema.runchange(ver, change, changeset.step)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 91, in runchange
> 2015-09-18 05:11:33.031 8475 TRACE keystone     change.run(self.engine, step)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 145, in run
> 2015-09-18 05:11:33.031 8475 TRACE keystone     script_func(engine)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/common/sql/migrate_repo/versions/011_endpoints_v3.py", line 33, in upgrade
> 2015-09-18 05:11:33.031 8475 TRACE keystone     migrate_engine)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 99, in rename_tables_with_constraints
> 2015-09-18 05:11:33.031 8475 TRACE keystone     remove_constraints(constraints)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 61, in remove_constraints
> 2015-09-18 05:11:33.031 8475 TRACE keystone     name=constraint_name).drop()
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/changeset/constraint.py", line 59, in drop
> 2015-09-18 05:11:33.031 8475 TRACE keystone     self.__do_imports('constraintdropper', *a, **kw)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/changeset/constraint.py", line 32, in __do_imports
> 2015-09-18 05:11:33.031 8475 TRACE keystone     run_single_visitor(engine, visitorcallable, self, *a, **kw)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/changeset/databases/visitor.py", line 75, in run_single_visitor
> 2015-09-18 05:11:33.031 8475 TRACE keystone     fn(element, **kwargs)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 284, in visit_migrate_foreign_key_constraint
> 2015-09-18 05:11:33.031 8475 TRACE keystone     self._visit_constraint(*p, **k)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 300, in _visit_constraint
> 2015-09-18 05:11:33.031 8475 TRACE keystone     constraint.name = self.get_constraint_name(constraint)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 278, in get_constraint_name
> 2015-09-18 05:11:33.031 8475 TRACE keystone     return self.preparer.quote(ret, cons.quote)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "<string>", line 2, in quote
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/deprecations.py", line 106, in warned
> 2015-09-18 05:11:33.031 8475 TRACE keystone     return fn(*args, **kwargs)
> 2015-09-18 05:11:33.031 8475 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/schema.py", line 90, in quote
> 2015-09-18 05:11:33.031 8475 TRACE keystone     return self.name.quote
> 2015-09-18 05:11:33.031 8475 TRACE keystone AttributeError: 'unicode' object has no attribute 'quote'


yum info python-{migrate,sqlalchemy}
> Loaded plugins: search-disabled-repos
> Installed Packages
> Name        : python-migrate
> Arch        : noarch
> Version     : 0.8.2
> Release     : 2.el7ost
> Size        : 770 k
> Repo        : installed
> From repo   : rhelosp-5.0-el7-puddle
> Summary     : Schema migration tools for SQLAlchemy
> URL         : https://github.com/stackforge/sqlalchemy-migrate
> License     : MIT
> Description : Schema migration tools for SQLAlchemy designed to support an agile approach
>             : to database design and make it easier to keep development and production
>             : databases in sync as schema changes are required.  It allows you to manage
>             : database change sets and database repository versioning.
> 
> Name        : python-sqlalchemy
> Arch        : x86_64
> Version     : 0.9.8
> Release     : 1.el7
> Size        : 20 M
> Repo        : installed
> From repo   : rhelosp-rhel-7-server
> Summary     : Modular and flexible ORM library for python
> URL         : http://www.sqlalchemy.org/
> License     : MIT
> Description : SQLAlchemy is an Object Relational Mappper (ORM) that provides a flexible,
>             : high-level interface to SQL databases.  Database and domain concepts are
>             : decoupled, allowing both sides maximum flexibility and power. SQLAlchemy
>             : provides a powerful mapping layer that can work as automatically or as manually
>             : as you choose, determining relationships based on foreign keys or letting you
>             : define the join conditions explicitly, to bridge the gap between database and
>             : domain.
>             : 
>             : This package includes the python 2 version of the module.

yum list --showduplicates python-sqlalchemy
> Loaded plugins: search-disabled-repos
> Installed Packages
> python-sqlalchemy.x86_64       0.9.8-1.el7        @rhelosp-rhel-7-server
> Available Packages
> python-sqlalchemy.x86_64       0.8.4-2.el7ost     rhelosp-5.0-el7-puddle
> python-sqlalchemy.x86_64       0.9.8-1.el7        rhelosp-rhel-7-server 

yum repolist -v  # cutted
> Repo-id      : rhelosp-rhel-7-server/x86_64
> Repo-name    : Red Hat Enterprise Linux 7Server - x86_64 - Server
> Repo-revision: 1440085913
> Repo-updated : Thu Aug 20 11:51:54 2015
> Repo-pkgs    : 4,610
> Repo-size    : 3.4 G
> Repo-baseurl : http://..snip../released/RHEL-7/7.2-Beta/Server/x86_64/os/
> Repo-expire  : 21,600 second(s) (last: Fri Sep 18 07:19:12 2015)
> Repo-filename: /etc/yum.repos.d/rhos-release-5-rhel-7.2.repo
> ...
> Repo-id      : rhelosp-5.0-el7-puddle/x86_64
> Repo-name    : RHOS-5.0 el7
> Repo-revision: 1441975196
> Repo-updated : Fri Sep 11 08:40:15 2015
> Repo-pkgs    : 302
> Repo-size    : 164 M
> Repo-baseurl : http://..snip../rel-eng/OpenStack/5.0-RHEL-7/latest/RH7-RHOS-5.0/x86_64/os
> Repo-expire  : 21,600 second(s) (last: Fri Sep 18 07:19:12 2015)
> Repo-filename: /etc/yum.repos.d/rhos-release-5-rhel-7.2.repo


Simplified reproducer calling keystone db sync directly (without packstack/puppet):
> # > /var/log/keystone/keystone.log
> # keystone-manage db_version
> 10
> # keystone-manage db_sync
> # tail /var/log/keystone/keystone.log
> 2015-09-22 11:26:40.258 20327 TRACE keystone     constraint.name = self.get_constraint_name(constraint)
> 2015-09-22 11:26:40.258 20327 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 278, in get_constraint_name
> 2015-09-22 11:26:40.258 20327 TRACE keystone     return self.preparer.quote(ret, cons.quote)
> 2015-09-22 11:26:40.258 20327 TRACE keystone   File "<string>", line 2, in quote
> 2015-09-22 11:26:40.258 20327 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/deprecations.py", line 106, in warned
> 2015-09-22 11:26:40.258 20327 TRACE keystone     return fn(*args, **kwargs)
> 2015-09-22 11:26:40.258 20327 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/schema.py", line 90, in quote
> 2015-09-22 11:26:40.258 20327 TRACE keystone     return self.name.quote
> 2015-09-22 11:26:40.258 20327 TRACE keystone AttributeError: 'unicode' object has no attribute 'quote'
> 2015-09-22 11:26:40.258 20327 TRACE keystone 
> # keystone-manage db_version
> 10

then downgrade to 'rhos-5' version of sqlalchemy
> # yum downgrade python-sqlalchemy-0.8.4-2.el7ost
> ..
>  Installing : python-sqlalchemy-0.8.4-2.el7ost.x86_64                                                                                                                                            1/2 
>  Cleanup    : python-sqlalchemy-0.9.8-1.el7.x86_64                                                                                                                                               2/2 
> ..

drop all keystone tables (/etc/keystone/keystone.conf has connection info)
> # mysql -ukeystone_admin -p keystone
> DROP TABLE `credential` CASCADE;
> DROP TABLE `domain` CASCADE;
> DROP TABLE `ec2_credential` CASCADE;
> DROP TABLE `endpoint` CASCADE;
> DROP TABLE `group` CASCADE;
> DROP TABLE `group_domain_metadata` CASCADE;
> DROP TABLE `group_project_metadata` CASCADE;
> DROP TABLE `metadata` CASCADE;
> DROP TABLE `migrate_version` CASCADE;
> DROP TABLE `policy` CASCADE;
> DROP TABLE `project` CASCADE;
> DROP TABLE `role` CASCADE;
> DROP TABLE `service` CASCADE;
> DROP TABLE `token` CASCADE;
> DROP TABLE `user` CASCADE;
> DROP TABLE `user_domain_metadata` CASCADE;
> DROP TABLE `user_group_membership` CASCADE;
> DROP TABLE `user_project_membership` CASCADE;
> DROP TABLE `user_project_metadata` CASCADE;

and retrying again, it works:
> # > /var/log/keystone.log
> # keystone-manage db_sync
> # keystone-manage db_version
> 44
> # tail /var/log/keystone.log
> 2015-09-22 11:41:11.377 3601 INFO migrate.versioning.api [-] 39 -> 40... 
> 2015-09-22 11:41:11.397 3601 INFO migrate.versioning.api [-] done
> 2015-09-22 11:41:11.398 3601 INFO migrate.versioning.api [-] 40 -> 41... 
> 2015-09-22 11:41:11.405 3601 INFO migrate.versioning.api [-] done
> 2015-09-22 11:41:11.406 3601 INFO migrate.versioning.api [-] 41 -> 42... 
> 2015-09-22 11:41:11.418 3601 INFO migrate.versioning.api [-] done
> 2015-09-22 11:41:11.418 3601 INFO migrate.versioning.api [-] 42 -> 43... 
> 2015-09-22 11:41:11.423 3601 INFO migrate.versioning.api [-] done
> 2015-09-22 11:41:11.423 3601 INFO migrate.versioning.api [-] 43 -> 44... 
> 2015-09-22 11:41:11.433 3601 INFO migrate.versioning.api [-] done

Comment 2 Michael Bayer 2015-09-22 21:48:20 UTC
> It seems that python-sqlalchemy got updated to version 0.9.8 in RHEL-7.2/Server,
 and is used instead of the one shipped with RHOS-5.0 which is 0.8.4.


for this part of it, I'm not sure what should be done; it should not be the case that the package is installed from both places, so I don't know the rationale for this.

As far as fixes, the fixes can include:

1. Rely on rhel-7.2/server python-sqlalchemy, rebase python-migrate to 0.9.1

2. Rely on rhos-5.0 python-sqlalchemy, make sure rhel-7.2/server is not installed

3. rebase rhos-5.0-rhel-7 python-sqlalchemy and python-migrate to 0.9.8 and 0.9.1, respectively, matching what we'd see from rhel-7.2/server if it happens to be there

Comment 3 Michael Bayer 2015-09-22 22:08:49 UTC
have a Q out to lon on IRC for what we should do here.

Comment 5 Michael Bayer 2015-09-24 18:01:33 UTC
issue in question was in https://review.openstack.org/#/c/77387/

Comment 6 Michael Bayer 2015-09-25 23:31:01 UTC
I rebased the package in git but there's already a python-migrate-0.9.1-1 package on brew, so ultimately I just tagged that.

Comment 7 Michael Bayer 2015-10-28 13:36:28 UTC
*** Bug 1275777 has been marked as a duplicate of this bug. ***

Comment 11 errata-xmlrpc 2015-11-19 23:31:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2494.html


Note You need to log in before you can comment on or make changes to this bug.