Bug 879387 - python-migrate-0.6 causes issues with openstack-glance db sync with postgresql
Summary: python-migrate-0.6 causes issues with openstack-glance db sync with postgresql
Alias: None
Product: RDO
Classification: Community
Component: distribution
Version: unspecified
Hardware: Unspecified
OS: Unspecified
Target Milestone: Milestone3
: Havana
Assignee: Pádraig Brady
QA Contact: Jay Turner
Depends On:
TreeView+ depends on / blocked
Reported: 2012-11-22 19:22 UTC by Attila Fazekas
Modified: 2016-01-04 14:46 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 905082 (view as bug list)
Last Closed: 2013-09-12 17:02:11 UTC

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Launchpad 1081172 0 None None None Never

Description Attila Fazekas 2012-11-22 19:22:21 UTC
See the external bug report.

It is working with 0.6.1

As a workaround I could use:
easy-install pip
pip install sqlalchemy-migrate==0.6.1

Comment 1 Pádraig Brady 2012-11-22 20:13:51 UTC
Thanks Attila for noticing the upstream bug,
and trying out the new version, with PostgreSQL.

While there is no mention of PostgreSQL in the 0.6.1 release notes,
I'll take your word that it fixes the bug :)

I'll prepare an update ASAP unless I hear otherwise.

BTW I'm a bit worried by the release notes mention of
"migrate.versioning.exceptions to migrate.exceptions"
as I needed to patch nova to handle that explicitly like:

    # Try the more specific path first (migrate <= 0.6)
    from migrate.versioning import exceptions as versioning_exceptions
  except ImportError:
    # Use the newer path (migrate >= 0.7)
    from migrate import exceptions as versioning_exceptions

So nova will be fine with either, but existing packages
expecting migrate.versioning.exceptions may fail?

Attila could you try this on your pip installed version:

python -c "from migrate.versioning import exceptions; print exceptions.__file__"

I expect it might error.

Comment 2 Attila Fazekas 2012-11-22 21:06:19 UTC
Looks like it just jumps that particular issue when I tried with 0.6.1 :(. It is just working (return 0) after you tried with the older version first.
I was also able to use it.

Just 0.6.1 wont be enough.

With 0.6.1 I got this:
python -c "from migrate.versioning import exceptions; print exceptions.__file__"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name exceptions

After drop and create database again round I got a new error message:
Traceback (most recent call last):
  File "/usr/bin/glance-manage", line 126, in <module>
  File "/usr/bin/glance-manage", line 122, in main
  File "/usr/bin/glance-manage", line 98, in dispatch_cmd
  File "/usr/bin/glance-manage", line 86, in do_db_sync
    glance.db.sqlalchemy.migration.db_sync(version, current_version)
  File "/usr/lib/python2.6/site-packages/glance/db/sqlalchemy/migration.py", line 127, in db_sync
  File "/usr/lib/python2.6/site-packages/glance/db/sqlalchemy/migration.py", line 67, in upgrade
    return versioning_api.upgrade(sql_connection, repo_path, version)
  File "/usr/lib/python2.6/site-packages/migrate/versioning/api.py", line 186, in upgrade
    return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  File "<string>", line 2, in _migrate
  File "/usr/lib/python2.6/site-packages/migrate/versioning/util/__init__.py", line 159, in with_engine
    return f(*a, **kw)
  File "/usr/lib/python2.6/site-packages/migrate/versioning/api.py", line 365, in _migrate
    schema.runchange(ver, change, changeset.step)
  File "/usr/lib/python2.6/site-packages/migrate/versioning/schema.py", line 84, in runchange
    change.run(self.engine, step)
  File "/usr/lib/python2.6/site-packages/migrate/versioning/script/py.py", line 145, in run
  File "/usr/lib/python2.6/site-packages/glance/db/sqlalchemy/migrate_repo/versions/003_add_disk_format.py", line 114, in upgrade
  File "/usr/lib/python2.6/site-packages/migrate/changeset/schema.py", line 552, in drop
    self.remove_from_table(self.table, unset_table=False)
  File "/usr/lib/python2.6/site-packages/migrate/changeset/schema.py", line 593, in remove_from_table
  File "/usr/lib/python2.6/site-packages/sqlalchemy/util/_collections.py", line 38, in _immutable
    raise TypeError("%s object is immutable" % self.__class__.__name__)
TypeError: ImmutableColumnCollection object is immutable

I had to upgrade it a much newer version (tried 0.7.2) in order to work from the beginning.

Comment 3 Attila Fazekas 2012-11-23 01:53:46 UTC
The http://code.google.com/p/sqlalchemy-migrate/ says the "SQLAlchemy-migrate 0.7.1 provides support for SQLAlchemy versions from 0.5.x to 0.7.x."

With 0.7.1 I was able to sync keystone, nova, glance and cinder.

I got the usual warning message too.
/usr/lib/python2.6/site-packages/glance/db/sqlalchemy/migrate_repo/versions/003_add_disk_format.py:46: SADeprecationWarning: useexisting is deprecated.  Use extend_existing.

Comment 4 Alan Pevec 2012-11-23 10:44:53 UTC
Pádraig, is insta||able python-migrate0.7-0.7.2 doable?
PostgreSQL support seems important enough to justify it and it would avoid sqlalchemy/migrate combination nobody else uses.

Comment 6 Pádraig Brady 2012-11-23 11:10:05 UTC
Parallel installable is doable but parallel installable packages are best avoided if possible. I'd be inclined to update to 0.7.1 in this case.

Comment 8 Pádraig Brady 2013-01-22 17:42:58 UTC
So python-migrate=0.7.2 is what's generally in use,
and required by nova, etc. That dependency was done upstream
as there were issues seen with v 0.7.1 in

Though Attila in comment #3 said all projects sync fine with 0.7.1
so maybe the code at issue with 0.7.1 was changed upstream in the meantime?

Given 0.7.2 is documented to not be compatible with python-sqlalchemy-0.5
I'll avoid that for now.

Though Attila why is this the only report of this?
Is there something specific about your environment?

Anyway I'll prepare an update for testing, which I won't
push to stable until it gets appropriate karma.

Comment 9 Pádraig Brady 2013-01-22 20:15:59 UTC
So that was not successful. Seems like contrary to the docs,
0.7.1 is not in fact compat with sqlalchemy 0.5 as the build
% check fails with lots of instances of:

File "/builddir/build/BUILD/sqlalchemy-migrate-0.7.1/migrate/versioning/schemadiff.py", line 72, in __init__
    self.affinity_A = self.type_A._type_affinity
AttributeError: 'Text' object has no attribute '_type_affinity'


I presume _type_affinity was introduced to sqlalchemy sometime
after the version currently in RHEL.

Comment 10 Pádraig Brady 2013-02-26 09:59:16 UTC
So the _type_affinity code was added since sqlalchemy 0.5.7.
So the version 0.5.5 in RHEL 6.4 (released Jul 2009)
is not supported by migrate 0.7.1.

To have a non parallel installable python-migrate-0.7.1 would therefore
require rebasing python-sqlalchemy in RHEL 6 to 0.5.8 (released Jul 2010)

Comment 11 Pádraig Brady 2013-09-11 14:17:48 UTC
Another report from Havana 3 test that with Postgres and migrate 0.6 you get:

glance AttributeError: 'PGSchemaChanger' object has no attribute '_validate_identifier'

The other option now available is to make a non parallel installable python-migrate-0.7.2 available to RDO and if postgres support required RHOS (LPC).
So given the risk/delay of updating the RHEL/EPEL components as described in comment 10, I'll take this non parallel approach.

Comment 12 Pádraig Brady 2013-09-12 00:07:14 UTC
I've built an updated python-migrate-0.7.2 that depends on sqlalchemy0.7 at:

It was tricky to avoid an (already) parallel installed sqlalchemy-0.5 in all cases, but I think it should work.
Details of the changes specific to the parallel sqlalchemy0.7 are:

Comment 14 Pádraig Brady 2013-09-12 17:02:11 UTC
Pushed to RDO Havana EL6 repo

Note You need to log in before you can comment on or make changes to this bug.