Bug 960990 - db migrate fails to create new table
db migrate fails to create new table
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
3.0
Unspecified Unspecified
unspecified Severity unspecified
: beta
: 4.0
Assigned To: Eric Harney
Dafna Ron
storage
: TestOnly
Depends On: 1010039
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-08 09:18 EDT by Dan Radez
Modified: 2016-04-26 09:26 EDT (History)
10 users (show)

See Also:
Fixed In Version: openstack-cinder-2013.2-1.el6ost
Doc Type: Bug Fix
Doc Text:
Previously, upgrades between major database versions failed with an SQLAlchemy error related to foreign keys. This update patches Block Storage SQLAlchemy migration scripts to specify the 'utf8' charset. As a result, database upgrades now function as expected when running 'cinder-manage db sync'.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-19 19:02:28 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1233861 None None None Never

  None (edit)
Description Dan Radez 2013-05-08 09:18:13 EDT
Description of problem:
When upgrading from RHOS 2.1 to 3.0 (I'm using RDO) the cinder db migrate fails to add a new table.

Version-Release number of selected component (if applicable):
upgrade from 2.1 to 3.0

How reproducible:
evertime

Steps to Reproduce:
1. start with a working copy of 2.1
2. stop cinder services
3. yum update
4. update config files so you're using the new config files with the existing ips and users and passwords etc.
5. start services and do an cinder-manage db migrate.
  
Actual results:
OperationalError: (OperationalError) (1005, "Can't create table 'cinder.volume_glance_metadata' (errno: 150)") '\nCREATE TABLE volume_glance_metadata (\n\tcreated_at DATETIME, \n\tupdated_at DATETIME, \n\tdeleted_at DATETIME, \n\tdeleted BOOL, \n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tvolume_id VARCHAR(36), \n\tsnapshot_id VARCHAR(36), \n\t`key` VARCHAR(255), \n\tvalue TEXT, \n\tPRIMARY KEY (id), \n\tCHECK (deleted IN (0, 1)), \n\tFOREIGN KEY(volume_id) REFERENCES volumes (id), \n\tFOREIGN KEY(snapshot_id) REFERENCES snapshots (id)\n)ENGINE=InnoDB\n\n' ()
notice: /Stage[main]/Cinder::Api/Exec[cinder-manage db_sync]/returns: 2013-05-08 06:09:54 CRITICAL [cinder] (OperationalError) (1005, "Can't create table 'cinder.volume_glance_metadata' (errno: 150)") '\nCREATE TABLE volume_glance_metadata (\n\tcreated_at DATETIME, \n\tupdated_at DATETIME, \n\tdeleted_at DATETIME, \n\tdeleted BOOL, \n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tvolume_id VARCHAR(36), \n\tsnapshot_id VARCHAR(36), \n\t`key` VARCHAR(255), \n\tvalue TEXT, \n\tPRIMARY KEY (id), \n\tCHECK (deleted IN (0, 1)), \n\tFOREIGN KEY(volume_id) REFERENCES volumes (id), \n\tFOREIGN KEY(snapshot_id) REFERENCES snapshots (id)\n)ENGINE=InnoDB\n\n' ()


Expected results:
new table created

Additional info:
http://stackoverflow.com/questions/1457305/mysql-creating-tables-with-foreign-keys-giving-errno-150
suggests the relationships are a good starting point for a solution
Comment 2 Alan Pevec 2013-05-08 10:33:28 EDT
F->G upstream upgrade testing (Grenade) seems to be working fine:
e.g. http://logs.openstack.org/28440/1/check/gate-grenade-devstack-vm/2643/console.html.gz

2013-05-07 17:25:34.174 | + cinder-manage db sync
2013-05-07 17:25:34.687 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] 2 -> 3... 
2013-05-07 17:25:34.723 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] done
2013-05-07 17:25:34.724 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] 3 -> 4... 
2013-05-07 17:25:34.741 | 2013-05-07 17:25:34     INFO [004_volume_type_to_uuid] Created foreign key volume_type_extra_specs_ibfk_1
2013-05-07 17:25:34.744 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] done
2013-05-07 17:25:34.745 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] 4 -> 5... 
2013-05-07 17:25:34.760 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] done
2013-05-07 17:25:34.760 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] 5 -> 6... 
2013-05-07 17:25:34.780 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] done
2013-05-07 17:25:34.780 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] 6 -> 7... 
2013-05-07 17:25:34.795 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] done
2013-05-07 17:25:34.795 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] 7 -> 8... 
2013-05-07 17:25:34.805 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] done
2013-05-07 17:25:34.817 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] 8 -> 9... 
2013-05-07 17:25:34.817 | 2013-05-07 17:25:34     INFO [migrate.versioning.api] done

Looks like you might have inconsistent (missing) data in volumes or snapshots tables?
Comment 3 Alan Pevec 2013-05-08 10:38:08 EDT
> 5. start services and do an cinder-manage db migrate.

Which services you mean -db should be migrated before starting services.
Comment 4 Dan Radez 2013-05-09 15:53:31 EDT
so I've been using packstack to test my upgrade. I used RHOS 2.1 this time and follow the steps above.

For #5 I merged a generated packstack answer file with the original answer file from the install. Then I ran packstack --answer-file with the merged file. Results are the same. volumes and snapshots tables are empty as shown. cinder-manage db sync throws the same error I get when running packstack.

[root@radez ~(keystone_admin)]$ packstack --answer-file packstack-answers-20130509-150633.txt 
Welcome to Installer setup utility
Packstack changed given value  to required value /root/.ssh/id_rsa.pub

Installing:
Clean Up...                                              [ DONE ]
Setting up ssh keys...                                   [ DONE ]
Adding pre install manifest entries...                   [ DONE ]
Adding MySQL manifest entries...                         [ DONE ]
Adding QPID manifest entries...                          [ DONE ]
Adding Keystone manifest entries...                      [ DONE ]
Adding Glance Keystone manifest entries...               [ DONE ]
Adding Glance manifest entries...                        [ DONE ]
Adding Cinder Keystone manifest entries...               [ DONE ]
Installing dependencies for Cinder...                    [ DONE ]
Checking if the Cinder server has a cinder-volumes vg... [ DONE ]
Adding Cinder manifest entries...                        [ DONE ]
Adding Nova API manifest entries...                      [ DONE ]
Adding Nova Keystone manifest entries...                 [ DONE ]
Adding Nova Cert manifest entries...                     [ DONE ]
Adding Nova Conductor manifest entries...                [ DONE ]
Adding Nova Compute manifest entries...                  [ DONE ]
Adding Nova Network manifest entries...                  [ DONE ]
Adding Nova Scheduler manifest entries...                [ DONE ]
Adding Nova VNC Proxy manifest entries...                [ DONE ]
Adding Nova Common manifest entries...                   [ DONE ]
Adding OpenStack Client manifest entries...              [ DONE ]
Adding Horizon manifest entries...                       [ DONE ]
Preparing servers...                                     [ DONE ]
Adding post install manifest entries...                  [ DONE ]
Installing Dependencies...                               [ DONE ]
Copying Puppet modules and manifests...                  [ DONE ]
Applying Puppet manifests...
Applying 10.13.128.22_prescript.pp
10.13.128.22_prescript.pp :                                          [ DONE ]
Applying 10.13.128.22_mysql.pp
Applying 10.13.128.22_qpid.pp
10.13.128.22_mysql.pp :                                              [ DONE ]
10.13.128.22_qpid.pp :                                               [ DONE ]
Applying 10.13.128.22_keystone.pp
Applying 10.13.128.22_glance.pp
Applying 10.13.128.22_cinder.pp
10.13.128.22_keystone.pp :                                           [ DONE ]
                                                                    
ERROR : Error during puppet run : notice: /Stage[main]/Cinder::Api/Exec[cinder-manage db_sync]/returns: OperationalError: (OperationalError) (1005, "Can't create table 'cinder.volume_glance_metadata' (errno: 150)") '\nCREATE TABLE volume_glance_metadata (\n\tcreated_at DATETIME, \n\tupdated_at DATETIME, \n\tdeleted_at DATETIME, \n\tdeleted BOOL, \n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tvolume_id VARCHAR(36), \n\tsnapshot_id VARCHAR(36), \n\t`key` VARCHAR(255), \n\tvalue TEXT, \n\tPRIMARY KEY (id), \n\tCHECK (deleted IN (0, 1)), \n\tFOREIGN KEY(volume_id) REFERENCES volumes (id), \n\tFOREIGN KEY(snapshot_id) REFERENCES snapshots (id)\n)ENGINE=InnoDB\n\n' ()
Please check log file /var/tmp/packstack/20130509-154023-yhCCkC/openstack-setup.log for more information

Additional information:
 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.
 * Did not create a cinder volume group, one already existed
 * To use the command line tools you need to source the file /root/keystonerc_admin created on 10.13.128.22
 * To use the console, browse to http://10.13.128.22/dashboard
[root@radez ~(keystone_admin)]$ mysql cinder
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 479
Server version: 5.1.69 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select * from volumes;
Empty set (0.00 sec)

mysql> select * from snapshots;
Empty set (0.00 sec)

mysql> Bye
[root@radez ~(keystone_admin)]$ cinder-manage db migrate
usage: cinder-manage db [-h] {version,sync} ...
cinder-manage db: error: argument action: invalid choice: 'migrate' (choose from 'version', 'sync')
[root@radez ~(keystone_admin)]$ cinder-manage db sync
2013-05-09 15:46:35    ERROR [003_glance_metadata] Exception while creating table 'volume_glance_metedata'
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/migrate_repo/versions/003_glance_metadata.py", line 60, in upgrade
    volume_glance_metadata.create()
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/schema.py", line 594, in create
    checkfirst=checkfirst)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2303, in _run_visitor
    conn._run_visitor(visitorcallable, element, **kwargs)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1973, in _run_visitor
    **kwargs).traverse_single(element)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/sql/visitors.py", line 106, in traverse_single
    return meth(obj, **kw)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/ddl.py", line 86, in visit_table
    self.connection.execute(schema.CreateTable(table))
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute
    params)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1542, in _execute_ddl
    compiled
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (OperationalError) (1005, "Can't create table 'cinder.volume_glance_metadata' (errno: 150)") '\nCREATE TABLE volume_glance_metadata (\n\tcreated_at DATETIME, \n\tupdated_at DATETIME, \n\tdeleted_at DATETIME, \n\tdeleted BOOL, \n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tvolume_id VARCHAR(36), \n\tsnapshot_id VARCHAR(36), \n\t`key` VARCHAR(255), \n\tvalue TEXT, \n\tPRIMARY KEY (id), \n\tCHECK (deleted IN (0, 1)), \n\tFOREIGN KEY(volume_id) REFERENCES volumes (id), \n\tFOREIGN KEY(snapshot_id) REFERENCES snapshots (id)\n)ENGINE=InnoDB\n\n' ()
2013-05-09 15:46:35 CRITICAL [cinder] (OperationalError) (1005, "Can't create table 'cinder.volume_glance_metadata' (errno: 150)") '\nCREATE TABLE volume_glance_metadata (\n\tcreated_at DATETIME, \n\tupdated_at DATETIME, \n\tdeleted_at DATETIME, \n\tdeleted BOOL, \n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tvolume_id VARCHAR(36), \n\tsnapshot_id VARCHAR(36), \n\t`key` VARCHAR(255), \n\tvalue TEXT, \n\tPRIMARY KEY (id), \n\tCHECK (deleted IN (0, 1)), \n\tFOREIGN KEY(volume_id) REFERENCES volumes (id), \n\tFOREIGN KEY(snapshot_id) REFERENCES snapshots (id)\n)ENGINE=InnoDB\n\n' ()
[root@radez ~(keystone_admin)]$ grep cinder /var/log/yum.log 
May 09 15:12:49 Installed: python-cinderclient-1.0.1-4.el6ost.noarch
May 09 15:12:51 Installed: python-cinder-2012.2.4-3.el6ost.noarch
May 09 15:12:54 Installed: openstack-cinder-2012.2.4-3.el6ost.noarch
May 09 15:34:01 Updated: python-cinderclient-1.0.3-1.el6.noarch
May 09 15:34:28 Updated: python-cinder-2013.1-2.el6.noarch
May 09 15:34:42 Updated: openstack-cinder-2013.1-2.el6.noarch
Comment 5 Dan Radez 2013-05-09 21:18:21 EDT
I just reran my upgrade, but I ran cinder-manage db sync before I ran packstack, and the sync worked.


Used these steps:
1. install RHOS 2.1 using packstack
2. installed rdo-release
3. yum update
4. stoped cinder services
5. cinder-manage db sync
6. packstack --answer-file merged-ans-file.txt

after the packstack run RDO came up and seemed mostly happy.

So maybe this ends up being a packstack bug?
Comment 6 Eric Harney 2013-05-20 17:04:35 EDT
I'm not sure exactly what to make of this yet.

There are two cases here:

1) The initial failure.  It looks like this is maybe expected because you have to run cinder-manage db sync before restarting the services?
2) Failure when using packstack to do an upgrade.

I tried to use packstack to upgrade from RHOS 2.1 to RDO as Dan did, basing my answer file on the RHOS 2.1 one and adding the required fields.  Packstack says it completes successfully but the Cinder packages are still 2012.2.4.  I'm not sure what is expected behavior -- can a packstack expert shed some light here?  (Either way I didn't hit the same failure while using packstack.)
Comment 7 Derek Higgins 2013-05-27 18:12:18 EDT
upgrades with packstack were never tested functionality so I'm not surprised your hitting problems. I'm not sure its something we would even want to attempt to support with packstack.

Of course we would want to allow people to upgrade an install that was originally setup with packstack. Do we know if this works with RHOS cinder where packstack was never involved?

Eric, I've just seen another error about packstack missing errors in the puppet logs, https://bugzilla.redhat.com/show_bug.cgi?id=965620  you may not still have them but if you do can you check the packstack logs for puppet errors, it may explain your false positive.
Comment 8 Ayal Baron 2013-07-02 05:20:44 EDT
Eric, why is this still on Cinder?
Comment 9 Eric Harney 2013-07-02 09:15:09 EDT
Changing component to packstack -- the core of this issue is around what happens when you attempt to use packstack to upgrade.  From Cinder's POV things are ok when following supported procedures.
Comment 11 Eric Harney 2013-11-01 14:30:33 EDT
I believe this has been fixed in Cinder 2013.2.
Comment 13 Scott Lewis 2013-11-19 11:54:21 EST
Auto adding >= MODIFIED bugs to beta
Comment 23 Dafna Ron 2013-12-12 11:55:57 EST
Folsom to Grizzly or Folsom to Havana will not be tested Grizzly to Havana was tested by Ami's team using 

http://shell.bos.redhat.com/~lkellogg/rhos-upgrade-docs/database-upgrades.html

I am moving this to verified.
Comment 25 errata-xmlrpc 2013-12-19 19:02:28 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html

Note You need to log in before you can comment on or make changes to this bug.