Description of problem: When upgrading from RHOS 2.1 to 3.0 (I'm using RDO) the cinder db migrate fails to add a new table. Version-Release number of selected component (if applicable): upgrade from 2.1 to 3.0 How reproducible: evertime Steps to Reproduce: 1. start with a working copy of 2.1 2. stop cinder services 3. yum update 4. update config files so you're using the new config files with the existing ips and users and passwords etc. 5. start services and do an cinder-manage db migrate. Actual results: OperationalError: (OperationalError) (1005, "Can't create table 'cinder.volume_glance_metadata' (errno: 150)") '\nCREATE TABLE volume_glance_metadata (\n\tcreated_at DATETIME, \n\tupdated_at DATETIME, \n\tdeleted_at DATETIME, \n\tdeleted BOOL, \n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tvolume_id VARCHAR(36), \n\tsnapshot_id VARCHAR(36), \n\t`key` VARCHAR(255), \n\tvalue TEXT, \n\tPRIMARY KEY (id), \n\tCHECK (deleted IN (0, 1)), \n\tFOREIGN KEY(volume_id) REFERENCES volumes (id), \n\tFOREIGN KEY(snapshot_id) REFERENCES snapshots (id)\n)ENGINE=InnoDB\n\n' () notice: /Stage[main]/Cinder::Api/Exec[cinder-manage db_sync]/returns: 2013-05-08 06:09:54 CRITICAL [cinder] (OperationalError) (1005, "Can't create table 'cinder.volume_glance_metadata' (errno: 150)") '\nCREATE TABLE volume_glance_metadata (\n\tcreated_at DATETIME, \n\tupdated_at DATETIME, \n\tdeleted_at DATETIME, \n\tdeleted BOOL, \n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tvolume_id VARCHAR(36), \n\tsnapshot_id VARCHAR(36), \n\t`key` VARCHAR(255), \n\tvalue TEXT, \n\tPRIMARY KEY (id), \n\tCHECK (deleted IN (0, 1)), \n\tFOREIGN KEY(volume_id) REFERENCES volumes (id), \n\tFOREIGN KEY(snapshot_id) REFERENCES snapshots (id)\n)ENGINE=InnoDB\n\n' () Expected results: new table created Additional info: http://stackoverflow.com/questions/1457305/mysql-creating-tables-with-foreign-keys-giving-errno-150 suggests the relationships are a good starting point for a solution
F->G upstream upgrade testing (Grenade) seems to be working fine: e.g. http://logs.openstack.org/28440/1/check/gate-grenade-devstack-vm/2643/console.html.gz 2013-05-07 17:25:34.174 | + cinder-manage db sync 2013-05-07 17:25:34.687 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] 2 -> 3... 2013-05-07 17:25:34.723 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] done 2013-05-07 17:25:34.724 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] 3 -> 4... 2013-05-07 17:25:34.741 | 2013-05-07 17:25:34 INFO [004_volume_type_to_uuid] Created foreign key volume_type_extra_specs_ibfk_1 2013-05-07 17:25:34.744 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] done 2013-05-07 17:25:34.745 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] 4 -> 5... 2013-05-07 17:25:34.760 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] done 2013-05-07 17:25:34.760 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] 5 -> 6... 2013-05-07 17:25:34.780 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] done 2013-05-07 17:25:34.780 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] 6 -> 7... 2013-05-07 17:25:34.795 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] done 2013-05-07 17:25:34.795 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] 7 -> 8... 2013-05-07 17:25:34.805 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] done 2013-05-07 17:25:34.817 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] 8 -> 9... 2013-05-07 17:25:34.817 | 2013-05-07 17:25:34 INFO [migrate.versioning.api] done Looks like you might have inconsistent (missing) data in volumes or snapshots tables?
> 5. start services and do an cinder-manage db migrate. Which services you mean -db should be migrated before starting services.
so I've been using packstack to test my upgrade. I used RHOS 2.1 this time and follow the steps above. For #5 I merged a generated packstack answer file with the original answer file from the install. Then I ran packstack --answer-file with the merged file. Results are the same. volumes and snapshots tables are empty as shown. cinder-manage db sync throws the same error I get when running packstack. [root@radez ~(keystone_admin)]$ packstack --answer-file packstack-answers-20130509-150633.txt Welcome to Installer setup utility Packstack changed given value to required value /root/.ssh/id_rsa.pub Installing: Clean Up... [ DONE ] Setting up ssh keys... [ DONE ] Adding pre install manifest entries... [ DONE ] Adding MySQL manifest entries... [ DONE ] Adding QPID manifest entries... [ DONE ] Adding Keystone manifest entries... [ DONE ] Adding Glance Keystone manifest entries... [ DONE ] Adding Glance manifest entries... [ DONE ] Adding Cinder Keystone manifest entries... [ DONE ] Installing dependencies for Cinder... [ DONE ] Checking if the Cinder server has a cinder-volumes vg... [ DONE ] Adding Cinder manifest entries... [ DONE ] Adding Nova API manifest entries... [ DONE ] Adding Nova Keystone manifest entries... [ DONE ] Adding Nova Cert manifest entries... [ DONE ] Adding Nova Conductor manifest entries... [ DONE ] Adding Nova Compute manifest entries... [ DONE ] Adding Nova Network manifest entries... [ DONE ] Adding Nova Scheduler manifest entries... [ DONE ] Adding Nova VNC Proxy manifest entries... [ DONE ] Adding Nova Common manifest entries... [ DONE ] Adding OpenStack Client manifest entries... [ DONE ] Adding Horizon manifest entries... [ DONE ] Preparing servers... [ DONE ] Adding post install manifest entries... [ DONE ] Installing Dependencies... [ DONE ] Copying Puppet modules and manifests... [ DONE ] Applying Puppet manifests... Applying 10.13.128.22_prescript.pp 10.13.128.22_prescript.pp : [ DONE ] Applying 10.13.128.22_mysql.pp Applying 10.13.128.22_qpid.pp 10.13.128.22_mysql.pp : [ DONE ] 10.13.128.22_qpid.pp : [ DONE ] Applying 10.13.128.22_keystone.pp Applying 10.13.128.22_glance.pp Applying 10.13.128.22_cinder.pp 10.13.128.22_keystone.pp : [ DONE ] ERROR : Error during puppet run : notice: /Stage[main]/Cinder::Api/Exec[cinder-manage db_sync]/returns: OperationalError: (OperationalError) (1005, "Can't create table 'cinder.volume_glance_metadata' (errno: 150)") '\nCREATE TABLE volume_glance_metadata (\n\tcreated_at DATETIME, \n\tupdated_at DATETIME, \n\tdeleted_at DATETIME, \n\tdeleted BOOL, \n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tvolume_id VARCHAR(36), \n\tsnapshot_id VARCHAR(36), \n\t`key` VARCHAR(255), \n\tvalue TEXT, \n\tPRIMARY KEY (id), \n\tCHECK (deleted IN (0, 1)), \n\tFOREIGN KEY(volume_id) REFERENCES volumes (id), \n\tFOREIGN KEY(snapshot_id) REFERENCES snapshots (id)\n)ENGINE=InnoDB\n\n' () Please check log file /var/tmp/packstack/20130509-154023-yhCCkC/openstack-setup.log for more information Additional information: * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components. * Did not create a cinder volume group, one already existed * To use the command line tools you need to source the file /root/keystonerc_admin created on 10.13.128.22 * To use the console, browse to http://10.13.128.22/dashboard [root@radez ~(keystone_admin)]$ mysql cinder Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 479 Server version: 5.1.69 Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> select * from volumes; Empty set (0.00 sec) mysql> select * from snapshots; Empty set (0.00 sec) mysql> Bye [root@radez ~(keystone_admin)]$ cinder-manage db migrate usage: cinder-manage db [-h] {version,sync} ... cinder-manage db: error: argument action: invalid choice: 'migrate' (choose from 'version', 'sync') [root@radez ~(keystone_admin)]$ cinder-manage db sync 2013-05-09 15:46:35 ERROR [003_glance_metadata] Exception while creating table 'volume_glance_metedata' Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/migrate_repo/versions/003_glance_metadata.py", line 60, in upgrade volume_glance_metadata.create() File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/schema.py", line 594, in create checkfirst=checkfirst) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2303, in _run_visitor conn._run_visitor(visitorcallable, element, **kwargs) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1973, in _run_visitor **kwargs).traverse_single(element) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/sql/visitors.py", line 106, in traverse_single return meth(obj, **kw) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/ddl.py", line 86, in visit_table self.connection.execute(schema.CreateTable(table)) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute params) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1542, in _execute_ddl compiled File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context context) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context context) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute cursor.execute(statement, parameters) File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute self.errorhandler(self, exc, value) File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue OperationalError: (OperationalError) (1005, "Can't create table 'cinder.volume_glance_metadata' (errno: 150)") '\nCREATE TABLE volume_glance_metadata (\n\tcreated_at DATETIME, \n\tupdated_at DATETIME, \n\tdeleted_at DATETIME, \n\tdeleted BOOL, \n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tvolume_id VARCHAR(36), \n\tsnapshot_id VARCHAR(36), \n\t`key` VARCHAR(255), \n\tvalue TEXT, \n\tPRIMARY KEY (id), \n\tCHECK (deleted IN (0, 1)), \n\tFOREIGN KEY(volume_id) REFERENCES volumes (id), \n\tFOREIGN KEY(snapshot_id) REFERENCES snapshots (id)\n)ENGINE=InnoDB\n\n' () 2013-05-09 15:46:35 CRITICAL [cinder] (OperationalError) (1005, "Can't create table 'cinder.volume_glance_metadata' (errno: 150)") '\nCREATE TABLE volume_glance_metadata (\n\tcreated_at DATETIME, \n\tupdated_at DATETIME, \n\tdeleted_at DATETIME, \n\tdeleted BOOL, \n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tvolume_id VARCHAR(36), \n\tsnapshot_id VARCHAR(36), \n\t`key` VARCHAR(255), \n\tvalue TEXT, \n\tPRIMARY KEY (id), \n\tCHECK (deleted IN (0, 1)), \n\tFOREIGN KEY(volume_id) REFERENCES volumes (id), \n\tFOREIGN KEY(snapshot_id) REFERENCES snapshots (id)\n)ENGINE=InnoDB\n\n' () [root@radez ~(keystone_admin)]$ grep cinder /var/log/yum.log May 09 15:12:49 Installed: python-cinderclient-1.0.1-4.el6ost.noarch May 09 15:12:51 Installed: python-cinder-2012.2.4-3.el6ost.noarch May 09 15:12:54 Installed: openstack-cinder-2012.2.4-3.el6ost.noarch May 09 15:34:01 Updated: python-cinderclient-1.0.3-1.el6.noarch May 09 15:34:28 Updated: python-cinder-2013.1-2.el6.noarch May 09 15:34:42 Updated: openstack-cinder-2013.1-2.el6.noarch
I just reran my upgrade, but I ran cinder-manage db sync before I ran packstack, and the sync worked. Used these steps: 1. install RHOS 2.1 using packstack 2. installed rdo-release 3. yum update 4. stoped cinder services 5. cinder-manage db sync 6. packstack --answer-file merged-ans-file.txt after the packstack run RDO came up and seemed mostly happy. So maybe this ends up being a packstack bug?
I'm not sure exactly what to make of this yet. There are two cases here: 1) The initial failure. It looks like this is maybe expected because you have to run cinder-manage db sync before restarting the services? 2) Failure when using packstack to do an upgrade. I tried to use packstack to upgrade from RHOS 2.1 to RDO as Dan did, basing my answer file on the RHOS 2.1 one and adding the required fields. Packstack says it completes successfully but the Cinder packages are still 2012.2.4. I'm not sure what is expected behavior -- can a packstack expert shed some light here? (Either way I didn't hit the same failure while using packstack.)
upgrades with packstack were never tested functionality so I'm not surprised your hitting problems. I'm not sure its something we would even want to attempt to support with packstack. Of course we would want to allow people to upgrade an install that was originally setup with packstack. Do we know if this works with RHOS cinder where packstack was never involved? Eric, I've just seen another error about packstack missing errors in the puppet logs, https://bugzilla.redhat.com/show_bug.cgi?id=965620 you may not still have them but if you do can you check the packstack logs for puppet errors, it may explain your false positive.
Eric, why is this still on Cinder?
Changing component to packstack -- the core of this issue is around what happens when you attempt to use packstack to upgrade. From Cinder's POV things are ok when following supported procedures.
I believe this has been fixed in Cinder 2013.2.
Auto adding >= MODIFIED bugs to beta
Folsom to Grizzly or Folsom to Havana will not be tested Grizzly to Havana was tested by Ami's team using http://shell.bos.redhat.com/~lkellogg/rhos-upgrade-docs/database-upgrades.html I am moving this to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2013-1859.html