Description of problem: If the database schema changed between two package versions (e.g. Havana-3 to Havana-rc1), the schema is not updated automatically by the package update. Version-Release number of selected component (if applicable): openstack-keystone-2013.2-0.14.rc1.el6.noarch How reproducible: Always. Steps to Reproduce: 1. Install OpenStack (pre-rc1) 2. Update packages to rc1 3. less /var/log/keystone/keystone.log Actual results: (OperationalError) (1054, "Unknown column 'user.default_project_id' in 'field list'") 'SELECT user.id AS user_id, user.name AS user_name, user.domain_id AS user_domain_id, user.password AS user_password, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id \nFROM user' () (OperationalError) (1054, "Unknown column 'user.default_project_id' in 'field list'") 'SELECT user.id AS user_id, user.name AS user_name, user.domain_id AS user_domain_id, user.password AS user_password, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id \nFROM user \nWHERE user.name = %s AND user.domain_id = %s' ('admin', 'default') (OperationalError) (1054, "Unknown column 'user.default_project_id' in 'field list'") 'SELECT user.id AS user_id, user.name AS user_name, user.domain_id AS user_domain_id, user.password AS user_password, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id \nFROM user \nWHERE user.name = %s AND user.domain_id = %s' ('glance', 'default') (OperationalError) (1054, "Unknown column 'user.default_project_id' in 'field list'") 'SELECT user.id AS user_id, user.name AS user_name, user.domain_id AS user_domain_id, user.password AS user_password, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id \nFROM user \nWHERE user.name = %s AND user.domain_id = %s' ('neutron', 'default') Expected results: DB schema is updated automatically by the package in %post. Additional info: keystone-manage db_sync successfully resolved the above issues.
This is a general issue also noticed during the havana _development_ cycle with glance for example. During updates to a stable series, this should not happen. Though we'll also hit the issue when upgrading between separate stable releases. So where to do the sync. In the package post upgrade script, or the service init script? Probably the former. One method might be to encapsulate this in the existing openstack-db script like `openstack-db --service keystone --update`. I've just added that support to openstack-db: # openstack-db --drop --rootpw ... --service cinder --password ... Verified connectivity to MySQL. Dropping 'cinder' database. Complete! # openstack-db --service cinder --update Can't determine the existing sync level. Please ensure the database is available and already initialised. # openstack-db --init --rootpw ... --service cinder --password ... Verified connectivity to MySQL. Creating 'cinder' database. Updating 'cinder' database password in /etc/cinder/cinder.conf Initializing the cinder database, please wait... Complete! # openstack-db --service cinder --update Complete! So a package post upgrade scriplet could safely make the --update call above, before the service is restarted. Would we need to stop the (old) service first? Probably not. Another caveat is that openstack-db doesn't currently manage the neutron db. Another caveat is that openstack-db doesn't current manage the ceilometer db, which defaults to mongo, and there is no ceilometer-manage command, so maybe this functionality isn't needed there?
I think the update command has to run as part of the service startup, because if you do it in %post then it won't work if the database server is unavailable -- which is certainly the case during initial system install (e.g., Anaconda). It's also entirely possible that someone would perform an upgrade while the database server was stopped. In either of these cases, we would end up in a situation where the schema upgrade didn't happen, leading to breakage when the service starts up. In some cases, I think the db_sync command is also responsible for creating the initial schema if it doesn't exist, right? So this is relevant not just for upgrades but for first-time installs, too. > Another caveat is that openstack-db doesn't currently manage the neutron db. Neutron database upgrades are surprisingly tricky vs. other openstack components.
During initial install, services aren't even enabled. This is standard Red Hat policy and thus higher level operations (such as openstack-db --init) are required to enable and configure services. Currently for initial installs, {packstack,foreman} call out to puppet modules which initiate the db sync. Those will ensure that the db is configured and running initially. It's a fair point on the DB being down on upgrade which would break stuff. However service start always trying to sync the DB may have other issues. Hmm I remember glance essex issues with that, looking... So we have these two commits that _disabled_ the auto sync on startup for glance: https://github.com/redhat-openstack/glance/commit/510fbfede http://pkgs.fedoraproject.org/cgit/openstack-glance.git/commit/?h=f17&id=25812c08c eglynn reminds me that we did this because it was inconsistent with the other services, and also led to race conditions when launching both the api and regsitry services, which new to the V2 API, both access/sync the DB.
> During initial install, services aren't even enabled. Yes, that's what I was saying, and was in general the main point of my comment :). Since the database might not even be local to the server (so no amount of cleverness in openstack-db can make it available), I don't think we can sanely perform this upgrade as part of the package scripts. If we can't do this as part of the service startup, then maybe it really belongs to the deployment tools? > eglynn reminds me that we did this because it was inconsistent with > the other services, Well, I agree that we don't want one-offs. If were to make this change for one service we would want to do it across the board. > and also led to race conditions when launching > both the api and regsitry services, which new to the V2 API, > both access/sync the DB. I guess that's a tougher problem to solve. I was going to suggest that one could solve this by simply sequencing the startup scripts appropriately, but that wouldn't work if they were running on different hosts. Couldn't we also run into the same sequencing problems with package upgrades? If server A running the API service is upgraded first, while server B is still running the registry service expecting the older database schema?
Yes this seems like the package upgrades should invoke as little logic as possible, and be driven by higher level tools that order upgrades (across hosts), while ensuring the DB is enabled etc. `openstack-db --update` is probably still useful in this context, but probably not to be called from the packages themselves.