Bug 1018074 - $openstack_service does not upgrade db schema on package update
$openstack_service does not upgrade db schema on package update
Status: CLOSED NOTABUG
Product: RDO
Classification: Community
Component: distribution (Show other bugs)
unspecified
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Lars Kellogg-Stedman
Ami Jeain
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-11 02:59 EDT by Sandro Mathys
Modified: 2016-04-26 16:03 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-09-08 15:00:20 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Sandro Mathys 2013-10-11 02:59:55 EDT
Description of problem:
If the database schema changed between two package versions (e.g. Havana-3 to Havana-rc1), the schema is not updated automatically by the package update.

Version-Release number of selected component (if applicable):
openstack-keystone-2013.2-0.14.rc1.el6.noarch

How reproducible:
Always.

Steps to Reproduce:
1. Install OpenStack (pre-rc1)
2. Update packages to rc1
3. less /var/log/keystone/keystone.log

Actual results:
(OperationalError) (1054, "Unknown column 'user.default_project_id' in 'field list'") 'SELECT user.id AS user_id, user.name AS user_name, user.domain_id AS user_domain_id, user.password AS user_password, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id \nFROM user' ()
(OperationalError) (1054, "Unknown column 'user.default_project_id' in 'field list'") 'SELECT user.id AS user_id, user.name AS user_name, user.domain_id AS user_domain_id, user.password AS user_password, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id \nFROM user \nWHERE user.name = %s AND user.domain_id = %s' ('admin', 'default')
(OperationalError) (1054, "Unknown column 'user.default_project_id' in 'field list'") 'SELECT user.id AS user_id, user.name AS user_name, user.domain_id AS user_domain_id, user.password AS user_password, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id \nFROM user \nWHERE user.name = %s AND user.domain_id = %s' ('glance', 'default')
(OperationalError) (1054, "Unknown column 'user.default_project_id' in 'field list'") 'SELECT user.id AS user_id, user.name AS user_name, user.domain_id AS user_domain_id, user.password AS user_password, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id \nFROM user \nWHERE user.name = %s AND user.domain_id = %s' ('neutron', 'default')

Expected results:
DB schema is updated automatically by the package in %post.

Additional info:
keystone-manage db_sync successfully resolved the above issues.
Comment 1 Pádraig Brady 2013-10-11 09:20:08 EDT
This is a general issue also noticed during the havana _development_ cycle
with glance for example. During updates to a stable series, this should not happen. Though we'll also hit the issue when upgrading between separate
stable releases.

So where to do the sync. In the package post upgrade script, or the service init script? Probably the former. One method might be to encapsulate this in the existing openstack-db script like `openstack-db --service keystone --update`.
I've just added that support to openstack-db:

  # openstack-db --drop --rootpw ... --service cinder --password ...
  Verified connectivity to MySQL.
  Dropping 'cinder' database.
  Complete!
  # openstack-db --service cinder --update
  Can't determine the existing sync level.
  Please ensure the database is available and already initialised.
  # openstack-db --init --rootpw ... --service cinder --password ...
  Verified connectivity to MySQL.
  Creating 'cinder' database.
  Updating 'cinder' database password in /etc/cinder/cinder.conf
  Initializing the cinder database, please wait...
  Complete!
  # openstack-db --service cinder --update
  Complete!

So a package post upgrade scriplet could safely
make the --update call above, before the service is restarted.
Would we need to stop the (old) service first? Probably not.

Another caveat is that openstack-db doesn't currently manage the neutron db.

Another caveat is that openstack-db doesn't current manage the ceilometer db,
which defaults to mongo, and there is no ceilometer-manage command, so maybe
this functionality isn't needed there?
Comment 2 Lars Kellogg-Stedman 2013-10-24 09:58:35 EDT
I think the update command has to run as part of the service startup, because if you do it in %post then it won't work if the database server is unavailable -- which is certainly the case during initial system install (e.g., Anaconda).  It's also entirely possible that someone would perform an upgrade while the database server was stopped.

In either of these cases, we would end up in a situation where the schema upgrade didn't happen, leading to breakage when the service starts up.  In some cases, I think the db_sync command is also responsible for creating the initial schema if it doesn't exist, right?  So this is relevant not just for upgrades but for first-time installs, too.

> Another caveat is that openstack-db doesn't currently manage the neutron db.

Neutron database upgrades are surprisingly tricky vs. other openstack components.
Comment 3 Pádraig Brady 2013-10-24 11:52:53 EDT
During initial install, services aren't even enabled.
This is standard Red Hat policy and thus higher level operations
(such as openstack-db --init) are required to enable and configure services.

Currently for initial installs, {packstack,foreman} call out to puppet modules
which initiate the db sync. Those will ensure that the db is configured
and running initially.

It's a fair point on the DB being down on upgrade which would break stuff.
However service start always trying to sync the DB may have other issues.
Hmm I remember glance essex issues with that, looking...

So we have these two commits that _disabled_ the auto sync on startup for glance:
https://github.com/redhat-openstack/glance/commit/510fbfede
http://pkgs.fedoraproject.org/cgit/openstack-glance.git/commit/?h=f17&id=25812c08c
eglynn reminds me that we did this because it was inconsistent with
the other services, and also led to race conditions when launching
both the api and regsitry services, which new to the V2 API,
both access/sync the DB.
Comment 4 Lars Kellogg-Stedman 2013-10-24 12:06:03 EDT
> During initial install, services aren't even enabled.

Yes, that's what I was saying, and was in general the main point of my comment :).  

Since the database might not even be local to the server (so no amount of cleverness in openstack-db can make it available), I don't think we can sanely perform this upgrade as part of the package scripts.

If we can't do this as part of the service startup, then maybe it really belongs to the deployment tools?

> eglynn reminds me that we did this because it was inconsistent with
> the other services,

Well, I agree that we don't want one-offs.  If were to make this change for one service we would want to do it across the board.

> and also led to race conditions when launching
> both the api and regsitry services, which new to the V2 API,
> both access/sync the DB.

I guess that's a tougher problem to solve.  I was going to suggest that one could solve this by simply sequencing the startup scripts appropriately, but that wouldn't work if they were running on different hosts.

Couldn't we also run into the same sequencing problems with package upgrades?  If server A running the API service is upgraded first, while server B is still running the registry service expecting the older database schema?
Comment 5 Pádraig Brady 2013-10-24 12:31:48 EDT
Yes this seems like the package upgrades should invoke as little logic as possible, and be driven by higher level tools that order upgrades (across hosts), while ensuring the DB is enabled etc.

`openstack-db --update` is probably still useful in this context,
but probably not to be called from the packages themselves.

Note You need to log in before you can comment on or make changes to this bug.