Bug 1031801 - Starting neutron before running datbase upgrade can hose database
Starting neutron before running datbase upgrade can hose database
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron (Show other bugs)
4.0
Unspecified Unspecified
urgent Severity urgent
: rc
: 4.0
Assigned To: Jakub Libosvar
Roey Dekel
:
Depends On: 1037675 1037681
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-18 14:45 EST by Lars Kellogg-Stedman
Modified: 2016-04-26 09:38 EDT (History)
11 users (show)

See Also:
Fixed In Version: openstack-neutron-2013.2-15.el6ost
Doc Type: Bug Fix
Doc Text:
Cause: The Neutron database was not in sync with neutron bits. Consequence: If the neutron-server service is started with wrong database version, it can damage the scheme and neutron-server does not operate properly. Fix: Before the neutron-server service is started, the database version is now verified against the value in /etc/sysconfig/openstack-neutron. Result: If the version from the file is not equal to the one in the database, neutron-server now handles this inconsistency properly, and will fail to start.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-19 19:37:13 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Proposed patch (2.92 KB, patch)
2013-12-03 10:31 EST, Jakub Libosvar
no flags Details | Diff
Proposed patch (6.71 KB, patch)
2013-12-04 03:59 EST, Jakub Libosvar
no flags Details | Diff
Proposed patch (6.53 KB, patch)
2013-12-05 10:10 EST, Jakub Libosvar
no flags Details | Diff
Patchset 3 (10.40 KB, patch)
2013-12-10 08:27 EST, Jakub Libosvar
no flags Details | Diff
Patchset 4 (6.57 KB, patch)
2013-12-10 11:11 EST, Jakub Libosvar
no flags Details | Diff
Patchset 5 (6.56 KB, patch)
2013-12-10 12:12 EST, Jakub Libosvar
no flags Details | Diff

  None (edit)
Description Lars Kellogg-Stedman 2013-11-18 14:45:57 EST
Description of problem:

Running "neutron" before upgrading the database schema will lead to inconsistent behavior and will prevent a future database upgrade from completing successfully.

Consider adding something to the service scripts that checks to make sure the database is the appropriate version. For example, one could:

if ! neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini current 2>/dev/null | awk '{print $NF}' | grep havana; then
  echo "ERROR: you must upgrade the neutron database before starting neutron.
  exit 1
fi
Comment 1 Terry Wilson 2013-11-21 12:30:04 EST
This seems like a decent idea to me. Always good to double check before doing something that can cause breakage.
Comment 3 Lars Kellogg-Stedman 2013-12-02 12:43:33 EST
A patched version of this package is available here:

https://github.com/larsks-packages/openstack-neutron/tree/bz/1031801

This is against the rhos-4.0-rhel-6 branch.  The patch introduces a new command, neutron-db-check, that checks the database version and exits with an error if the version is incorrect.

This script is called from the neutron-server init script, but could also be called as an ExecStartPre command for systemd.
Comment 4 Lars Kellogg-Stedman 2013-12-02 14:55:12 EST
I have confirmed that with this change in place:

- Neutron services other than neutron-server start up correctly, AND
- The neutron-server process does not start (and logs a message to stderr and to syslog), AND
- No changes are made to the database, AND
- After running a database upgrade (neutron-db-manage ... upgrade havana), the neutron-server service starts correctly.
Comment 5 Lars Kellogg-Stedman 2013-12-02 15:35:44 EST
RPMS built with mock against epel-6-x86_64 are available from:

http://people.redhat.com/lkellogg/rpms/
Comment 6 Terry Wilson 2013-12-02 17:25:49 EST
Lars: I don't think this patch will work. 'neutron-db-manage current' will display 'None' for both grizzly and havana installs, because AFAICT, neither stamps the database. I just did a havana install to verify (though my repo may be out of date) I think we may have to revert to grepping the output of 'neutron db-manage history' for something like "-> havana".

Also, just so you know, jlibosvar is also assigned the bug for this and is working on it as well.
Comment 7 Terry Wilson 2013-12-02 17:29:49 EST
So basically, if you just do a fresh install of neutron/havana, without having to upgrade the db, no db stamp exists, so the service would never get to start.
Comment 8 Lars Kellogg-Stedman 2013-12-02 20:18:12 EST
Terry: Yeah, that happens :/.  I would argue that we should require a database version to be present, and handle the lack of version information like this:

- Update the install documentation to include the necessary steps for stamping the database, and
- Provide a separate error message if the current database version is reported as "None".  E.g.:

if [ "$neutron_db_version" = None ]; then
        logger -s -p daemon.error -t neutron-db-check \
                "ERROR: you must run the \"neutron-db-manage stamp\" command before running neutron services." 
        exit 1
elif ! [ "$neutron_db_version" = "$NEUTRON_EXPECTED_DB_VERSION" ]; then
        logger -s -p daemon.error -t neutron-db-check \
                "ERROR: you must upgrade the neutron database before starting neutron services." 
        exit 1
fi
Comment 9 Lars Kellogg-Stedman 2013-12-02 20:31:48 EST
I've updated openstack-db-check in git@github.com:larsks-packages/openstack-neutron.git to reflect my suggestion in comment 8.
Comment 10 Lars Kellogg-Stedman 2013-12-02 20:37:20 EST
Ugh, that's possibly a useless URL.  Sorry.  I meant:

https://github.com/larsks-packages/openstack-neutron
Comment 11 Jakub Libosvar 2013-12-03 06:32:38 EST
The current version of db state is stored in the db itself in alembic_version table. Maybe we can create upstream patch that puts head of alembic history in this table. This might be worth discussing upstream, I think.
Comment 12 Jakub Libosvar 2013-12-03 08:42:57 EST
(In reply to Jakub Libosvar from comment #11)
> The current version of db state is stored in the db itself in
> alembic_version table. Maybe we can create upstream patch that puts head of
> alembic history in this table. This might be worth discussing upstream, I
> think.

The database is probably created by packstack. How about stamping the database in packstack. I cannot find how the db is created in packstack but devstack uses 'upgrade head'. Based on this, I think it would be sufficient to stamp the database with current head after db is created as a part of deployment process.
Comment 13 Lars Kellogg-Stedman 2013-12-03 09:55:09 EST
For the record, I've tested running "upgrade head" against an empty database and it Does The Right Thing (it creates the database tables including the alembic_version table with the appropriate version stamp).
Comment 14 Jakub Libosvar 2013-12-03 10:31:20 EST
Created attachment 832154 [details]
Proposed patch

I opened bug 1037681 for 3.0.z and bug 1037675 for 4.0 to have database stamped always with current version. Based on the comment above, creating database with neutron-db-manage upgrade should do the thing. This patch depends on fixing previous two bugs.
Comment 15 Jakub Libosvar 2013-12-04 03:59:45 EST
Created attachment 832459 [details]
Proposed patch

Proposing Lars's patch
Comment 16 Terry Wilson 2013-12-04 12:34:29 EST
There is an unrelated change to the plugin.ini handling in the attached patch. Someone needs to verify that it is correct--I remember needing it to be the .rpmsave when I was originally doing the patch, since I believe that RHOS 3.0 actually packages the plugin.ini file and therefor it will end up being .rmpsave'd. RDO never packaged plugin.ini, so it isn't necessary there.
Comment 17 Jakub Libosvar 2013-12-05 08:33:00 EST
(In reply to Terry Wilson from comment #16)
> There is an unrelated change to the plugin.ini handling in the attached
> patch. Someone needs to verify that it is correct--I remember needing it to
> be the .rpmsave when I was originally doing the patch, since I believe that
> RHOS 3.0 actually packages the plugin.ini file and therefor it will end up
> being .rmpsave'd. RDO never packaged plugin.ini, so it isn't necessary there.

I tried and plugin.ini symlink is created by packstack. When quantum is removed, symlink is preserved.

# ll /etc/quantum/plugin.ini
ls: cannot access /etc/quantum/plugin.ini: No such file or directory
# packstack --answer-file=packstack-answers-20131204-222447.txt 
...
# ll /etc/quantum/plugin.ini 
lrwxrwxrwx. 1 root root 55 Dec  5 12:16 /etc/quantum/plugin.ini -> /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini
# yum remove openstack-quantum
...
warning: /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini saved as /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini.rpmsave
warning: /etc/quantum/quantum.conf saved as /etc/quantum/quantum.conf.rpmsave
warning: /etc/quantum/metadata_agent.ini saved as /etc/quantum/metadata_agent.ini.rpmsave
warning: /etc/quantum/l3_agent.ini saved as /etc/quantum/l3_agent.ini.rpmsave
warning: /etc/quantum/dhcp_agent.ini saved as /etc/quantum/dhcp_agent.ini.rpmsave
warning: /etc/quantum/api-paste.ini saved as /etc/quantum/api-paste.ini.rpmsave
...
# ll /etc/quantum/plugin.ini 
lrwxrwxrwx. 1 root root 55 Dec  5 12:16 /etc/quantum/plugin.ini -> /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini
Comment 18 Terry Wilson 2013-12-05 09:40:17 EST
jlibosvar, are you sure that you are using rhos and not rdo?
Comment 19 Jakub Libosvar 2013-12-05 09:44:08 EST
(In reply to Terry Wilson from comment #18)
> jlibosvar, are you sure that you are using rhos and not rdo?

oh, right. I'm sure I used RDO :) I'll recheck.
Comment 20 Jakub Libosvar 2013-12-05 10:10:32 EST
Created attachment 833198 [details]
Proposed patch

From the spec file it seems that rpm creates plugin.ini. 
I'm attaching rebased patch without change in handling plugin.ini. Once I try the RHOS quantum plugin.ini is renamed after uninstalling quantum, I'll obsolete the old patch.
Comment 21 Jakub Libosvar 2013-12-05 12:08:15 EST
Comment on attachment 832459 [details]
Proposed patch

Terry was right, in RHOS plugin.ini is saved:

warning: /etc/quantum/plugin.ini saved as /etc/quantum/plugin.ini.rpmsave
# ll /etc/quantum/plugin.ini.rpmsave 
lrwxrwxrwx. 1 root root 55 Dec  5 17:26 /etc/quantum/plugin.ini.rpmsave -> /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.in
Comment 22 Terry Wilson 2013-12-09 22:40:23 EST
The attached patch *looks* right to me. I haven't run through a regular install or an upgrade with it, yet. If you have done both, I'm good with it.
Comment 23 Terry Wilson 2013-12-09 22:42:28 EST
The attached patch *looks* right to me. I haven't run through a regular install or an upgrade with it, yet. If you have done both, I'm good with it.
Comment 24 Jakub Libosvar 2013-12-10 08:27:11 EST
Created attachment 834750 [details]
Patchset 3

I tested upgrade from rhos-3.0 and clean rhos-4.0 install that included fix of bug 1037675 (packstack build from source) and works. Attaching rebased patch.
Comment 25 Jakub Libosvar 2013-12-10 11:11:34 EST
Created attachment 834845 [details]
Patchset 4

I accidentally removed patches, pushing new patchset
Comment 26 Jakub Libosvar 2013-12-10 12:12:10 EST
Created attachment 834870 [details]
Patchset 5

Tested basic installation and upgrade
Comment 29 Ofer Blaut 2013-12-18 07:55:56 EST
Roey tested it , more details later
Comment 30 Lars Kellogg-Stedman 2013-12-18 09:08:27 EST
pbrady pointed out that this should go into RDO also.  Do we need to open a new bug?
Comment 31 Roey Dekel 2013-12-19 05:08:08 EST
Verified on Grizzly -> Havana with  :

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
Grizzly Puddle: 2013-11-14.2
Havana Puddle: 2013-12-17.2
openstack-neutron-2013.2-16.el6ost.noarch

Results:
--------
After upgrading quantum to neutron, without update database, the next error message appreaed:
neutron-db-check: ERROR: you must upgrade the neutron database before starting neutron services.
Comment 33 errata-xmlrpc 2013-12-19 19:37:13 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html

Note You need to log in before you can comment on or make changes to this bug.