Bug 1031801

This seems like a decent idea to me. Always good to double check before doing something that can cause breakage.

A patched version of this package is available here:

https://github.com/larsks-packages/openstack-neutron/tree/bz/1031801

This is against the rhos-4.0-rhel-6 branch.  The patch introduces a new command, neutron-db-check, that checks the database version and exits with an error if the version is incorrect.

This script is called from the neutron-server init script, but could also be called as an ExecStartPre command for systemd.

I have confirmed that with this change in place:

- Neutron services other than neutron-server start up correctly, AND
- The neutron-server process does not start (and logs a message to stderr and to syslog), AND
- No changes are made to the database, AND
- After running a database upgrade (neutron-db-manage ... upgrade havana), the neutron-server service starts correctly.

RPMS built with mock against epel-6-x86_64 are available from:

http://people.redhat.com/lkellogg/rpms/

Lars: I don't think this patch will work. 'neutron-db-manage current' will display 'None' for both grizzly and havana installs, because AFAICT, neither stamps the database. I just did a havana install to verify (though my repo may be out of date) I think we may have to revert to grepping the output of 'neutron db-manage history' for something like "-> havana".

Also, just so you know, jlibosvar is also assigned the bug for this and is working on it as well.

So basically, if you just do a fresh install of neutron/havana, without having to upgrade the db, no db stamp exists, so the service would never get to start.

Terry: Yeah, that happens :/.  I would argue that we should require a database version to be present, and handle the lack of version information like this:

- Update the install documentation to include the necessary steps for stamping the database, and
- Provide a separate error message if the current database version is reported as "None".  E.g.:

if [ "$neutron_db_version" = None ]; then
        logger -s -p daemon.error -t neutron-db-check \
                "ERROR: you must run the \"neutron-db-manage stamp\" command before running neutron services." 
        exit 1
elif ! [ "$neutron_db_version" = "$NEUTRON_EXPECTED_DB_VERSION" ]; then
        logger -s -p daemon.error -t neutron-db-check \
                "ERROR: you must upgrade the neutron database before starting neutron services." 
        exit 1
fi

I've updated openstack-db-check in git:larsks-packages/openstack-neutron.git to reflect my suggestion in comment 8.

Ugh, that's possibly a useless URL.  Sorry.  I meant:

https://github.com/larsks-packages/openstack-neutron

The current version of db state is stored in the db itself in alembic_version table. Maybe we can create upstream patch that puts head of alembic history in this table. This might be worth discussing upstream, I think.

(In reply to Jakub Libosvar from comment #11)
> The current version of db state is stored in the db itself in
> alembic_version table. Maybe we can create upstream patch that puts head of
> alembic history in this table. This might be worth discussing upstream, I
> think.

The database is probably created by packstack. How about stamping the database in packstack. I cannot find how the db is created in packstack but devstack uses 'upgrade head'. Based on this, I think it would be sufficient to stamp the database with current head after db is created as a part of deployment process.

For the record, I've tested running "upgrade head" against an empty database and it Does The Right Thing (it creates the database tables including the alembic_version table with the appropriate version stamp).

Created attachment 832154 [details]
Proposed patch

I opened bug 1037681 for 3.0.z and bug 1037675 for 4.0 to have database stamped always with current version. Based on the comment above, creating database with neutron-db-manage upgrade should do the thing. This patch depends on fixing previous two bugs.

Created attachment 832459 [details]
Proposed patch

Proposing Lars's patch

There is an unrelated change to the plugin.ini handling in the attached patch. Someone needs to verify that it is correct--I remember needing it to be the .rpmsave when I was originally doing the patch, since I believe that RHOS 3.0 actually packages the plugin.ini file and therefor it will end up being .rmpsave'd. RDO never packaged plugin.ini, so it isn't necessary there.

(In reply to Terry Wilson from comment #16)
> There is an unrelated change to the plugin.ini handling in the attached
> patch. Someone needs to verify that it is correct--I remember needing it to
> be the .rpmsave when I was originally doing the patch, since I believe that
> RHOS 3.0 actually packages the plugin.ini file and therefor it will end up
> being .rmpsave'd. RDO never packaged plugin.ini, so it isn't necessary there.

I tried and plugin.ini symlink is created by packstack. When quantum is removed, symlink is preserved.

# ll /etc/quantum/plugin.ini
ls: cannot access /etc/quantum/plugin.ini: No such file or directory
# packstack --answer-file=packstack-answers-20131204-222447.txt 
...
# ll /etc/quantum/plugin.ini 
lrwxrwxrwx. 1 root root 55 Dec  5 12:16 /etc/quantum/plugin.ini -> /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini
# yum remove openstack-quantum
...
warning: /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini saved as /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini.rpmsave
warning: /etc/quantum/quantum.conf saved as /etc/quantum/quantum.conf.rpmsave
warning: /etc/quantum/metadata_agent.ini saved as /etc/quantum/metadata_agent.ini.rpmsave
warning: /etc/quantum/l3_agent.ini saved as /etc/quantum/l3_agent.ini.rpmsave
warning: /etc/quantum/dhcp_agent.ini saved as /etc/quantum/dhcp_agent.ini.rpmsave
warning: /etc/quantum/api-paste.ini saved as /etc/quantum/api-paste.ini.rpmsave
...
# ll /etc/quantum/plugin.ini 
lrwxrwxrwx. 1 root root 55 Dec  5 12:16 /etc/quantum/plugin.ini -> /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini

jlibosvar, are you sure that you are using rhos and not rdo?

(In reply to Terry Wilson from comment #18)
> jlibosvar, are you sure that you are using rhos and not rdo?

oh, right. I'm sure I used RDO :) I'll recheck.

Created attachment 833198 [details]
Proposed patch

From the spec file it seems that rpm creates plugin.ini. 
I'm attaching rebased patch without change in handling plugin.ini. Once I try the RHOS quantum plugin.ini is renamed after uninstalling quantum, I'll obsolete the old patch.

Comment on attachment 832459 [details]
Proposed patch

Terry was right, in RHOS plugin.ini is saved:

warning: /etc/quantum/plugin.ini saved as /etc/quantum/plugin.ini.rpmsave
# ll /etc/quantum/plugin.ini.rpmsave 
lrwxrwxrwx. 1 root root 55 Dec  5 17:26 /etc/quantum/plugin.ini.rpmsave -> /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.in

The attached patch *looks* right to me. I haven't run through a regular install or an upgrade with it, yet. If you have done both, I'm good with it.

The attached patch *looks* right to me. I haven't run through a regular install or an upgrade with it, yet. If you have done both, I'm good with it.

Created attachment 834750 [details]
Patchset 3

I tested upgrade from rhos-3.0 and clean rhos-4.0 install that included fix of bug 1037675 (packstack build from source) and works. Attaching rebased patch.

Created attachment 834845 [details]
Patchset 4

I accidentally removed patches, pushing new patchset

Created attachment 834870 [details]
Patchset 5

Tested basic installation and upgrade

Roey tested it , more details later

pbrady pointed out that this should go into RDO also.  Do we need to open a new bug?

Verified on Grizzly -> Havana with  :

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
Grizzly Puddle: 2013-11-14.2
Havana Puddle: 2013-12-17.2
openstack-neutron-2013.2-16.el6ost.noarch

Results:
--------
After upgrading quantum to neutron, without update database, the next error message appreaed:
neutron-db-check: ERROR: you must upgrade the neutron database before starting neutron services.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html