+++ This bug was initially created as a clone of Bug #1045680 +++ After upgrading from Beta to GA of neutron-server: [root@fab3 neutron]# /etc/init.d/neutron-server start neutron-db-check: ERROR: neutron database does not have version information. You must run the "neutron-db-manage stamp" command before starting neutron services. [root@fab3 neutron]# neutron-db-manage stamp usage: neutron-db-manage stamp [-h] [--sql] revision neutron-db-manage stamp: error: too few arguments You have mail in /var/spool/mail/root [root@fab3 neutron]# neutron-db-manage stamp --sql 1 No handlers could be found for logger "neutron.common.legacy" Traceback (most recent call last): File "/usr/bin/neutron-db-manage", line 10, in <module> sys.exit(main()) File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 143, in main CONF.command.func(config, CONF.command.name) File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 86, in do_stamp sql=CONF.command.sql) File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 59, in do_alembic_command getattr(alembic_command, cmd)(config, *args, **kwargs) File "/usr/lib/python2.6/site-packages/alembic/command.py", line 214, in stamp script.run_env() File "/usr/lib/python2.6/site-packages/alembic/script.py", line 191, in run_env util.load_python_file(self.dir, 'env.py') File "/usr/lib/python2.6/site-packages/alembic/util.py", line 186, in load_python_file module = imp.load_source(module_id, path, open(path, 'rb')) File "/usr/lib/python2.6/site-packages/neutron/db/migration/alembic_migrations/env.py", line 44, in <module> importutils.import_class(class_path) File "/usr/lib/python2.6/site-packages/neutron/openstack/common/importutils.py", line 35, in import_class traceback.format_exception(*sys.exc_info()))) ImportError: Class cannot be found (['Traceback (most recent call last):\n', ' File "/usr/lib/python2.6/site-packages/neutron/openstack/common/importutils.py", line 30, in import_class\n __import__(mod_str)\n', 'ValueError: Empty module name\n']) --- Additional comment from Fabio Massimo Di Nitto on 2013-12-21 01:38:25 EST --- neutron-db-manage --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini stamp head appears to solve the issue but: 1) it's all but obvious. 2) no this is not related to https://bugzilla.redhat.com/show_bug.cgi?id=1037675 the installation was done manually. 3) I didn't notice any call to neutron-db-manage in the official docs either. 4) neutron-db-manage should seriously either be invoked automatically or provide a man page explaining how to invoke it. --- Additional comment from lpeer on 2013-12-23 04:42:41 EST --- > the installation was done manually. In case you installed it manually you need to stamp the data-base manually exactly as you did. I think you are missing the documentation part so I'm changing this bug to documentation. The documentation should say that if you install Neutron manually you should run neutron-db-manage manually to stamp the DB. > 3) I didn't notice any call to neutron-db-manage in the official docs either. I agree, that's why I'm moving it to documentation > 4) neutron-db-manage should seriously either be invoked automatically or > provide a man page explaining how to invoke it. I agree that a man page would be useful, Not sure about the automatic invocation, by which process? it is part of the installation process and we take care of that within packstack and foreman (should be in the puppet modules they are sharing) --- Additional comment from Fabio Massimo Di Nitto on 2013-12-23 10:34:45 EST --- (In reply to lpeer from comment #2) > > the installation was done manually. > In case you installed it manually you need to stamp the data-base manually > exactly as you did. > > I think you are missing the documentation part so I'm changing this bug to > documentation. > The documentation should say that if you install Neutron manually you should > run > neutron-db-manage manually to stamp the DB. > That's not enough. You still need to fix the traceback and provide useful output from invoking neutron-db-manage. > > 3) I didn't notice any call to neutron-db-manage in the official docs either. > > I agree, that's why I'm moving it to documentation If anything it needs to be cloned > > > 4) neutron-db-manage should seriously either be invoked automatically or > > provide a man page explaining how to invoke it. > > I agree that a man page would be useful, Not sure about the automatic > invocation, by which process? it is part of the installation process and we > take care of that within packstack and foreman (should be in the puppet > modules they are sharing) rpm %pre %post exists to fit this kind of situations. You can easily verify from what version to what version of the package you are upgrading and invoke neutron-db-manage with proper arguments.
> rpm %pre %post exists to fit this kind of situations. You can easily verify > from what version to what version of the package you are upgrading and > invoke neutron-db-manage with proper arguments. We do handle this in the upgrade process because then we can assume the DB is installed, but for new installation there is no reason to assume the DB is there.
(In reply to lpeer from comment #2) > > rpm %pre %post exists to fit this kind of situations. You can easily verify > > from what version to what version of the package you are upgrading and > > invoke neutron-db-manage with proper arguments. > > We do handle this in the upgrade process because then we can assume the DB > is installed, but for new installation there is no reason to assume the DB > is there. I think you misunderstood the problem here. I had installed the neutron-server from Beta, done my config et all. Upgrading to GA did break as it does require manual intervention. An intervention that can easily be taken care in the rpm itself. So I really have an issue to understand why this bug cannot be fixed. An upgrade that breaks current installations is just a broken upgrade.
Fabio, the problem *can't* be taken care of reliably in the RPM. There is no guarantee that the database is available at all during the RPM transaction. We can try to fix it (and we do in the case of grizzly -> havana upgrade solely because starting the neutron-server w/o a stamped db when upgrading from grizzly causes the database to be badly broken), but it really isn't the right place to do this kind of thing. When doing a manual install, it is currently broken behavior to not manually stamp/upgrade the database--something we need to document, at least from what I can tell. All of the install tools that are supported now do this. It looks like we still ship the various neutron-*-setup scripts, and they *don't* stamp the database--which is a problem. But I'm not sure if the problem is that we ship those scripts at all or if it is that they should be modified.
(In reply to Terry Wilson from comment #4) > Fabio, the problem *can't* be taken care of reliably in the RPM. There is no > guarantee that the database is available at all during the RPM transaction. > We can try to fix it (and we do in the case of grizzly -> havana upgrade > solely because starting the neutron-server w/o a stamped db when upgrading > from grizzly causes the database to be badly broken), but it really isn't > the right place to do this kind of thing. Well it's even worst breaking in between Beta and GA to be honest. A change of this kind should have landed way before Beta (requiring a stamped db). > When doing a manual install, it is currently broken behavior to not manually > stamp/upgrade the database--something we need to document, at least from > what I can tell. There is a bug filed already for the documentation. > All of the install tools that are supported now do this. It > looks like we still ship the various neutron-*-setup scripts, and they > *don't* stamp the database--which is a problem. But I'm not sure if the > problem is that we ship those scripts at all or if it is that they should be > modified. The problem here is what we promise to RHEL customers about API/ABI compatibility and such. Breaking between Beta and GA is pretty bad. It's a matter of customer perception and the fact that the way out of that error is all but obvious. First thing would be to modify the error to be useful. See what's printed vs what needs to be executed in description of the bug. Also noting that the DB needs to be running when executing (just to be 100% clear to less knowledgeable sysadmins). Another simple thing to do would be to have a wrapper to fix it and especially make it easier to express from what to what we are upgrading. We have probably 3 options right now (sorry I don't have the full history of openstack handy): neutron-db-stamp --upgrade-from folsom|grizzly|havana (make it explicit in the help or the init error that docs pre version had an error and didn't stamp the db... so havana is a valid option if updating from package version X.Y-1....) An automatic upgrade/stamp of the db, since you made a good point about the DB that might not be available at rpm transaction time, can only be done using a combination of rpm and neutron-server runtime. That is, as you already guessed, just not worth the troubles (rpm would tell us from where we are upgrading, runtime would have the logic to fix the db by calling into whatever tool does it). Let's at least fix the error with a proper explanation on how to move on. Agreed?
> Let's at least fix the error with a proper explanation on how to move on. > Agreed? Which message you think should be fixed - 1. when you start the server without a stamp, you get the error - neutron-db-check: ERROR: neutron database does not have version information. You must run the "neutron-db-manage stamp" command before starting neutron services. 2. When you use 'neutron-db-manage', you get a message on how to use the tool - usage: neutron-db-manage [-h] [--config-dir DIR] [--config-file PATH] [--version] {upgrade,branches,check_migration,stamp,current,downgrade,revision,history} ... 3. When you use 'neutron-db-manage stamp' you get a message how to use the stamp option in the tool [root@vpn-6-140 ~]# neutron-db-manage stamp usage: neutron-db-manage stamp [-h] [--sql] revision
(In reply to lpeer from comment #6) > > Let's at least fix the error with a proper explanation on how to move on. > > Agreed? > > Which message you think should be fixed - > > 1. when you start the server without a stamp, you get the error - > > neutron-db-check: ERROR: neutron database does not have version information. > You must run the "neutron-db-manage stamp" command before starting neutron > services. > this one needs to be fixed. Just provide the full correct invocation instead of redirecting to man pages and help text. > 2. When you use 'neutron-db-manage', you get a message on how to use the > tool - > > usage: neutron-db-manage [-h] [--config-dir DIR] [--config-file PATH] > [--version] > > {upgrade,branches,check_migration,stamp,current,downgrade,revision,history} > ... > > 3. When you use 'neutron-db-manage stamp' you get a message how to use the > stamp option in the tool > > [root@vpn-6-140 ~]# neutron-db-manage stamp > usage: neutron-db-manage stamp [-h] [--sql] revision This is the command I had to use to stamp the db: neutron-db-manage --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini stamp head based on the help text, I had to combine 2 help text and guess "head" because I couldn't find a reference to what revision is (found out about "head" only thanks to google). A reasonable output from init script would be: neutron-db-check: ERROR: neutron database does not have version information. You must run the "neutron-db-manage stamp" command before starting neutron services. Example: "neutron-db-manage --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini stamp {previous_install_version}" Where {previous_install_version} can be folsom|grizzly|havana based on.. (explain detecting criteria here for sysadmins that are only performing the upgrade and no idea of what was installed before hand) Yes it's long, it saves admins time to figure out how to comine several help texts and finding config files that are not in obvious locations (FSH dictates config file should be in /etc and not in /usr for example).
I concur with Fabio that we should address this sooner rather than later by modifying the error message in the presence of an unstamped database to be more verbose as in comment #6. In particular, since we're only changing the error message and making no changes in functionality I see no reason to push this to 5.0. I'll put together a patch with an updated error message this afternoon.
...where by "comment #6" I actually meant "comment #7".
Created attachment 845001 [details] Extended error messages for neutron-db-check This patch adds extended error messages to neutron-db-check. There's a chunk of whitespace changes, too, because here documents don't work with spaces.
Lars: The patch looks good. I haven't tested an upgrade with it, though.
Verified on Grizzly -> Havana with : Version-Release number of selected component (if applicable): ------------------------------------------------------------- Grizzly puddle: 2014-01-02.2 Havana puddle: 2014-01-16.1 python-neutronclient-2.3.1-2.el6ost.noarch openstack-neutron-2013.2.1-4.el6ost.noarch openstack-neutron-openvswitch-2013.2.1-4.el6ost.noarch python-neutron-2013.2.1-4.el6ost.noarch Results: -------- Trying to start neutron with unupgraded database: neutron-db-check: ERROR: you must upgrade the neutron database before starting neutron services. You can upgrade the neutron database by running: openstack-db --service neutron --update Comment: -------- The error for the following command: neutron-db-manage wasn't changed.
(In reply to Roey Dekel from comment #14) > Verified on Grizzly -> Havana with : > > Version-Release number of selected component (if applicable): > ------------------------------------------------------------- > Grizzly puddle: 2014-01-02.2 > Havana puddle: 2014-01-16.1 > python-neutronclient-2.3.1-2.el6ost.noarch > openstack-neutron-2013.2.1-4.el6ost.noarch > openstack-neutron-openvswitch-2013.2.1-4.el6ost.noarch > python-neutron-2013.2.1-4.el6ost.noarch > > Results: > -------- > Trying to start neutron with unupgraded database: > > neutron-db-check: ERROR: you must upgrade the neutron database before > starting neutron services. > > You can upgrade the neutron database by running: > > openstack-db --service neutron --update > > Comment: > -------- > The error for the following command: > neutron-db-manage > wasn't changed. You did set needinfo on me, but what's the question exactly? If the command reported by init script works, and it's simple, than it's fine by me. As long the user gets a one shot thing to do that doesn't involve black magic, we are all good.
> Comment: > -------- > The error for the following command: > neutron-db-manage > wasn't changed. is the question. I think the current fix help as long as the: openstack-db --service neutron --update works. I can't think of a case it will fail, but if so, you're bug about neutron-db-manage (or it's unclearness in other words) will be relevent.
(In reply to Roey Dekel from comment #16) > > Comment: > > -------- > > The error for the following command: > > neutron-db-manage > > wasn't changed. > is the question. > > I think the current fix help as long as the: > openstack-db --service neutron --update > works. I can't think of a case it will fail, but if so, you're bug about > neutron-db-manage (or it's unclearness in other words) will be relevent. Understood. I think as long as the first command display is "easy", we should be fine. If something goes wrong on update, there might be a deeper issue that requires investigation anyway.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2014-0091.html
*** Bug 1077431 has been marked as a duplicate of this bug. ***