Bug 1396360 - nova_api database not populated/configured after OSP8->OSP9 upgrade
Summary: nova_api database not populated/configured after OSP8->OSP9 upgrade
Keywords:
Status: CLOSED DUPLICATE of bug 1413686
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Sofer Athlan-Guyot
QA Contact: Arik Chernetsky
URL:
Whiteboard:
Depends On:
Blocks: 1373538 1400606
TreeView+ depends on / blocked
 
Reported: 2016-11-18 07:27 UTC by kamil.rogon
Modified: 2017-03-17 10:15 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-17 10:15:48 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 428093 0 None MERGED Ensure working compute node during the whole migration. 2020-12-10 13:55:39 UTC

Description kamil.rogon 2016-11-18 07:27:14 UTC
Description of problem:
During one of the upgrade steps I have included additional template called "major-upgrade-pacemaker.yaml". After this step I am not able to boot new instance on overcloud. It seems like nova is missing connection parameter under api_database section for new database, database is not created/populated and there are no grants for nova user in database. 

Version-Release number of selected component (if applicable):
[root@osp9-director ~]# rpm -qa | grep heat
openstack-heat-templates-0-0.3.96a0b0bgit.el7ost.noarch
openstack-tripleo-heat-templates-kilo-0.8.14-16.el7ost.noarch
openstack-heat-engine-6.0.0-11.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-34.el7ost.noarch
openstack-heat-api-cfn-6.0.0-11.el7ost.noarch
openstack-heat-api-cloudwatch-6.0.0-11.el7ost.noarch
openstack-heat-common-6.0.0-11.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-34.el7ost.noarch
python-heatclient-1.2.0-1.el7ost.noarch
openstack-heat-api-6.0.0-11.el7ost.noarch
python-heat-tests-6.0.0-11.el7ost.noarch

[root@overcloud-controller-0 ~]# rpm -qa | grep nova
openstack-nova-compute-13.1.1-7.el7ost.noarch
openstack-nova-console-13.1.1-7.el7ost.noarch
python-nova-13.1.1-7.el7ost.noarch
openstack-nova-novncproxy-13.1.1-7.el7ost.noarch
openstack-nova-common-13.1.1-7.el7ost.noarch
openstack-nova-conductor-13.1.1-7.el7ost.noarch
python-novaclient-3.3.2-1.el7ost.noarch
openstack-nova-api-13.1.1-7.el7ost.noarch
openstack-nova-scheduler-13.1.1-7.el7ost.noarch
openstack-nova-cert-13.1.1-7.el7ost.noarch

How reproducible:
always


Steps to Reproduce:
1. execute major version upgrade from director including "major-upgrade-pacemaker.yaml" step.
2. Try to boot VM from director using
[stack@osp9-director ~]$ source overcloudrc
[stack@osp8-director ~]$ nova --debug boot --flavor 2 --image rhel7 --nic net-id=cc5c67be-1ce0-4606-9d63-5f31fe361ffa novaapi_test

Actual results:
[...]
DEBUG (shell:1082) Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_db.exception.CantStartEngineError'> (HTTP 500) (Request-ID: req-767a0ecb-f794-4f99-abbb-b4c87d52f903)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1080, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1007, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 589, in do_boot
    server = cs.servers.create(*boot_args, **boot_kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1233, in create
    **boot_kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 667, in _boot
    return_raw=return_raw, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 345, in _create
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 179, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 94, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_db.exception.CantStartEngineError'> (HTTP 500) (Request-ID: req-767a0ecb-f794-4f99-abbb-b4c87d52f903)
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_db.exception.CantStartEngineError'> (HTTP 500) (Request-ID: req-767a0ecb-f794-4f99-abbb-b4c87d52f903)

Expected results:
VM started (available as building/active in "nova list").


Additional info:
Traceback from nova-api.log

2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions [req-d5f68d5d-18b1-4d3d-a1f2-e0f7b73cde86 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] Unexpecte
d exception in API method
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 629, in create
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     **create_kwargs)
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1563, in create
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1176, in _create_instance
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     instance_group, check_server_group_quota, filter_properties)
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1034, in _provision_instances
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     quotas.rollback()
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     self.force_reraise()
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 976, in _provision_instances
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     req_spec.create()
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 223, in wrapper
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     return fn(self, *args, **kwargs)
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/request_spec.py", line 432, in create
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     db_spec = self._create_in_db(self._context, updates)
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 708, in wrapper
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     with self._transaction_scope(context):
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     return self.gen.next()
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 759, in _transaction_scope
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     allow_async=self._allow_async) as resource:
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     return self.gen.next()
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 491, in _session
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     bind=self.connection, mode=self.mode)
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 272, in _create_session
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     self._start()
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 338, in _start
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     engine_args, maker_args)
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 360, in _setup_for_connection
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions     "No sql_connection parameter is established")
2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions CantStartEngineError: No sql_connection parameter is established

Comment 2 kamil.rogon 2016-11-18 07:34:35 UTC
The missing configuration on each controller node:
# openstack-config --set /etc/nova/nova.conf api_database connection "$(openstack-config --get /etc/nova/nova.conf database connection)_api"

One primary-controller execute:
# nova-manage api_db sync
# mysql -e "grant all privileges on nova_api.* to 'nova'@'%'"
# mysql -e "grant all privileges on nova_api.* to 'nova'@'%'"
# mysqladmin flush-privileges

Restart nova daemons on each controllers:
# systemctl | grep nova | awk '{print $1}' | while read service; do systemctl restart $service ;done

Comment 3 Sofer Athlan-Guyot 2017-02-09 17:59:33 UTC
hi,

This is a duplicate of this one https://bugzilla.redhat.com/show_bug.cgi?id=1413686.  The same solution apply and it's more involved.  See comment https://bugzilla.redhat.com/show_bug.cgi?id=1413686#c25 for the current code.

You can apply #c25 above just after having upgraded the undercloud.  This make vm creation working in-between upgrade steps form controller upgrade to convergence.

I keep this one open for you to be able to separate the client, but it's really the same thing.

We're currently waiting on the upstream review to get merged https://review.openstack.org/#/c/428093/.

Regards,

Comment 4 Sofer Athlan-Guyot 2017-03-17 10:15:48 UTC
Hi,

fixed in openstack-tripleo-heat-templates-2.0.0-45.el7ost

Closing this bug as a duplicate.  If you have any trouble to get the relevant package don't hesitate to re-open the bug.

Regards,

*** This bug has been marked as a duplicate of bug 1413686 ***


Note You need to log in before you can comment on or make changes to this bug.