Description of problem: During one of the upgrade steps I have included additional template called "major-upgrade-pacemaker.yaml". After this step I am not able to boot new instance on overcloud. It seems like nova is missing connection parameter under api_database section for new database, database is not created/populated and there are no grants for nova user in database. Version-Release number of selected component (if applicable): [root@osp9-director ~]# rpm -qa | grep heat openstack-heat-templates-0-0.3.96a0b0bgit.el7ost.noarch openstack-tripleo-heat-templates-kilo-0.8.14-16.el7ost.noarch openstack-heat-engine-6.0.0-11.el7ost.noarch openstack-tripleo-heat-templates-liberty-2.0.0-34.el7ost.noarch openstack-heat-api-cfn-6.0.0-11.el7ost.noarch openstack-heat-api-cloudwatch-6.0.0-11.el7ost.noarch openstack-heat-common-6.0.0-11.el7ost.noarch openstack-tripleo-heat-templates-2.0.0-34.el7ost.noarch python-heatclient-1.2.0-1.el7ost.noarch openstack-heat-api-6.0.0-11.el7ost.noarch python-heat-tests-6.0.0-11.el7ost.noarch [root@overcloud-controller-0 ~]# rpm -qa | grep nova openstack-nova-compute-13.1.1-7.el7ost.noarch openstack-nova-console-13.1.1-7.el7ost.noarch python-nova-13.1.1-7.el7ost.noarch openstack-nova-novncproxy-13.1.1-7.el7ost.noarch openstack-nova-common-13.1.1-7.el7ost.noarch openstack-nova-conductor-13.1.1-7.el7ost.noarch python-novaclient-3.3.2-1.el7ost.noarch openstack-nova-api-13.1.1-7.el7ost.noarch openstack-nova-scheduler-13.1.1-7.el7ost.noarch openstack-nova-cert-13.1.1-7.el7ost.noarch How reproducible: always Steps to Reproduce: 1. execute major version upgrade from director including "major-upgrade-pacemaker.yaml" step. 2. Try to boot VM from director using [stack@osp9-director ~]$ source overcloudrc [stack@osp8-director ~]$ nova --debug boot --flavor 2 --image rhel7 --nic net-id=cc5c67be-1ce0-4606-9d63-5f31fe361ffa novaapi_test Actual results: [...] DEBUG (shell:1082) Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_db.exception.CantStartEngineError'> (HTTP 500) (Request-ID: req-767a0ecb-f794-4f99-abbb-b4c87d52f903) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1080, in main OpenStackComputeShell().main(argv) File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1007, in main args.func(self.cs, args) File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 589, in do_boot server = cs.servers.create(*boot_args, **boot_kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1233, in create **boot_kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 667, in _boot return_raw=return_raw, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 345, in _create resp, body = self.api.client.post(url, body=body) File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 179, in post return self.request(url, 'POST', **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 94, in request raise exceptions.from_response(resp, body, url, method) ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_db.exception.CantStartEngineError'> (HTTP 500) (Request-ID: req-767a0ecb-f794-4f99-abbb-b4c87d52f903) ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_db.exception.CantStartEngineError'> (HTTP 500) (Request-ID: req-767a0ecb-f794-4f99-abbb-b4c87d52f903) Expected results: VM started (available as building/active in "nova list"). Additional info: Traceback from nova-api.log 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions [req-d5f68d5d-18b1-4d3d-a1f2-e0f7b73cde86 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] Unexpecte d exception in API method 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 629, in create 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions **create_kwargs) 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs) 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1563, in create 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota) 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1176, in _create_instance 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions instance_group, check_server_group_quota, filter_properties) 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1034, in _provision_instances 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions quotas.rollback() 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions self.force_reraise() 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb) 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 976, in _provision_instances 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions req_spec.create() 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 223, in wrapper 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions return fn(self, *args, **kwargs) 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/request_spec.py", line 432, in create 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions db_spec = self._create_in_db(self._context, updates) 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 708, in wrapper 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions with self._transaction_scope(context): 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions return self.gen.next() 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 759, in _transaction_scope 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions allow_async=self._allow_async) as resource: 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions return self.gen.next() 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 491, in _session 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions bind=self.connection, mode=self.mode) 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 272, in _create_session 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions self._start() 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 338, in _start 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions engine_args, maker_args) 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 360, in _setup_for_connection 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions "No sql_connection parameter is established") 2016-11-08 16:30:35.562 16062 ERROR nova.api.openstack.extensions CantStartEngineError: No sql_connection parameter is established
Upstream information: https://github.com/openstack/nova/blob/df0fca62cf5324f5b6eae0fed1f88c6c9ed61c71/releasenotes/notes/request-spec-api-db-b9cc6e0624d563c5.yaml https://wiki.openstack.org/wiki/ReleaseNotes/Kilo#OpenStack_Compute_.28Nova.29
The missing configuration on each controller node: # openstack-config --set /etc/nova/nova.conf api_database connection "$(openstack-config --get /etc/nova/nova.conf database connection)_api" One primary-controller execute: # nova-manage api_db sync # mysql -e "grant all privileges on nova_api.* to 'nova'@'%'" # mysql -e "grant all privileges on nova_api.* to 'nova'@'%'" # mysqladmin flush-privileges Restart nova daemons on each controllers: # systemctl | grep nova | awk '{print $1}' | while read service; do systemctl restart $service ;done
hi, This is a duplicate of this one https://bugzilla.redhat.com/show_bug.cgi?id=1413686. The same solution apply and it's more involved. See comment https://bugzilla.redhat.com/show_bug.cgi?id=1413686#c25 for the current code. You can apply #c25 above just after having upgraded the undercloud. This make vm creation working in-between upgrade steps form controller upgrade to convergence. I keep this one open for you to be able to separate the client, but it's really the same thing. We're currently waiting on the upstream review to get merged https://review.openstack.org/#/c/428093/. Regards,
Hi, fixed in openstack-tripleo-heat-templates-2.0.0-45.el7ost Closing this bug as a duplicate. If you have any trouble to get the relevant package don't hesitate to re-open the bug. Regards, *** This bug has been marked as a duplicate of bug 1413686 ***