+++ This bug was initially created as a clone of Bug #1929275 +++ Description of problem: Unable to start nova instances after upgrading the controller node[1] >> Instance stays in shutdown error due to error(nova-compute.log) while parsing connector_password by scaleio driver ~~~ Instance UUID - 3ddd9dd219cb4b7d9a456e5b0448165a (overcloud) [stack@director-<hostname> operation]$ date ; openstack server start rhel8-nova-volume-tenantnet-9 Tue Feb 16 12:13:39 CET 2021 (overcloud) [stack@director-<hostname> operation]$ date ; openstack server show rhel8-nova-volume-tenantnet-9 | grep status Tue Feb 16 12:13:56 CET 2021 | host_status | UP | | status | SHUTOFF ~~~ >> compute is still on osp13 with nova_compute container running with osp16.1 container images(which get created after running "nova_hybrid_state" of upgrade framework procedure[2]) ~~ [msufiyan@supportshell sosreport-<compute-hostname>-2021-02-16-nxnvzoj]$ cat etc/rhosp-release Red Hat OpenStack Platform release 13.0.14 (Queens) [msufiyan@supportshell sosreport--<compute-hostname>-2021-02-16-nxnvzoj]$ cat sos_commands/docker/docker_ps | grep -i nova_compute 7e27d7fdc85b satellite--<hostname>:5000/<image>-openstack16_1-osp16_1_containers-nova-compute:16.1 "dumb-init --singl..." 7 days ago Up 3 days (unhealthy) nova_compute ~~~ >> var/log/containers/nova/nova-compute.log ~~~ 2021-02-16 12:13:43.370 8 INFO nova.compute.manager [req-5331b556-08f8-4581-aca7-289cdb99e10b 3ddd9dd219cb4b7d9a456e5b0448165a e9073ad763e348e68f52b02dfafd0011 - default default] [instance: 30e17b48-41e3-488d-bc59-8c486f836146] Successfully reverted task state from powering-on on failure for instance. 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server [req-5331b556-08f8-4581-aca7-289cdb99e10b 3ddd9dd219cb4b7d9a456e5b0448165a e9073ad763e348e68f52b02dfafd0011 - default default] Exception during message handling: os_brick.exception.BrickException: Error getting ScaleIO connector password from configuration file: 'failed_over' 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/scaleio.py", line 103, in _get_password_token 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server connection_properties['failed_over']) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server KeyError: 'failed_over' 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server raise value 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 191, in decorated_function 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server raise value 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 161, in decorated_function 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1372, in decorated_function 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 219, in decorated_function 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server raise value 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 3140, in start_instance 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server self._power_on(context, instance) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 3110, in _power_on 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server block_device_info) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3466, in power_on 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server self._hard_reboot(context, instance, network_info, block_device_info) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3313, in _hard_reboot 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server block_device_info=block_device_info) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1318, in destroy 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server destroy_disks) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1391, in cleanup 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server cleanup_instance_disks=cleanup_instance_disks) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1476, in _cleanup 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server instance=instance) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server raise value 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1463, in _cleanup 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server self._disconnect_volume(context, connection_info, instance) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1692, in _disconnect_volume 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server vol_driver.disconnect_volume(connection_info, instance) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/volume/scaleio.py", line 59, in disconnect_volume 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server self.connector.disconnect_volume(connection_info['data'], None) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/scaleio.py", line 469, in disconnect_volume 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server self.get_config(connection_properties) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/scaleio.py", line 332, in get_config 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server connection_properties) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/scaleio.py", line 109, in _get_password_token 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server raise exception.BrickException(message=msg) 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server os_brick.exception.BrickException: Error getting ScaleIO connector password from configuration file: 'failed_over' 2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server ~~~ ~~~ [root@-<compute-hostname> ~]# docker exec --user=root nova_compute rpm -qa |grep os-brick python3-os-brick-2.10.5-1.20200916163418.634fb4a.el8ost.noarch ()[root@compute-hostname /]# grep -in failed_over /usr/lib/python3.6/site-packages/os_brick/privileged/scaleio.py 79:def get_connector_password(filename, config_group, failed_over): 86: :param failed_over: flag representing if storage is in failed over state 87: :type failed_over: bool 102: "replicating_san_password" if failed_over else "san_password" [msufiyan@supportshell sosreport-controller-hostname-2021-02-11]$ cat var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf | grep "volume_driver=" volume_driver=cinder.volume.drivers.dell_emc.scaleio.driver.ScaleIODriver ~~~ Version-Release number of selected component (if applicable): OSP16.1 How reproducible: During FFU with scaleio driver [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/framework_for_upgrades_13_to_16.1/index#upgrading-a-split-controller-overcloud [2] openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:3483