Bug 1929745 - [OSP-16.2] Unable to start instances after FFU due to ScaleIO brick connector unable to parse connector_password
Summary: [OSP-16.2] Unable to start instances after FFU due to ScaleIO brick connector...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-os-brick
Version: 16.2 (Train)
Hardware: x86_64
OS: Linux
medium
urgent
Target Milestone: beta
: 16.2 (Train on RHEL 8.4)
Assignee: Alan Bishop
QA Contact: Tzach Shefi
URL:
Whiteboard:
Depends On: 1929275
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-02-17 14:46 UTC by Alan Bishop
Modified: 2023-10-25 06:15 UTC (History)
11 users (show)

Fixed In Version: python-os-brick-2.10.5-2.20210217184950.634fb4a.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of: 1929275
Environment:
Last Closed: 2021-09-15 07:12:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 775890 0 None NEW ScaleIO: More connection info backward compatibility 2021-02-17 14:59:37 UTC
Red Hat Issue Tracker OSP-2998 0 None None None 2023-10-25 06:15:28 UTC
Red Hat Product Errata RHEA-2021:3483 0 None Closed [BZ] Viewer role can't see RPM name in content view filter 2022-06-03 07:06:32 UTC

Description Alan Bishop 2021-02-17 14:46:24 UTC
+++ This bug was initially created as a clone of Bug #1929275 +++

Description of problem:

Unable to start nova instances after upgrading the controller node[1]

>> Instance stays in shutdown error due to error(nova-compute.log) while parsing connector_password by scaleio driver

~~~
Instance UUID - 3ddd9dd219cb4b7d9a456e5b0448165a

(overcloud) [stack@director-<hostname> operation]$ date ; openstack server start rhel8-nova-volume-tenantnet-9                                                                                     
Tue Feb 16 12:13:39 CET 2021


(overcloud) [stack@director-<hostname> operation]$ date ; openstack server show rhel8-nova-volume-tenantnet-9 | grep status                                                                        
Tue Feb 16 12:13:56 CET 2021
| host_status                         | UP                                                                                      |                                                            
| status                              | SHUTOFF  
~~~


>> compute is still on osp13 with nova_compute container running with osp16.1 container images(which get created after running "nova_hybrid_state" of upgrade framework procedure[2])

~~
[msufiyan@supportshell sosreport-<compute-hostname>-2021-02-16-nxnvzoj]$ cat etc/rhosp-release 
Red Hat OpenStack Platform release 13.0.14 (Queens)
[msufiyan@supportshell sosreport--<compute-hostname>-2021-02-16-nxnvzoj]$ cat sos_commands/docker/docker_ps | grep -i nova_compute
7e27d7fdc85b        satellite--<hostname>:5000/<image>-openstack16_1-osp16_1_containers-nova-compute:16.1                               "dumb-init --singl..."   7 days ago          Up 3 days (unhealthy)                       nova_compute
~~~

>> var/log/containers/nova/nova-compute.log

~~~
2021-02-16 12:13:43.370 8 INFO nova.compute.manager [req-5331b556-08f8-4581-aca7-289cdb99e10b 3ddd9dd219cb4b7d9a456e5b0448165a e9073ad763e348e68f52b02dfafd0011 - default default] [instance: 30e17b48-41e3-488d-bc59-8c486f836146] Successfully reverted task state from powering-on on failure for instance.
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server [req-5331b556-08f8-4581-aca7-289cdb99e10b 3ddd9dd219cb4b7d9a456e5b0448165a e9073ad763e348e68f52b02dfafd0011 - default default] Exception during message handling: os_brick.exception.BrickException: Error getting ScaleIO connector password from configuration file: 'failed_over'
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/scaleio.py", line 103, in _get_password_token
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     connection_properties['failed_over'])
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server KeyError: 'failed_over'
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     raise value
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 191, in decorated_function
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     raise value
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 161, in decorated_function
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1372, in decorated_function
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 219, in decorated_function
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     raise value
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 3140, in start_instance
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 3110, in _power_on
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     block_device_info)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3466, in power_on
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3313, in _hard_reboot
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1318, in destroy
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     destroy_disks)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1391, in cleanup
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     cleanup_instance_disks=cleanup_instance_disks)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1476, in _cleanup
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     instance=instance)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     raise value
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1463, in _cleanup
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self._disconnect_volume(context, connection_info, instance)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1692, in _disconnect_volume
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     vol_driver.disconnect_volume(connection_info, instance)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/volume/scaleio.py", line 59, in disconnect_volume
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self.connector.disconnect_volume(connection_info['data'], None)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/scaleio.py", line 469, in disconnect_volume
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self.get_config(connection_properties)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/scaleio.py", line 332, in get_config
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     connection_properties)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/scaleio.py", line 109, in _get_password_token
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     raise exception.BrickException(message=msg)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server os_brick.exception.BrickException: Error getting ScaleIO connector password from configuration file: 'failed_over'
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server
~~~

~~~
[root@-<compute-hostname> ~]# docker exec --user=root  nova_compute rpm -qa |grep os-brick
python3-os-brick-2.10.5-1.20200916163418.634fb4a.el8ost.noarch

()[root@compute-hostname /]# grep -in failed_over /usr/lib/python3.6/site-packages/os_brick/privileged/scaleio.py                                                                                   
79:def get_connector_password(filename, config_group, failed_over):                                                                                                                          
86:    :param failed_over: flag representing if storage is in failed over state                                                                                                              
87:    :type failed_over: bool
102:        "replicating_san_password" if failed_over else "san_password"

[msufiyan@supportshell sosreport-controller-hostname-2021-02-11]$ cat var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf | grep "volume_driver="
volume_driver=cinder.volume.drivers.dell_emc.scaleio.driver.ScaleIODriver
~~~

Version-Release number of selected component (if applicable):
OSP16.1

How reproducible:
During FFU with scaleio driver


[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/framework_for_upgrades_13_to_16.1/index#upgrading-a-split-controller-overcloud
[2] openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

Comment 5 errata-xmlrpc 2021-09-15 07:12:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:3483


Note You need to log in before you can comment on or make changes to this bug.