Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1929275

Summary: [OSP-16.1] Unable to start instances after FFU due to ScaleIO brick connector unable to parse connector_password
Product: Red Hat OpenStack Reporter: MD Sufiyan <msufiyan>
Component: python-os-brickAssignee: Alan Bishop <abishop>
Status: CLOSED ERRATA QA Contact: Tzach Shefi <tshefi>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 16.1 (Train)CC: abishop, acanan, apevec, ealcaniz, gsciorti, igallagh, jschluet, jvisser, kholtz, kthakre, lhh, morazi, slinaber, spower
Target Milestone: z4Keywords: OtherQA, Triaged, ZStream
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: python-os-brick-2.10.5-1.20201114041629.el8ost Doc Type: Bug Fix
Doc Text:
Before this update, instances that were created on a RHOSP 13 environment with PowerFlex, VxFlex and ScaleIO volume attachments failed restarting after an upgrade to RHOSP 16.x. This was because the RHOSP 16.x Compute service uses a new PowerFlex driver connection property to access volume attachments, which is not present in the connection properties of volumes attached to instances running on a RHOSP 13 environment. With this update, the error is no longer thrown if this connection property is missing, and instances with PowerFlex volume attachments created on a RHOSP 13 environment continue to function correctly after upgrading to RHOSP 16.x.
 Story Points: ---
Clone Of:
: 1929745 (view as bug list) Environment:
Last Closed: 2021-03-17 15:36:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1929745    

Description MD Sufiyan 2021-02-16 15:51:43 UTC 
Description of problem:

Unable to start nova instances after upgrading the controller node[1]

>> Instance stays in shutdown error due to error(nova-compute.log) while parsing connector_password by scaleio driver

~~~
Instance UUID - 3ddd9dd219cb4b7d9a456e5b0448165a

(overcloud) [stack@director-<hostname> operation]$ date ; openstack server start rhel8-nova-volume-tenantnet-9                                                                                     
Tue Feb 16 12:13:39 CET 2021


(overcloud) [stack@director-<hostname> operation]$ date ; openstack server show rhel8-nova-volume-tenantnet-9 | grep status                                                                        
Tue Feb 16 12:13:56 CET 2021
| host_status                         | UP                                                                                      |                                                            
| status                              | SHUTOFF  
~~~


>> compute is still on osp13 with nova_compute container running with osp16.1 container images(which get created after running "nova_hybrid_state" of upgrade framework procedure[2])

~~
[msufiyan@supportshell sosreport-<compute-hostname>-2021-02-16-nxnvzoj]$ cat etc/rhosp-release 
Red Hat OpenStack Platform release 13.0.14 (Queens)
[msufiyan@supportshell sosreport--<compute-hostname>-2021-02-16-nxnvzoj]$ cat sos_commands/docker/docker_ps | grep -i nova_compute
7e27d7fdc85b        satellite--<hostname>:5000/<image>-openstack16_1-osp16_1_containers-nova-compute:16.1                               "dumb-init --singl..."   7 days ago          Up 3 days (unhealthy)                       nova_compute
~~~

>> var/log/containers/nova/nova-compute.log

~~~
2021-02-16 12:13:43.370 8 INFO nova.compute.manager [req-5331b556-08f8-4581-aca7-289cdb99e10b 3ddd9dd219cb4b7d9a456e5b0448165a e9073ad763e348e68f52b02dfafd0011 - default default] [instance: 30e17b48-41e3-488d-bc59-8c486f836146] Successfully reverted task state from powering-on on failure for instance.
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server [req-5331b556-08f8-4581-aca7-289cdb99e10b 3ddd9dd219cb4b7d9a456e5b0448165a e9073ad763e348e68f52b02dfafd0011 - default default] Exception during message handling: os_brick.exception.BrickException: Error getting ScaleIO connector password from configuration file: 'failed_over'
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/scaleio.py", line 103, in _get_password_token
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     connection_properties['failed_over'])
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server KeyError: 'failed_over'
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     raise value
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 191, in decorated_function
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     raise value
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 161, in decorated_function
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1372, in decorated_function
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 219, in decorated_function
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     raise value
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 3140, in start_instance
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 3110, in _power_on
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     block_device_info)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3466, in power_on
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3313, in _hard_reboot
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1318, in destroy
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     destroy_disks)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1391, in cleanup
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     cleanup_instance_disks=cleanup_instance_disks)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1476, in _cleanup
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     instance=instance)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     raise value
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1463, in _cleanup
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self._disconnect_volume(context, connection_info, instance)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1692, in _disconnect_volume
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     vol_driver.disconnect_volume(connection_info, instance)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/volume/scaleio.py", line 59, in disconnect_volume
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self.connector.disconnect_volume(connection_info['data'], None)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/scaleio.py", line 469, in disconnect_volume
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     self.get_config(connection_properties)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/scaleio.py", line 332, in get_config
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     connection_properties)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/scaleio.py", line 109, in _get_password_token
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server     raise exception.BrickException(message=msg)
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server os_brick.exception.BrickException: Error getting ScaleIO connector password from configuration file: 'failed_over'
2021-02-16 12:13:43.373 8 ERROR oslo_messaging.rpc.server
~~~

~~~
[root@-<compute-hostname> ~]# docker exec --user=root  nova_compute rpm -qa |grep os-brick
python3-os-brick-2.10.5-1.20200916163418.634fb4a.el8ost.noarch

()[root@compute-hostname /]# grep -in failed_over /usr/lib/python3.6/site-packages/os_brick/privileged/scaleio.py                                                                                   
79:def get_connector_password(filename, config_group, failed_over):                                                                                                                          
86:    :param failed_over: flag representing if storage is in failed over state                                                                                                              
87:    :type failed_over: bool
102:        "replicating_san_password" if failed_over else "san_password"

[msufiyan@supportshell sosreport-controller-hostname-2021-02-11]$ cat var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf | grep "volume_driver="
volume_driver=cinder.volume.drivers.dell_emc.scaleio.driver.ScaleIODriver
~~~

Version-Release number of selected component (if applicable):
OSP16.1

How reproducible:
During FFU with scaleio driver


[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/framework_for_upgrades_13_to_16.1/index#upgrading-a-split-controller-overcloud
[2] openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all
Comment 17 errata-xmlrpc 2021-03-17 15:36:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.4 director bug fix advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0817