Bug 1713669

Summary: [OSP15] 3PAR Cinder Driver:Multi attach feature backport
Product: Red Hat OpenStack Reporter: Brian Rosmaita <brian.rosmaita>
Component: openstack-cinderAssignee: Sofia Enriquez <senrique>
Status: CLOSED EOL QA Contact: Tzach Shefi <tshefi>
Severity: high Docs Contact: Tana <tberry>
Priority: high    
Version: 15.0 (Stein)CC: abishop, eharney, senrique, tshefi
Target Milestone: z3Keywords: FeatureBackport, Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-cinder-14.0.4-0.20200226170452.d906d81.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1713047 Environment:
Last Closed: 2020-09-21 10:18:23 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1725012, 1731275, 1731276    
Bug Blocks: 1713047    

Description Brian Rosmaita 2019-05-24 13:13:09 UTC 
+++ This bug was initially created as a clone of Bug #1713047 +++

We need to get the fix for this backported to RHOSP 13.  It has been merged into master (Train) upstream, but the stable maintenance team has rejected it as a feature backport to Stein, etc.

This BZ will track the backport of the fix to RHOSP 15.

----
This is the original description from Bug #1708705:
----

Description of problem:3PAR Cinder Driver:Multi attach feature backport to RHOSP13


Version-Release number of selected component (if applicable): RHOSP 13


This bug has been created for tracking purpose for the backport of 3PAR Cinder driver changes to enable Multi attach feature per a Customer ask.

This need to be delivered by June 2019.

Added https://review.opendev.org/#/c/560067/2/cinder/volume/drivers/hpe/hpe_3par_common.py change in the code.

3PAR Engineering facing Nova attach issue due to driver mismatch version (QEMU and Libvirt)

[[local|localrc]]
ENABLE_VOLUME_MULTIATTACH=True
ENABLE_UBUNTU_CLOUD_ARCHIVE=False

Added backend details in /etc/cinder/cinder.conf:
[3pariscsi_1]
hpe3par_api_url = https://192.168.1.7:8080/api/v1
hpe3par_username = user
hpe3par_password = password
san_ip = 192.168.1.7
san_login = user
san_password = password
volume_backend_name = 3pariscsi_1
hpe3par_cpg = my_cpg
hpe3par_iscsi_ips = 192.168.11.2,192.168.11.3
volume_driver = cinder.volume.drivers.hpe.hpe_3par_iscsi.HPE3PARISCSIDriver
hpe3par_iscsi_chap_enabled = True
hpe3par_debug = True
image_volume_cache_enabled = True

Added policy in /etc/cinder/policy.json:
'volume:multiattach': 'rule:admin_or_owner'

Added https://review.opendev.org/#/c/560067/2/cinder/volume/drivers/hpe/hpe_3par_common.py change in the code.

But I am getting below error in the nova log while attaching nova instance to my volume:
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [None req-2cda6e90-fd45-4bfe-960a-7fca9ba4abab demo admin] [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] Instance failed block device setup: MultiattachNotSupportedByVirtDriver: Volume dc25f09a-6ae1-4b06-a814-73a8afaba62f has 'multiattach' set, which is not supported for this instance.
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] Traceback (most recent call last):
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]   File "/opt/stack/nova/nova/compute/manager.py", line 1615, in _prep_block_device
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]     wait_func=self._await_block_device_map_created)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]   File "/opt/stack/nova/nova/virt/block_device.py", line 840, in attach_block_devices
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]     _log_and_attach(device)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]   File "/opt/stack/nova/nova/virt/block_device.py", line 837, in _log_and_attach
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]     bdm.attach(*attach_args, **attach_kwargs)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]   File "/opt/stack/nova/nova/virt/block_device.py", line 46, in wrapped
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]     ret_val = method(obj, context, *args, **kwargs)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]   File "/opt/stack/nova/nova/virt/block_device.py", line 620, in attach
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]     virt_driver, do_driver_attach)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]     return f(*args, **kwargs)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]   File "/opt/stack/nova/nova/virt/block_device.py", line 617, in _do_locked_attach
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]     self._do_attach(*args, **_kwargs)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]   File "/opt/stack/nova/nova/virt/block_device.py", line 602, in _do_attach
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]     do_driver_attach)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]   File "/opt/stack/nova/nova/virt/block_device.py", line 509, in _volume_attach
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]     volume_id=volume_id)
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] MultiattachNotSupportedByVirtDriver: Volume dc25f09a-6ae1-4b06-a814-73a8afaba62f has 'multiattach' set, which is not supported for this instance.
Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa]


Apr 29 05:41:20 CSSOSBE04-B09 nova-compute[20455]: DEBUG nova.virt.libvirt.driver [-] Volume multiattach is not supported based on current versions of QEMU and libvirt. QEMU must be less than 2.10 or libvirt must be greater than or equal to 3.10. {{(pid=20455) _set_multiattach_support /opt/stack/nova/nova/virt/libvirt/driver.py:619}}


stack@CSSOSBE04-B09:/tmp$ virsh --version
3.6.0
stack@CSSOSBE04-B09:/tmp$ kvm --version
QEMU emulator version 2.10.1(Debian 1:2.10+dfsg-0ubuntu3.8~cloud1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers


openstack volume show -c  multiattach -c status sneha1
+-------------+-----------+
| Field       | Value     |
+-------------+-----------+
| multiattach | True      |
| status      | available |
+-------------+-----------+


cinder extra-specs-list
+--------------------------------------+-------------+--------------------------------------------------------------------+
| ID                                   | Name        | extra_specs                                                        |
+--------------------------------------+-------------+--------------------------------------------------------------------+
| bd077fde-51c3-4581-80d5-5855e8ab2f6b | 3pariscsi_1 | {'volume_backend_name': '3pariscsi_1', 'multiattach': '<is> True'}|
+--------------------------------------+-------------+--------------------------------------------------------------------+


echo $OS_COMPUTE_API_VERSION
2.60

pip list | grep python-novaclient
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
python-novaclient            13.0.0

Need help from Redhat engineering to close the testing.
Comment 5 Shelley Dunne 2019-09-19 18:29:43 UTC 
Re-setting Target Milestone z1 to --- to begin the 15z1 Maintenance Release.
Comment 7 Luigi Toscano 2020-09-21 10:18:23 UTC 
OSP15 is EOL, and this feature has been already implemented in the newer releases and backported to older ones, so this bug is not needed anymore.