Description of problem: Trying to run multiattach on rhos14 , a setup with all in once , 2 controllers , 2 computes. When trying to attach instance to multi attach volume , getting this traceback 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server [req-29bc734e-129f-43d6-98d1-e2ffa1bfe96b 2601ee20746e4bf987a240c5ed90e2ff 3b2fd8a7a4004408bf31606f3e65e523 - default default] Exception during message handling: MultiattachNotSupportedByVirtDriver: Volume 9392b9cb-a690-4828-bf3b-afbb0390a6eb has 'multiattach' set, which is not supported for this instance. 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb) 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server self.force_reraise() 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance) 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server self.force_reraise() 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 158, in decorated_function 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server self.force_reraise() 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5428, in reserve_block_device_name 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server volume_id=volume_id) 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server MultiattachNotSupportedByVirtDriver: Volume 9392b9cb-a690-4828-bf3b-afbb0390a6eb has 'multiattach' set, which is not supported for this instance. 2018-08-29 07:56:14.015 1 ERROR oslo_messaging.rpc.server 2018-08-29 07:56:19.065 1 DEBUG nova.compute.manager [req-9b4e31f8-8b32-4aef-8377-b5892df08893 ba740824511440768121146ee6e4c021 bb70c6ed41aa495db80606268004be3d - default default] [instance: 6c35ef08-2400-42ee-8 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
How to reproduce : (LVM backend) #1 Create multi-attach type: (overcloud) [stack@undercloud-0 ~]$ cinder type-show c963d843-5670-4ac3-a6d7-63ffa33bb0c3 +---------------------------------+--------------------------------------+ | Property | Value | +---------------------------------+--------------------------------------+ | description | None | | extra_specs | multiattach : <is> True | | id | c963d843-5670-4ac3-a6d7-63ffa33bb0c3 | | is_public | True | | name | multiattach | | os-volume-type-access:is_public | True | | qos_specs_id | None | +---------------------------------+--------------------------------------+ #2 create a volume with this type (multi-attach) (overcloud) [stack@undercloud-0 ~]$ cinder show e3279478-6434-4606-8281-23ae1bb5635d +--------------------------------+---------------------------------------+ | Property | Value | +--------------------------------+---------------------------------------+ | attached_servers | [] | | attachment_ids | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2018-08-29T08:22:14.000000 | | description | None | | encrypted | False | | id | e3279478-6434-4606-8281-23ae1bb5635d | | metadata | | | migration_status | None | | multiattach | True | | name | ttt | | os-vol-host-attr:host | hostgroup@tripleo_iscsi#tripleo_iscsi | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 0e838e1e87ca4771829590536082da4b | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | available | | updated_at | 2018-08-29T08:22:15.000000 | | user_id | 8280ea313ca64c58860fa33eb332dc08 | | volume_type | multiattach | +--------------------------------+---------------------------------------+ #3 boot an instance (overcloud) [stack@undercloud-0 ~]$ nova show 187952f2-9166-4125-98b5-00f36b235aa5 +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | compute-1.localdomain | | OS-EXT-SRV-ATTR:hostname | test | | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1.localdomain | | OS-EXT-SRV-ATTR:instance_name | instance-000002bd | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-6dui1oxv | | OS-EXT-SRV-ATTR:root_device_name | /dev/vda | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 1 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2018-08-29T08:24:49.000000 | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2018-08-29T08:24:31Z | | description | - | | flavor:disk | 1 | | flavor:ephemeral | 0 | | flavor:extra_specs | {} | | flavor:original_name | test | | flavor:ram | 64 | | flavor:swap | 0 | | flavor:vcpus | 1 | | hostId | f19bae6bad7f47b0cde8a0ae46be684f608a0ae32a89cb871fc7cb75 | | host_status | UP | | id | 187952f2-9166-4125-98b5-00f36b235aa5 | | image | cirros (a499431f-fcfe-4eea-a722-be34bb6ba6f0) | | key_name | - | | locked | False | | metadata | {} | | name | test | | os-extended-volumes:volumes_attached | [] | | private network | 100.100.100.10 | | progress | 0 | | security_groups | default | | status | ACTIVE | | tags | [] | | tenant_id | 0e838e1e87ca4771829590536082da4b | | trusted_image_certificates | - | | updated | 2018-08-29T08:27:13Z | | user_id | 8280ea313ca64c58860fa33eb332dc08 | +--------------------------------------+----------------------------------------------------------+ #4 Try to attach the instance to the volume (overcloud) [stack@undercloud-0 ~]$ nova --os-compute-api-version 2.62 volume-attach 187952f2-9166-4125-98b5-00f36b235aa5 e3279478-6434-4606-8281-23ae1bb5635d ERROR (Conflict): Volume e3279478-6434-4606-8281-23ae1bb5635d has 'multiattach' set, which is not supported for this instance. (HTTP 409) (Request-ID: req-e96f504c-a8d3-4859-bc1f-15fe816d09b8)
on compute side (versions): root@compute-1 heat-admin]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8898c6826c69 192.168.24.1:8787/rhosp14/openstack-neutron-openvswitch-agent:2018-08-23.1 "kolla_start" 22 hours ago Up 22 hours (healthy) neutron_ovs_agent e5f0109f4f48 192.168.24.1:8787/rhosp14/openstack-cron:2018-08-23.1 "kolla_start" 22 hours ago Up 22 hours logrotate_crond f39adfefbd9b 192.168.24.1:8787/rhosp14/openstack-nova-compute:2018-08-23.1 "kolla_start" 22 hours ago Up 22 hours (healthy) nova_migration_target f2b8905505cf 192.168.24.1:8787/rhosp14/openstack-ceilometer-compute:2018-08-23.1 "kolla_start" 22 hours ago Up 22 hours ceilometer_agent_compute 8ee095dd0f8d 192.168.24.1:8787/rhosp14/openstack-nova-compute:2018-08-23.1 "kolla_start" 22 hours ago Up 22 hours (healthy) nova_compute 0eaefffd328d 192.168.24.1:8787/rhosp14/openstack-iscsid:2018-08-23.1 "kolla_start" 22 hours ago Up 22 hours (healthy) iscsid 6c28e9a255a7 192.168.24.1:8787/rhosp14/openstack-nova-libvirt:2018-08-23.1 "kolla_start" 22 hours ago Up 22 hours (healthy) nova_libvirt 22b54f6e3a82 192.168.24.1:8787/rhosp14/openstack-nova-libvirt:2018-08-23.1 "kolla_start" 22 hours ago Up 22 hours nova_virtlogd Attaching /var/log/containers/nova/nova-compute.log Benny
Created attachment 1479407 [details] nova-compute.log
It appears that the following QEMU and Libvirt checks need to be updated downstream as https://bugzilla.redhat.com/show_bug.cgi?id=1378242 was backported to 3.9 in RHEL 7.5: nova/virt/libvirt/driver.py 576 def _set_multiattach_support(self): 577 # Check to see if multiattach is supported. Based on bugzilla 578 # https://bugzilla.redhat.com/show_bug.cgi?id=1378242 and related 579 # clones, the shareable flag on a disk device will only work with 580 # qemu<2.10 or libvirt>=3.10. So check those versions here and set 581 # the capability appropriately. 582 if (self._host.has_min_version(lv_ver=MIN_LIBVIRT_MULTIATTACH) or 583 not self._host.has_min_version(hv_ver=(2, 10, 0))): 584 self.capabilities['supports_multiattach'] = True 585 else: 586 LOG.debug('Volume multiattach is not supported based on current ' 587 'versions of QEMU and libvirt. QEMU must be less than ' 588 '2.10 or libvirt must be greater than or equal to 3.10.')
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045