Bug 1495158

Summary: Failed to attach a second volume with virtio-scsi
Product: Red Hat OpenStack Reporter: Robin Cernin <rcernin>
Component: openstack-novaAssignee: Sahid Ferdjaoui <sferdjao>
Status: CLOSED ERRATA QA Contact: Joe H. Rahme <jhakimra>
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: awaugama, berrange, dasmith, eglynn, kchamart, lruzicka, lyarwood, mbooth, sbauza, sferdjao, sgordon, srevivo, vromanso
Target Milestone: z6Keywords: Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-nova-14.0.8-4.el7ost Doc Type: Bug Fix
Doc Text:
Cause: Attached second SCSI device Consequence: An error happened which makes the process t o fail because Nova re-used the same device address Fix: A new process in nova was implemented to correctly generate device address Result: it's now possible to attach more than one SCSI device.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-15 13:48:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Robin Cernin 2017-09-25 11:00:33 UTC
Description of problem:

We are seeing the same problem as described in https://bugs.launchpad.net/nova/+bug/1699228 however with already attaching the second volume using the virtio-scsi.

2017-09-22 11:50:42.951 640086 ERROR nova.compute.manager [instance: 79bf1f1f-2bb7-4955-b827-9374d2703c7e] 
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server [req-fc6bbe1e-f0e1-489b-a327-100b16344ed3 5c8cf456b70a4de5bf0c890f68be90f3 d2ec148db73c4bfc8181e96b0d05b714 - - -] Exception during message handling
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4675, in attach_volume
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     do_attach_volume(context, instance, driver_bdm)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4673, in do_attach_volume
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     bdm.destroy()
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4670, in do_attach_volume
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     return self._attach_volume(context, instance, driver_bdm)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4693, in _attach_volume
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     self.volume_api.unreserve_volume(context, bdm.volume_id)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4685, in _attach_volume
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     do_check_attach=False, do_driver_attach=True)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 49, in wrapped
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     ret_val = method(obj, context, *args, **kwargs)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 283, in attach
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     connector)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 274, in attach
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     device_type=self['device_type'], encryption=encryption)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1166, in attach_volume
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     self._disconnect_volume(connection_info, disk_dev)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1155, in attach_volume
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     guest.attach_device(conf, persistent=True, live=live)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 294, in attach_device
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-09-22 11:50:43.198 640086 ERROR oslo_messaging.rpc.server libvirtError: internal error: unable to execute QEMU command '__com.redhat_drive_add': Duplicate ID 'drive-scsi0-0-0-0' for drive


We believe this might be fixed by applying the chain up to https://review.openstack.org/#/c/459741/. This is not yet backported to Newton. Please could you backport the patches to OSP10?

Version-Release number of selected component (if applicable):

# rpm -qa |grep nova
openstack-nova-console-14.0.2-7.el7ost.noarch
python-novaclient-6.0.0-1.el7ost.noarch
openstack-nova-cert-14.0.2-7.el7ost.noarch
python-nova-14.0.2-7.el7ost.noarch
openstack-nova-api-14.0.2-7.el7ost.noarch
openstack-nova-scheduler-14.0.2-7.el7ost.noarch
openstack-nova-conductor-14.0.2-7.el7ost.noarch
openstack-nova-compute-14.0.2-7.el7ost.noarch
openstack-nova-novncproxy-14.0.2-7.el7ost.noarch
openstack-nova-common-14.0.2-7.el7ost.noarch
puppet-nova-9.4.0-2.el7ost.noarch

How reproducible:

Create VM with image using virtio-scsi scsi model and scsi bus:

1. openstack image create --os-image-api-version 1 --public --file CentOS-7-x86_64 --disk-format qcow2 --container-format bare --property os_type=linux --property os_distro=centos-7.2 --property hw_scsi_model=virtio-scsi --property hw_disk_bus=scsi --min-disk 8 CentOS-7-x86_64
2. Attach additional volume to the VM

Actual results:

This fails with the above libvirtError.

Expected results:

Being able to attach additional volume.

Additional info:

This could be most likely fixed by applying chain up to https://review.openstack.org/#/c/459741

Comment 2 Sahid Ferdjaoui 2017-09-25 11:13:11 UTC
It's a valid issue. The chain you have mentioned should fix the issue. I'm going to see whether we could backport it. If not we would have to probably provide a downstream-only fix.

Comment 3 Robin Cernin 2017-09-25 11:14:09 UTC
Other Bugzilla bug might be relevant as it fixes the issue in later release https://bugzilla.redhat.com/show_bug.cgi?id=1269577

Comment 8 Joe H. Rahme 2017-11-02 20:54:50 UTC
Verification steps:

1. Create the image and boot a VM

[stack@undercloud-0 testingdir]$ openstack image create --os-image-api-version 1 --public --file rhel-guest-image-7.4-latest.x86_64.qcow2 --disk-format qcow2 --container-format bare --property os_type=linux --property os_distro=centos-7.4 --property hw_scsi_model=virtio-scsi --property hw_disk_bus=scsi --min-disk 8 RHEL-7-x86_64
+------------------+------------------------------------------------------------------------------------------+
| Field            | Value                                                                                    |
+------------------+------------------------------------------------------------------------------------------+
| checksum         | 4c762b186f05049da8871f9ee9f47666                                                         |
| container_format | bare                                                                                     |
| created_at       | 2017-11-02T20:34:07.000000                                                               |
| deleted          | False                                                                                    |
| deleted_at       | None                                                                                     |
| disk_format      | qcow2                                                                                    |
| id               | 1ff3cdb0-ac6f-4af3-a104-638f44701ee9                                                     |
| is_public        | True                                                                                     |
| min_disk         | 8                                                                                        |
| min_ram          | 0                                                                                        |
| name             | RHEL-7-x86_64                                                                            |
| owner            | 9430d96b274046f7a0b2905343c4ebd0                                                         |
| properties       | hw_disk_bus='scsi', hw_scsi_model='virtio-scsi', os_distro='centos-7.4', os_type='linux' |
| protected        | False                                                                                    |
| size             | 534011392                                                                                |
| status           | active                                                                                   |
| updated_at       | 2017-11-02T20:34:13.000000                                                               |
| virtual_size     | None                                                                                     |
+------------------+------------------------------------------------------------------------------------------+
[stack@undercloud-0 testingdir]$ nova boot --poll --image 1ff3cdb0-ac6f-4af3-a104-638f44701ee9 --flavor m1.micro --nic net-id=$SID vm1
+--------------------------------------+------------------------------------------------------+
| Property                             | Value                                                |
+--------------------------------------+------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                               |
| OS-EXT-AZ:availability_zone          |                                                      |
| OS-EXT-SRV-ATTR:host                 | -                                                    |
| OS-EXT-SRV-ATTR:hostname             | vm1                                                  |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                    |
| OS-EXT-SRV-ATTR:instance_name        |                                                      |
| OS-EXT-SRV-ATTR:kernel_id            |                                                      |
| OS-EXT-SRV-ATTR:launch_index         | 0                                                    |
| OS-EXT-SRV-ATTR:ramdisk_id           |                                                      |
| OS-EXT-SRV-ATTR:reservation_id       | r-jah0pt0i                                           |
| OS-EXT-SRV-ATTR:root_device_name     | -                                                    |
| OS-EXT-SRV-ATTR:user_data            | -                                                    |
| OS-EXT-STS:power_state               | 0                                                    |
| OS-EXT-STS:task_state                | scheduling                                           |
| OS-EXT-STS:vm_state                  | building                                             |
| OS-SRV-USG:launched_at               | -                                                    |
| OS-SRV-USG:terminated_at             | -                                                    |
| accessIPv4                           |                                                      |
| accessIPv6                           |                                                      |
| adminPass                            | 8u52Y3Z2Tx33                                         |
| config_drive                         |                                                      |
| created                              | 2017-11-02T20:40:04Z                                 |
| description                          | -                                                    |
| flavor                               | m1.micro (02e0eee5-4a65-4bb9-9b09-40410a865a9e)      |
| hostId                               |                                                      |
| host_status                          |                                                      |
| id                                   | 6030f79b-177a-4753-9cb8-00e96539eeb5                 |
| image                                | RHEL-7-x86_64 (1ff3cdb0-ac6f-4af3-a104-638f44701ee9) |
| key_name                             | -                                                    |
| locked                               | False                                                |
| metadata                             | {}                                                   |
| name                                 | vm1                                                  |
| os-extended-volumes:volumes_attached | []                                                   |
| progress                             | 0                                                    |
| security_groups                      | default                                              |
| status                               | BUILD                                                |
| tags                                 | []                                                   |
| tenant_id                            | 9430d96b274046f7a0b2905343c4ebd0                     |
| updated                              | 2017-11-02T20:40:04Z                                 |
| user_id                              | d1cfcf10d34045e1ae6bcd950fec4ab7                     |
+--------------------------------------+------------------------------------------------------+

Server building... 100% complete
Finished

2. Create 2 volumes and attach them

[stack@undercloud-0 testingdir]$ cinder create --name vol1 1
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2017-11-02T20:44:07.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 0ab99417-c56d-4390-818f-e537605e969f |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | vol1                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 9430d96b274046f7a0b2905343c4ebd0     |
| replication_status             | disabled                             |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | d1cfcf10d34045e1ae6bcd950fec4ab7     |
| volume_type                    | None                                 |
+--------------------------------+--------------------------------------+
[stack@undercloud-0 testingdir]$ cinder create --name vol2 1
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2017-11-02T20:44:14.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | ba389b8a-4251-4792-b21c-834dee44f8fa |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | vol2                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 9430d96b274046f7a0b2905343c4ebd0     |
| replication_status             | disabled                             |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | d1cfcf10d34045e1ae6bcd950fec4ab7     |
| volume_type                    | None                                 |
+--------------------------------+--------------------------------------+
[stack@undercloud-0 testingdir]$ nova volume-attach vm1 0ab99417-c56d-4390-818f-e537605e969f /dev/vdf
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/sdb                             |
| id       | 0ab99417-c56d-4390-818f-e537605e969f |
| serverId | 6030f79b-177a-4753-9cb8-00e96539eeb5 |
| volumeId | 0ab99417-c56d-4390-818f-e537605e969f |
+----------+--------------------------------------+
[stack@undercloud-0 testingdir]$ nova volume-attach vm1 ba389b8a-4251-4792-b21c-834dee44f8fa  /dev/vdg
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/sdc                             |
| id       | ba389b8a-4251-4792-b21c-834dee44f8fa |
| serverId | 6030f79b-177a-4753-9cb8-00e96539eeb5 |
| volumeId | ba389b8a-4251-4792-b21c-834dee44f8fa |
+----------+--------------------------------------+

Comment 11 errata-xmlrpc 2017-11-15 13:48:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3229