+++ This bug was initially created as a clone of Bug #1463897 +++ Description of problem: The I/O latency of a cinder volume after live migration of an instance to which it's attached increases significantly. This stays increased till the VM is stopped and started again.[ vm is booted with cinder volume] This is not the case when using a disk from a nova store backend [ without cinder volume] (or at least the difference isn't so significantly high after a live migration). Ceph 2.0 is backend Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. create a vm with cinder volume live migrate it 2. check using ioping 3. Actual results: Expected results: Additional info: he I/O latency of a cinder volume after live migration of an instance to which it's attached increases significantly. This stays increased till the VM is stopped and started again. --- Additional comment from Kashyap Chamarthy on 2017-07-29 11:21:04 EDT --- The patch for Git master is merged. And here's the upstream stable/newton backport, in-progress: https://review.openstack.org/#/c/488959/ --- Additional comment from Kashyap Chamarthy on 2017-09-18 09:37:30 EDT --- Verification notes for this bug: *Without* this bug fix (from openstack-nova-14.0.8-2.el7ost), when you migrate a Nova instance with a Cinder volume -- where both Nova instance's disk and the Cinder volume are on Ceph -- the cache value for the Cinder volume (erroneously) changes from 'writeback' to 'none': [Check by doing `ps -ef | grep qemu`, and look for the relevant QEMU process associated with the Nova instance.] Pre-migration, QEMU command-line for the Nova instance: [...] -drive file=rbd:volumes/volume-[...],cache=writeback Post-migration, QEMU command-line for the Nova instance: [...] -drive file=rbd:volumes/volume-[...],cache=none *With* the bug fix (from openstack-nova-14.0.8-2.el7ost), the cache value for the Cinder volume should remain 'writeback': Pre-migration, QEMU command-line for the Nova instance: [...] -drive file=rbd:volumes/volume-[...],cache=writeback Post-migration, QEMU command-line for the Nova instance: [...] -drive file=rbd:volumes/volume-[...],cache=writeback
OSP8 is also affected by this: # rpm -q openstack-nova-compute openstack-nova-compute-12.0.6-21.el7ost.noarch * before migration: <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writeback' discard='unmap'/> <auth username='cinder'> <secret type='ceph' uuid='475b69d9-9ea3-4356-ac22-762b17a875e3'/> </auth> <source protocol='rbd' name='osp8-vms/9715a493-60be-4d76-9d4c-34b37dad7366_disk'> <host name='192.168.122.5' port='6789'/> <host name='192.168.122.6' port='6789'/> <host name='192.168.122.7' port='6789'/> </source> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <auth username='cinder'> <secret type='ceph' uuid='475b69d9-9ea3-4356-ac22-762b17a875e3'/> </auth> <source protocol='rbd' name='osp8-volumes/volume-ce556e6c-dab1-40c2-b186-762d1f8afd4e'> <host name='192.168.122.5' port='6789'/> <host name='192.168.122.6' port='6789'/> <host name='192.168.122.7' port='6789'/> </source> <backingStore/> <target dev='vdb' bus='virtio'/> <serial>ce556e6c-dab1-40c2-b186-762d1f8afd4e</serial> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> * after migration: <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writeback' discard='unmap'/> <auth username='cinder'> <secret type='ceph' uuid='475b69d9-9ea3-4356-ac22-762b17a875e3'/> </auth> <source protocol='rbd' name='osp8-vms/9715a493-60be-4d76-9d4c-34b37dad7366_disk'> <host name='192.168.122.5' port='6789'/> <host name='192.168.122.6' port='6789'/> <host name='192.168.122.7' port='6789'/> </source> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='none'/> <auth username='cinder'> <secret type='ceph' uuid='475b69d9-9ea3-4356-ac22-762b17a875e3'/> </auth> <source protocol='rbd' name='osp8-volumes/volume-ce556e6c-dab1-40c2-b186-762d1f8afd4e'> <host name='192.168.122.5' port='6789'/> <host name='192.168.122.6' port='6789'/> <host name='192.168.122.7' port='6789'/> </source> <backingStore/> <target dev='vdb' bus='virtio'/> <serial>ce556e6c-dab1-40c2-b186-762d1f8afd4e</serial> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> works with: # diff -u driver.py.org driver.py --- driver.py.org 2018-02-14 11:00:23.986251918 -0500 +++ driver.py 2018-02-14 11:12:07.310126939 -0500 @@ -1074,8 +1074,10 @@ driver.disconnect_volume(connection_info, disk_dev) def _get_volume_config(self, connection_info, disk_info): - driver = self._get_volume_driver(connection_info) - return driver.get_config(connection_info, disk_info) + vol_driver = self._get_volume_driver(connection_info) + conf = vol_driver.get_config(connection_info, disk_info) + self._set_cache_mode(conf) + return conf def _get_volume_encryptor(self, connection_info, encryption): encryptor = encryptors.get_volume_encryptor(connection_info, @@ -1119,7 +1121,6 @@ instance, CONF.libvirt.virt_type, image_meta, bdm) self._connect_volume(connection_info, disk_info) conf = self._get_volume_config(connection_info, disk_info) - self._set_cache_mode(conf) try: state = guest.get_power_state(self._host) @@ -3489,9 +3490,6 @@ vol['connection_info'] = connection_info vol.save() - for d in devices: - self._set_cache_mode(d) - if image_meta.properties.get('hw_scsi_model'): hw_scsi_model = image_meta.properties.hw_scsi_model scsi_controller = vconfig.LibvirtConfigGuestController()
According to our records, this should be resolved by openstack-nova-12.0.6-28.el7ost. This build is available now.