+++ This bug was initially created as a clone of Bug #1545324 +++

+++ This bug was initially created as a clone of Bug #1463897 +++

Description of problem:

The I/O latency of a cinder volume after live migration of an instance to which it's attached increases significantly. This stays increased till the VM is stopped and started again.[ vm is booted with cinder volume]

This is not the case when using a disk from a nova store backend [ without cinder volume] (or at least the difference isn't so significantly high after a live migration).

Ceph 2.0 is backend 

Version-Release number of selected component (if applicable):

 


How reproducible:


Steps to Reproduce:
1. create a vm with cinder volume live migrate it 
2. check using ioping 
3.

Actual results:


Expected results:


Additional info:
he I/O latency of a cinder volume after live migration of an instance to which it's attached increases significantly. This stays increased till the VM is stopped and started again.

--- Additional comment from Kashyap Chamarthy on 2017-07-29 11:21:04 EDT ---

The patch for Git master is merged.

And here's the upstream stable/newton backport, in-progress:

    https://review.openstack.org/#/c/488959/

--- Additional comment from Kashyap Chamarthy on 2017-09-18 09:37:30 EDT ---

Verification notes for this bug:

*Without* this bug fix (from openstack-nova-14.0.8-2.el7ost), when you 
migrate a Nova instance with a Cinder volume -- where both Nova
instance's disk and the Cinder volume are on Ceph -- the cache value for 
the Cinder volume (erroneously) changes from 'writeback' to 'none':

    [Check by doing `ps -ef | grep qemu`, and look for the relevant QEMU
    process associated with the Nova instance.]

    Pre-migration, QEMU command-line for the Nova instance:

        [...] -drive file=rbd:volumes/volume-[...],cache=writeback

    Post-migration, QEMU command-line for the Nova instance:

        [...] -drive file=rbd:volumes/volume-[...],cache=none

*With* the bug fix (from openstack-nova-14.0.8-2.el7ost), the cache 
value for the Cinder volume should remain 'writeback':

    Pre-migration, QEMU command-line for the Nova instance:

        [...] -drive file=rbd:volumes/volume-[...],cache=writeback

    Post-migration, QEMU command-line for the Nova instance:

        [...] -drive file=rbd:volumes/volume-[...],cache=writeback

--- Additional comment from Martin Schuppert on 2018-02-14 11:25:30 EST ---

OSP8 is also affected by this:

# rpm -q openstack-nova-compute
openstack-nova-compute-12.0.6-21.el7ost.noarch

* before migration:
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='475b69d9-9ea3-4356-ac22-762b17a875e3'/>
      </auth>
      <source protocol='rbd' name='osp8-vms/9715a493-60be-4d76-9d4c-34b37dad7366_disk'>
        <host name='192.168.122.5' port='6789'/>
        <host name='192.168.122.6' port='6789'/>
        <host name='192.168.122.7' port='6789'/>
      </source>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='475b69d9-9ea3-4356-ac22-762b17a875e3'/>
      </auth>
      <source protocol='rbd' name='osp8-volumes/volume-ce556e6c-dab1-40c2-b186-762d1f8afd4e'>
        <host name='192.168.122.5' port='6789'/>
        <host name='192.168.122.6' port='6789'/>
        <host name='192.168.122.7' port='6789'/>
      </source>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <serial>ce556e6c-dab1-40c2-b186-762d1f8afd4e</serial>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>


* after migration:

    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='475b69d9-9ea3-4356-ac22-762b17a875e3'/>
      </auth>
      <source protocol='rbd' name='osp8-vms/9715a493-60be-4d76-9d4c-34b37dad7366_disk'>
        <host name='192.168.122.5' port='6789'/>
        <host name='192.168.122.6' port='6789'/>
        <host name='192.168.122.7' port='6789'/>
      </source>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='475b69d9-9ea3-4356-ac22-762b17a875e3'/>
      </auth>
      <source protocol='rbd' name='osp8-volumes/volume-ce556e6c-dab1-40c2-b186-762d1f8afd4e'>
        <host name='192.168.122.5' port='6789'/>
        <host name='192.168.122.6' port='6789'/>
        <host name='192.168.122.7' port='6789'/>
      </source>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <serial>ce556e6c-dab1-40c2-b186-762d1f8afd4e</serial>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>

works with:
# diff -u driver.py.org driver.py
--- driver.py.org       2018-02-14 11:00:23.986251918 -0500
+++ driver.py   2018-02-14 11:12:07.310126939 -0500
@@ -1074,8 +1074,10 @@
         driver.disconnect_volume(connection_info, disk_dev)
 
     def _get_volume_config(self, connection_info, disk_info):
-        driver = self._get_volume_driver(connection_info)
-        return driver.get_config(connection_info, disk_info)
+        vol_driver = self._get_volume_driver(connection_info)
+        conf = vol_driver.get_config(connection_info, disk_info)
+        self._set_cache_mode(conf)
+        return conf
 
     def _get_volume_encryptor(self, connection_info, encryption):
         encryptor = encryptors.get_volume_encryptor(connection_info,
@@ -1119,7 +1121,6 @@
             instance, CONF.libvirt.virt_type, image_meta, bdm)
         self._connect_volume(connection_info, disk_info)
         conf = self._get_volume_config(connection_info, disk_info)
-        self._set_cache_mode(conf)
 
         try:
             state = guest.get_power_state(self._host)
@@ -3489,9 +3490,6 @@
             vol['connection_info'] = connection_info
             vol.save()
 
-        for d in devices:
-            self._set_cache_mode(d)
-
         if image_meta.properties.get('hw_scsi_model'):
             hw_scsi_model = image_meta.properties.hw_scsi_model
             scsi_controller = vconfig.LibvirtConfigGuestController()