Bug 1176673 - [Rhel7.1] After live storage migration on block storage vdsm extends migrated drive using all free space in the vg
Summary: [Rhel7.1] After live storage migration on block storage vdsm extends migrated...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.5.0
Hardware: Unspecified
OS: Linux
urgent
urgent
Target Milestone: ovirt-3.6.0-rc
: 3.6.0
Assignee: Nir Soffer
QA Contact: Ori Gofen
URL:
Whiteboard:
Depends On: 1091094 1174791 1196066 1196067
Blocks: 1035038 1082754 1196049 1197615 1198128
TreeView+ depends on / blocked
 
Reported: 2014-12-22 19:04 UTC by Ori Gofen
Modified: 2016-05-26 01:52 UTC (History)
24 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1174791
: 1195461 1196049 1197615 1198128 (view as bug list)
Environment:
Last Closed: 2016-03-09 19:27:55 UTC
oVirt Team: Storage
Target Upstream Version:


Attachments (Terms of Use)
vdsm debug log on fedora 21 (1.48 MB, text/x-vhdl)
2015-02-21 13:17 UTC, Nir Soffer
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0362 normal SHIPPED_LIVE vdsm 3.6.0 bug fix and enhancement update 2016-03-09 23:49:32 UTC
oVirt gerrit 37726 None None None Never
oVirt gerrit 38018 master ABANDONED virt: Trace libvirt domain xml Never
oVirt gerrit 38086 None None None Never
oVirt gerrit 38087 None None None Never
oVirt gerrit 38088 ovirt-3.5 ABANDONED virt: Fix limit when calculating next volume size Never
oVirt gerrit 38142 master MERGED vm: Fix LSM when using libvirt >= 1.2.8 Never
oVirt gerrit 38201 ovirt-3.5 MERGED vm: Fix LSM when using libvirt >= 1.2.8 Never
oVirt gerrit 39551 master MERGED spec: Update libvirt-daemon and libvirt-python requirements Never
oVirt gerrit 39670 ovirt-3.5 MERGED spec: Simplify libvirt dependencies Never
oVirt gerrit 39671 ovirt-3.5 MERGED spec: Update libvirt-daemon and libvirt-python requirements Never

Comment 1 Allon Mureinik 2014-12-23 08:10:11 UTC
Ori, iiuc, the version information above (except for VDSM) pertains to a 7.0 host. Can you please include the libvirt version you tested this on?

Comment 2 Ori Gofen 2014-12-23 09:23:52 UTC
libvirt-daemon-driver-network-1.2.8-10.el7.x86_64
libvirt-daemon-driver-nodedev-1.2.8-10.el7.x86_64
libvirt-lock-sanlock-1.2.8-10.el7.x86_64
libvirt-client-1.2.8-10.el7.x86_64
libvirt-daemon-driver-qemu-1.2.8-10.el7.x86_64
libvirt-daemon-driver-secret-1.2.8-10.el7.x86_64
libvirt-daemon-config-nwfilter-1.2.8-10.el7.x86_64
libvirt-python-1.2.8-6.el7.x86_64
libvirt-daemon-1.2.8-10.el7.x86_64
libvirt-daemon-driver-interface-1.2.8-10.el7.x86_64
libvirt-daemon-kvm-1.2.8-10.el7.x86_64
libvirt-daemon-driver-storage-1.2.8-10.el7.x86_64
libvirt-daemon-driver-nwfilter-1.2.8-10.el7.x86_64

Comment 3 Nir Soffer 2014-12-23 10:18:39 UTC
(In reply to Allon Mureinik from comment #1)
> Ori, iiuc, the version information above (except for VDSM) pertains to a 7.0

libvirt version on rhel7.0 host:
# rpm -q libvirt-daemon
libvirt-daemon-1.1.1-29.el7_0.4.x86_64

Comment 4 Daniel Erez 2015-02-15 10:27:20 UTC
*** Bug 1186348 has been marked as a duplicate of this bug. ***

Comment 5 Nir Soffer 2015-02-17 15:07:57 UTC
Patch 37726 does not fix the root cause of this issue, but at least it limits the damage. Instead of growing until the vg is full, the device will grow to
the virtual size, making the device practically pre-allocated.

Comment 6 Ori Gofen 2015-02-17 16:00:52 UTC
The description of this bug contains some mistakes I'll briefly sum things up here in order to avoid any miss understandings.

This bug deals with live migrating cow sparse disk while doing lot of io (like installing OS and lsm) the result is that the live snapshot volume on the targeted domain, loops through lv extension which use up all free space on the targeted domain.
the behavior on the source domain seems correct, the lv's are successfully removed

Comment 7 Eric Blake 2015-02-20 23:05:06 UTC
I'm trying to see if this is a libvirt problem, by isolating it down to a smaller testcase.  Is the problem here that you are doing some action that creates a snapshot, and libvirt then treats the <disk> as a file instead of a block device?  Libvirt reports allocation differently for files (based on file size) than for block devices (based on querying qemu for highest sector written), so anything that happens that accidentally converts from a block device to a file could then cause vdsm to see a much larger allocation number, and enter a death spiral of trying to enlarge the device to account for the larger allocation.  To know for sure, I'd need to know what the domain XML looks like before and after the point where it goes into the allocation growth spiral, and it would also be nice to have a trace of what libvirt commands are being used (especially if there is a tight loop of querying disk statistics to learn the allocation number).

Comment 8 Nir Soffer 2015-02-21 13:17:03 UTC
Created attachment 994165 [details]
vdsm debug log on fedora 21

This log show libvirt domain xml during live storage migration flow.

In this flow:
1. We create a snapshot on block storage
2. We use blockRebase to mirror the sanpshot to a disk on another block storage
3. Finally we use blockAbort to pivot to the new disk

To look for domain xml, search _logDomainXML

Comment 9 Nir Soffer 2015-02-21 13:37:31 UTC
(In reply to Eric Blake from comment #7)
> for the larger allocation.  To know for sure, I'd need to know what the
> domain XML looks like before and after the point where it goes into the
> allocation growth spiral

Please check domain xml in attachment 994165 [details] - look for _logDomainXML.

1. Before snapshotCreateXML)

    <disk type='block' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native'/>
      <source dev='/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/54ba4f22-ee1a-4851-9710-0b242ebdc289/images/04e5c687-b7e4-4968-9f45-d359aa98938b/4b4e65a2-22dc-4166-a097-fc9662c80d11'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>04e5c687-b7e4-4968-9f45-d359aa98938b</serial>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>

2. Snapshot xml

<domainsnapshot>
    <disks>
        <disk name="vda" snapshot="external" type="block">
            <source dev="/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/54ba4f22-ee1a-4851-9710-0b242ebdc289/images/04e5c687-b7e4-4968-9f45-d359aa98938b/466e7cf0-ee80-499d-8d2a-c05e605a07cc" file="/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/54ba4f22-ee1a-4851-9710-0b242ebdc289/images/04e5c687-b7e4-4968-9f45-d359aa98938b/466e7cf0-ee80-499d-8d2a-c05e605a07cc" type="block"/>
        </disk>
    </disks>
</domainsnapshot>

(We are using both dev= and file= as suggested by you to support older
libvirt version which do not support type=block.)

3. After createSnapshotXML

    <disk type='block' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native'/>
      <source dev='/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/54ba4f22-ee1a-4851-9710-0b242ebdc289/images/04e5c687-b7e4-4968-9f45-d359aa98938b/466e7cf0-ee80-499d-8d2a-c05e605a07cc'/>
      <backingStore type='block' index='1'>
        <format type='qcow2'/>
        <source dev='/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/54ba4f22-ee1a-4851-9710-0b242ebdc289/images/04e5c687-b7e4-4968-9f45-d359aa98938b/../04e5c687-b7e4-4968-9f45-d359aa98938b/4b4e65a2-22dc-4166-a097-fc9662c80d11'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <serial>04e5c687-b7e4-4968-9f45-d359aa98938b</serial>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>

4. After blockRebase

    <disk type='block' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native'/>
      <source dev='/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/54ba4f22-ee1a-4851-9710-0b242ebdc289/images/04e5c687-b7e4-4968-9f45-d359aa98938b/466e7cf0-ee80-499d-8d2a-c05e605a07cc'/>
      <backingStore type='block' index='1'>
        <format type='qcow2'/>
        <source dev='/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/54ba4f22-ee1a-4851-9710-0b242ebdc289/images/04e5c687-b7e4-4968-9f45-d359aa98938b/../04e5c687-b7e4-4968-9f45-d359aa98938b/4b4e65a2-22dc-4166-a097-fc9662c80d11'/>
        <backingStore/>
      </backingStore>
      <mirror type='file' file='/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/04e5c687-b7e4-4968-9f45-d359aa98938b/466e7cf0-ee80-499d-8d2a-c05e605a07cc' format='qcow2' job='copy'>
        <format type='qcow2'/>
        <source file='/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/04e5c687-b7e4-4968-9f45-d359aa98938b/466e7cf0-ee80-499d-8d2a-c05e605a07cc'/>
      </mirror>
      <target dev='vda' bus='virtio'/>
      <serial>04e5c687-b7e4-4968-9f45-d359aa98938b</serial>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>

Note <mirror type=file..

5. After blockAboort

    <disk type='file' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native'/>
      <source file='/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/04e5c687-b7e4-4968-9f45-d359aa98938b/466e7cf0-ee80-499d-8d2a-c05e605a07cc'/>
      <backingStore type='block' index='1'>
        <format type='qcow2'/>
        <source dev='/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/04e5c687-b7e4-4968-9f45-d359aa98938b/../04e5c687-b7e4-4968-9f45-d359aa98938b/4b4e65a2-22dc-4166-a097-fc9662c80d11'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <serial>04e5c687-b7e4-4968-9f45-d359aa98938b</serial>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>

The disk type was converted to file, explaining the bad blockInfo readings.

Comment 10 Nir Soffer 2015-02-23 21:34:27 UTC
The attached patches are a partial fix that limits the damage. For the real fix we are blocked on libvirt bug 1195461.

Comment 11 Nir Soffer 2015-02-24 19:09:02 UTC
The new patch (http://gerrit.ovirt.org/38142) fixes the root cause, depending on fixed version of libvirt and libvirt-python.

Comment 14 Allon Mureinik 2015-04-08 11:21:34 UTC
The current proposed doctext is:

"""
Libvirt 1.2.8 introduced a regression where disk type is converted from "block" to "file" after live storage migration. This breaks the disk extension logic, leading to unwanted extension using all free space in the storage domain.

libvirt-daemon-1.2.8-16.el7_1.1 and libvirt-python-1.2.8-6.el7_1.1 fixed this issue by adding a new flag. RHEV-3.5.0-2 is using the new flag when available, fixing this issue.

However, the fix in RHEV-3.5.0-2 is effective only when using fixed versions of libvirt-daemon and libvirt-python.

Do not perform live storage migration on block storage in RHEL 7.1 unless unless you have the fixed version of libvirt-daemon (libvirt-daemon-1.2.8-16.el7_1.1) and libvirt-python (libvirt-python-1.2.8-6.el7_1.1). These versions should be available as a zero-day async release with RHEL 7.1.
"""

Most of this is no longer relevant in RHEV 3.6.0. 
RHEV 3.6.0 requires appropriate libvirt and libvirt-python versions with the relevant fix, so this message becomes a non issue.

Should I set requires-doctext- ?

Comment 15 Andrew Dahms 2015-04-09 10:28:16 UTC
Hi Allon,

Thank you for the needinfo request.

Shall I add this bug to the release notes for the RHEV 3.5.1 release? The flag is already set to '?', so we can review the text and add it in for you.

Also, understood that most of this will not be relevant for 3.6.0.

Kind regards,

Andrew

Comment 16 Allon Mureinik 2015-04-09 12:29:19 UTC
(In reply to Andrew Dahms from comment #15)
> Hi Allon,
> 
> Thank you for the needinfo request.
> 
> Shall I add this bug to the release notes for the RHEV 3.5.1 release? The
> flag is already set to '?', so we can review the text and add it in for you.
> 
> Also, understood that most of this will not be relevant for 3.6.0.
The current text is no longer relevant for neither 3.6.0 or 3.5.1.
Can you please remove it from both release notes?

Comment 17 Andrew Dahms 2015-04-13 11:07:50 UTC
Hi Allon,

Understood, and thank you for the clarification.
Done and done!

Please let me know if there is anything else I can do for you.

Kind regards,

Andrew

Comment 18 Ori Gofen 2015-04-26 14:50:57 UTC
verified on downstream

Comment 21 errata-xmlrpc 2016-03-09 19:27:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0362.html


Note You need to log in before you can comment on or make changes to this bug.