Bug 1099978

Summary: Maintain relative path to backing file image during live merge (block-commit)
Product: Red Hat Enterprise Linux 7 Reporter: Adam Litke <alitke>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: high    
Version: 7.0CC: alitke, bmcclain, bsarathy, dyuan, eblake, fsimonce, jsuchane, juzhang, mzhan, pkrempa, rbalakri, scohen, shyu, snagar, xuzhang
Target Milestone: pre-dev-freeze   
Target Release: 7.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.2.7-1.el7 Doc Type: Bug Fix
Doc Text:
Cause: management applications that use libvirt's snapshot feature sometimes create backing chains of disks with relative backing names so that they can be accessed via different methods without any change to the image itself. Subsequently when such backing chain would be target of operations that shorten it (snapshot deletion, merging, live merge, block commit, etc) a full path would be used by qemu in the image. This would then make the image inaccessible via different means as an absolute path would be hardcoded. Consequence: image chains became corrupted from point of view of the management application using libvirt as the relative links would be converted to absolute Fix: libvirt allows now to specify a flag that allows to keep the backing chain addresed via relative links in cooperation with qemu. (In case the backing chain was relative before) Result: complex storage architectures using relative paths between backing chain layers are now possible
 Story Points: ---
Clone Of:
: 1150322 (view as bug list) Environment:
Last Closed: 2015-03-05 07:35:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1061827    
Bug Blocks: 647386, 1035038, 1082754, 1083310, 1150322, 1196199    

Description Adam Litke 2014-05-21 17:11:10 UTC 
When live merge (block-commit) updates the backing file path of an image, it always uses an absolute path.
Given:

 [A] <- [B(bak=../a/img.raw)] <- [C(bak=../b/img.qcow)]

merging B into C would result in:

 [A] <- [B+C(bak=/nfs/a/img.raw)]

This is extremely problematic for image chains residing on NFS shares (as the mountpoint could change).

If the path that qemu needs to update is relative, it should be maintained relative:

 [A] <- [B+C(bak=../a/img.raw)]

This is being fixed in qemu-kvm (bug 1061827) but libvirt will need to adapt to the new API that is implemented in qemu.  Additionally, libvirt should provide a way for a management application to determine if this fix is in place so that virDomainBlockCommit may be called safely.
Comment 1 Dave Allan 2014-05-21 18:45:51 UTC 
Hey Peter, I'm pretty sure there's already a BZ for this, so please close as a dup if that's the case.  -Dave
Comment 6 Peter Krempa 2014-07-08 11:34:38 UTC 
Fixed upstream:

commit c6bf2f0ffc5d51f03eeb83103c323ff7949e8b59
Author: Peter Krempa <pkrempa>
Date:   Mon Apr 28 15:39:19 2014 +0200

    qemu: Add support for networked disks for block pull/block rebase
    
    Now that we are able to select images from the backing chain via indexed
    access we should also convert possible network sources to
    qemu-compatible strings before passing them to qemu.

commit 60244b56afb3438080b55ac838766af3a0037e03
Author: Peter Krempa <pkrempa>
Date:   Mon Apr 28 15:39:19 2014 +0200

    qemu: Add support for networked disks for block commit
    
    Now that we are able to select images from the backing chain via indexed
    access we should also convert possible network sources to
    qemu-compatible strings before passing them to qemu.

commit 37183e5db83fd476be552fd889edc74d82927981
Author: Peter Krempa <pkrempa>
Date:   Fri May 16 15:40:06 2014 +0200

    lib: Introduce flag VIR_DOMAIN_BLOCK_REBASE_RELATIVE
    
    Introduce flag for the block rebase API to allow the rebase operation to
    leave the chain relatively addressed. Also adds a virsh switch to enable
    this behavior.

commit bda44ca2ca5bf06721df07b09df4e102d929bd7c
Author: Peter Krempa <pkrempa>
Date:   Tue May 13 17:59:32 2014 +0200

    lib: Introduce flag VIR_DOMAIN_BLOCK_COMMIT_RELATIVE
    
    Introduce flag for the block commit API to allow the commit operation to
    leave the chain relatively addressed. Also adds a virsh switch to enable
    this behavior.

v1.2.6-67-gc6bf2f0
Comment 8 Shanzhi Yu 2014-09-28 10:35:31 UTC 
Hi Adam,

I am a little consused about how to reproduce/verify this bug. Could you please help to confirm if below steps are valid? 
If not, please give some more hints how to reproduce/verify this bug

Thanks

Package

# rpm -q qemu-kvm-rhev libvirt
qemu-kvm-rhev-2.1.2-1.el7.x86_64
libvirt-1.2.8-4.el7.x86_64

Step:

1. Mount same nfs to /mnt/nfs1, /mnt/nfs2,

# mount|grep /mnt/nfs
10.66.6.111:/mnt/nfs on /mnt/nfs1 type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.66.83.202,local_lock=none,addr=10.66.6.111)
10.66.6.111:/mnt/nfs on /mnt/nfs2 type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.66.83.202,local_lock=none,addr=10.66.6.111)

2. change dir to /mnt/nfs and create backing-chain with relative path

#pwd
/mnt/nfs

# qemu-img create -f qcow2 ../nfs1/rhel6.s1 -b /var/lib/libvirt/images/rhel6.img -o backing_fmt=qcow2 
Formatting '../nfs1/rhel6.s1', fmt=qcow2 size=4294967296 backing_file='/var/lib/libvirt/images/rhel6.img' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off 

# qemu-img create -f qcow2 ../nfs2/rhel6.s2 -b ../nfs1/rhel6.s1 -o backing_fmt=qcow2 
Formatting '../nfs2/rhel6.s2', fmt=qcow2 size=4294967296 backing_file='../nfs1/rhel6.s1' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off

# qemu-img info ../nfs2/rhel6.s2 --backing-chain
image: ../nfs2/rhel6.s2
file format: qcow2
virtual size: 4.0G (4294967296 bytes)
disk size: 196K
cluster_size: 65536
backing file: ../nfs1/rhel6.s1 (actual path: ../nfs2/../nfs1/rhel6.s1)
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false

image: ../nfs2/../nfs1/rhel6.s1
file format: qcow2
virtual size: 4.0G (4294967296 bytes)
disk size: 196K
cluster_size: 65536
backing file: /var/lib/libvirt/images/rhel6.img
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false

image: /var/lib/libvirt/images/rhel6.img
file format: qcow2
virtual size: 4.0G (4294967296 bytes)
disk size: 2.7G
cluster_size: 65536
Format specific information:
    compat: 0.10

3. Start guest with /mnt/nfs2/rhel6.s2

# virsh domblklist rhel6 
Target     Source
------------------------------------------------
vda        /mnt/nfs2/rhel6.s2

#virsh dumpxml rhel6|grep disk -A 20 
..
 <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/mnt/nfs2/rhel6.s2'>
        <seclabel model='selinux' labelskip='yes'/>
      </source>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/mnt/nfs2/../nfs1/rhel6.s1'/>
        <backingStore type='file' index='2'>
          <format type='qcow2'/>
          <source file='/var/lib/libvirt/images/rhel6.img'/>
          <backingStore/>
        </backingStore>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
..

4. Do active blockcommit 

4.1 
# virsh blockcommit rhel6 vda --shallow --active --verbose --pivot 
Block Commit: [100 %]
Successfully pivoted

#virsh dumpxml rhel6|grep disk -A 20 
..
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/mnt/nfs2/../nfs1/rhel6.s1'>
        <seclabel model='selinux' labelskip='yes'/>
      </source>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/var/lib/libvirt/images/rhel6.img'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
..

4.2
# virsh blockcommit rhel6 vda --shallow --active --verbose --pivot 
Block Commit: [100 %]
Successfully pivoted
#virsh dumpxml rhel6|grep disk -A 20 
..
 <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/rhel6.img'>
        <seclabel model='selinux' labelskip='yes'/>
      </source>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
..

5. Clean env and repeat step 1-2, also mount nfs to /var/lib/libvirt/mount

# mount|grep /mnt/nfs
10.66.6.111:/mnt/nfs on /mnt/nfs1 type nfs4 
10.66.6.111:/mnt/nfs on /mnt/nfs2 type nfs4 
10.66.6.111:/mnt/nfs on /var/lib/libvirt/mount type nfs4 

6. Check the chain

6.1 Check the disk chain from /mnt/nfs2(nfs1)

# qemu-img info /mnt/nfs2/rhel6.s2 --backing-chain

image: /mnt/nfs2/rhel6.s2
file format: qcow2
virtual size: 4.0G (4294967296 bytes)
disk size: 196K
cluster_size: 65536
backing file: ../nfs1/rhel6.s1 (actual path: /mnt/nfs2/../nfs1/rhel6.s1)
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false

image: /mnt/nfs2/../nfs1/rhel6.s1
file format: qcow2
virtual size: 4.0G (4294967296 bytes)
disk size: 196K
cluster_size: 65536
backing file: /var/lib/libvirt/images/rhel6.img
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false

image: /var/lib/libvirt/images/rhel6.img
file format: qcow2
virtual size: 4.0G (4294967296 bytes)
disk size: 2.7G
cluster_size: 65536
Format specific information:
    compat: 0.10


6.2 check the disk chain from /var/lib/libvirt/mount

# qemu-img info /var/lib/libvirt/mount/rhel6.s2 --backing-chain
qemu-img: Could not open '/var/lib/libvirt/mount/../nfs1/rhel6.s1': Could not open '/var/lib/libvirt/mount/../nfs1/rhel6.s1': No such file or directory

# qemu-img info /var/lib/libvirt/mount/rhel6.s2
image: /var/lib/libvirt/mount/rhel6.s2
file format: qcow2
virtual size: 4.0G (4294967296 bytes)
disk size: 196K
cluster_size: 65536
backing file: ../nfs1/rhel6.s1 (actual path: /var/lib/libvirt/mount/../nfs1/rhel6.s1)
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false


About the fail in step 6.2, should a bug or related with bug 1061827 ?

Many thanks
Comment 9 Adam Litke 2014-09-30 14:24:56 UTC 
Actually, you should do an internal merge not active.  An active commit does not rewrite the backing chain so it would not test the case described in this bug.

It is not necessary to use NFS to verify this bug.

To for an internal merge you'll need a longer backing chain:
  rhel6.img <-- rhel6.s1 <-- rhel6.s2 <-- rhel6.s3

Start the vm on rhel6.s3.

Then merge s2 into s1 with the following virsh command:

virsh blockcommit rhel6 vda /mnt/nfs1/rhel6.s1 /mnt/nfs1/rhel6.s2 --wait --keep-relative

Then verify that s2 is no longer in the backing chain and that s3 uses a relative path to reference s1 as its new backing file.

As for the problem in your #6.2, it looks like a legitimate issue.
Comment 11 Shanzhi Yu 2014-10-08 08:24:03 UTC 
(In reply to Adam Litke from comment #9)
> Actually, you should do an internal merge not active.  An active commit does
> not rewrite the backing chain so it would not test the case described in
> this bug.
> 
> It is not necessary to use NFS to verify this bug.
> 
> To for an internal merge you'll need a longer backing chain:
>   rhel6.img <-- rhel6.s1 <-- rhel6.s2 <-- rhel6.s3
> 
> Start the vm on rhel6.s3.
> 
> Then merge s2 into s1 with the following virsh command:
> 
> virsh blockcommit rhel6 vda /mnt/nfs1/rhel6.s1 /mnt/nfs1/rhel6.s2 --wait
> --keep-relative
> 
> Then verify that s2 is no longer in the backing chain and that s3 uses a
> relative path to reference s1 as its new backing file.
> 
> As for the problem in your #6.2, it looks like a legitimate issue.

Adam,
Thanks for your help.

My test steps are as below:

1. Prepare a backing chains with relative path
# pwd
/mnt/nfs

# qemu-img create -f qcow2 /var/lib/libvirt/images/rhel6.img 10G 
Formatting '/var/lib/libvirt/images/rhel6.img', fmt=qcow2 size=10737418240 encryption=off cluster_size=65536 lazy_refcounts=off 

# qemu-img create -f qcow2 /mnt/nfs1/rhel6.s1 -b /var/lib/libvirt/images/rhel6.img -o backing_fmt=qcow2 
Formatting '/mnt/nfs1/rhel6.s1', fmt=qcow2 size=10737418240 backing_file='/var/lib/libvirt/images/rhel6.img' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off

# qemu-img create -f qcow2 /mnt/nfs1/rhel6.s2 -b ../nfs1/rhel6.s1 -o backing_fmt=qcow2 
Formatting '/mnt/nfs1/rhel6.s2', fmt=qcow2 size=10737418240 backing_file='../nfs1/rhel6.s1' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off 

# qemu-img create -f qcow2 /var/lib/libvirt/images/rhel6.s3 -b /mnt/nfs1/rhel6.s2 -o backing_fmt=qcow2 
Formatting '/var/lib/libvirt/images/rhel6.s3', fmt=qcow2 size=10737418240 backing_file='/mnt/nfs1/rhel6.s2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off

# qemu-img info /var/lib/libvirt/images/rhel6.s3 --backing-chain
image: /var/lib/libvirt/images/rhel6.s3
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 196K
cluster_size: 65536
backing file: /mnt/nfs1/rhel6.s2
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false

image: /mnt/nfs1/rhel6.s2
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 196K
cluster_size: 65536
backing file: ../nfs1/rhel6.s1 (actual path: /mnt/nfs1/../nfs1/rhel6.s1)
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false

image: /mnt/nfs1/../nfs1/rhel6.s1
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 196K
cluster_size: 65536
backing file: /var/lib/libvirt/images/rhel6.img
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false

image: /var/lib/libvirt/images/rhel6.img
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 1.3G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false

2. Boot up a guest with rhel6.s3 as source file
#virsh start rh6

#virsh dumpxml rh6|grep disk -A 16
 <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/rhel6.s3'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/mnt/nfs1/rhel6.s2'/>
        <backingStore type='file' index='2'>
          <format type='qcow2'/>
          <source file='/mnt/nfs1/../nfs1/rhel6.s1'/>
          <backingStore type='file' index='3'>
            <format type='qcow2'/>
            <source file='/var/lib/libvirt/images/rhel6.img'/>
            <backingStore/>
          </backingStore>
        </backingStore>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
..

3. Do blockcommit from s2 to s1

# virsh blockcommit rh6 vda --top vda[1] --base vda[2] --wait  --keep-relative 
error: Requested operation is not valid: can't keep relative backing relationship

# virsh blockcommit rh6 vda --top vda[1] --base vda[2] --wait  

Now in synchronized phase

4. Check guest xml

# virsh dumpxml rh6 |grep disk -A 12
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/rhel6.s3'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/mnt/nfs1/../nfs1/rhel6.s1'/>
        <backingStore type='file' index='2'>
          <format type='qcow2'/>
          <source file='/var/lib/libvirt/images/rhel6.img'/>
          <backingStore/>
        </backingStore>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
..

So after blockcommit, rhel6.s3 uses a relative path to reference rhel6.s1
Comment 12 Shanzhi Yu 2014-10-08 09:03:19 UTC 
With a more longer backing chain contain relative path

1. Prepare guest 
#virsh dumpxml rh6 |grep disk -A 20

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/rhel6.s4'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/mnt/nfs1/rhel6.s3'/>
        <backingStore type='file' index='2'>
          <format type='qcow2'/>
          <source file='/mnt/nfs1/../nfs1/rhel6.s2'/>
          <backingStore type='file' index='3'>
            <format type='qcow2'/>
            <source file='/mnt/nfs1/../nfs1/../nfs1/rhel6.s1'/>
            <backingStore type='file' index='4'>
              <format type='qcow2'/>
              <source file='/var/lib/libvirt/images/rhel6.img'/>
              <backingStore/>
            </backingStore>
          </backingStore>
        </backingStore>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
..

2. Do blockcommit with "--keep-relative" options

# virsh blockcommit rh6 vda --top vda[2] --base vda[3] --wait --keep-relative 

Now in synchronized phase

3. Check guest xml

# virsh dumpxml rh6 |grep disk -A 16
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/rhel6.s4'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/mnt/nfs1/rhel6.s3'/>
        <backingStore type='file' index='2'>
          <format type='qcow2'/>
          <source file='/mnt/nfs1/../nfs1/../nfs1/rhel6.s1'/>
          <backingStore type='file' index='3'>
            <format type='qcow2'/>
            <source file='/var/lib/libvirt/images/rhel6.img'/>
            <backingStore/>
          </backingStore>
        </backingStore>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
Comment 13 Shanzhi Yu 2014-10-08 09:07:14 UTC 
Peter,

I would change this bug to VERIFIED if you can confirm the test in comment 11 and comment 12 is valid and enough.


Thanks
Comment 15 Shanzhi Yu 2014-12-09 08:11:20 UTC 
Move to VERIFIED according comment 14, as bug 1150322 is VERIFIED.
Comment 20 errata-xmlrpc 2015-03-05 07:35:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html