This may be relevant; grepping the output of 'query-block' before the two commit attempts shows:

*** Before first commits
                            "filename": "/tmp/images/c/../b/../a/a",
                        "backing-filename-format": "qcow2",
                        "filename": "/tmp/images/c/../b/b",
                        "backing-filename": "/tmp/images/c/../b/../a/a",
                    "backing-filename-format": "qcow2",
                    "filename": "/tmp/images/c/c",
                    "backing-filename": "/tmp/images/c/../b/b",
*** Before second commits
                            "filename": "/tmp/images/c/../b/../a/a",
                        "backing-filename-format": "qcow2",
                        "filename": "/tmp/images/b/b",
                        "backing-filename": "/tmp/images/c/../b/../a/a",
                    "backing-filename-format": "qcow2",
                    "filename": "/tmp/images/c/c",
                    "backing-filename": "/tmp/images/b/b",

That is, when qemu locates 'b' by chasing 'c's metadata, it knows the file as /tmp/images/c/../b/b.  But when qemu is told to locate 'b' during snapshot creation, it knows the file as /tmp/images/b/b.

Libvirt is executing this (modulo different 'id' fields) for BOTH commit sequences:
 "{\"execute\":\"block-commit\",\"arguments\":{\"device\":\"drive-virtio-disk0\",\"top\":\"/tmp/images/c/c\",\"base\":\"/tmp/images/c/../b/b\"},\"id\":\"libvirt-11\"}"
"{\"execute\":\"block-commit\",\"arguments\":{\"device\":\"drive-virtio-disk0\",\"top\":\"/tmp/images/c/../b/b\",\"base\":\"/tmp/images/c/../b/../a/a\"},\"id\":\"libvirt-17\"}"

But because qemu is tracking the name of b differently between the two attempts, it may explain why the second attempt fails to locate the starting point (especially if qemu is using strcmp on original strings rather than checking canonical file names).

Created attachment 998591 [details]
reproducer using only qemu

This script reproduces the bug; specifically, it does the test 3 ways:

1) With the image chain pre-created, and then block-commits of the active layer.  This is successful.

2) With the image chain formed dynamically via live snapshots, and then block-commits of the active layer, using the relative pathname for each active layer (e.g. /tmp/images/b/../c/c).  This is successful.

3.) With the image chain formed dynamically via live snapshots, and then block-commits of the active layer, using the absolute pathname for the initial block-commit active layer (e.g. /tmp/images/c/c).  This fails, as QEMU cannot find the image.

(In reply to Eric Blake from comment #1)

[...]

> 
> That is, when qemu locates 'b' by chasing 'c's metadata, it knows the file
> as /tmp/images/c/../b/b.  But when qemu is told to locate 'b' during
> snapshot creation, it knows the file as /tmp/images/b/b.
> 

[...]


This is indeed what is happening.  Please see my attachment #998591 [details] and the info in comment #3 - each run through dumps the block info before the relevant block-commit commands.

If you are looking for a solution that does not involve a new QEMU version, could libvirt run block-query prior to issuing block-commit?  The filename string QEMU will be searching for will be present in the block info chain returned, and libvirt could parse that to determine what to use for the filename for image in the chain.

(This is more reason why using node-names at some point for all operations would be preferable to filenames)

The ultimate upstream solution would be converting libvirt over to use node name rather than filename references, but that won't be backportable. Also, it would be nice to clone this bug to qemu to handle all filenames that resolve to the same location, rather than just a string compare on a particular (and not necessarily canonical) spelling of a filename.  But such a clone can be deferred to 7.2 if we can teach libvirt to work with existing qemu.  The idea of having libvirt check 'query-block' to track actual names in use by qemu seems worthwhile, and from the perspective of a minimal fix, it is appealing (if we backport to 7.1.z, doing JUST a libvirt patch is nicer than requiring both a libvirt and a qemu patch).  So I'm currently in the middle of coding up an attempted solution to teach libvirt to use 'query-block' to reset its notion of the filenames qemu is using after any operation that alters the chain of files in use by a disk device.

I can reproduce it with steps just as in comment 0.

1. Prepare a backing chains with relative path

2. Create external snapshot with using the existing backing chains when guest is running

3. Do active blockcommit, will met error on second time

(In reply to Jeff Cody from comment #3)
> Created attachment 998591 [details]
> reproducer using only qemu
> 
> This script reproduces the bug; specifically, it does the test 3 ways:
> 
> 1) With the image chain pre-created, and then block-commits of the active
> layer.  This is successful.
> 
> 2) With the image chain formed dynamically via live snapshots, and then
> block-commits of the active layer, using the relative pathname for each
> active layer (e.g. /tmp/images/b/../c/c).  This is successful.
> 
> 3.) With the image chain formed dynamically via live snapshots, and then
> block-commits of the active layer, using the absolute pathname for the
> initial block-commit active layer (e.g. /tmp/images/c/c).  This fails, as
> QEMU cannot find the image.

Hi Jeff,

Do we need a bug for qemu?

Bests,
Shaolong

Hi Eric,

Do you have any updates on your progress?  Are you still planning to deliver the libvirt-based workaround?  If so, what upstream libvirt release would you expect to have the fix and when would you expect a zStream for RHEL7?

Thanks!

Upstream proposed patch:
https://www.redhat.com/archives/libvir-list/2015-March/msg00605.html

bug 1199036 regarding blockjob lock problems explains some of the crashes observed while testing this bug (and Peter's patches mentioned in comment 16, for those that can read it); marking it as a dependency (any z-stream build needs both sets of patches)

Potential 7.1.z patches: http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-March/msg00447.html

POST for 7.2 via rebase

(In reply to Eric Blake from comment #20)
> Potential 7.1.z patches:
> http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-March/
> msg00447.html

Updated version for 7.1.z: http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-March/msg00479.html

Hi Eric,
Regarding the issue mentioned in comment#26, I can always reproduce it on a disk on which no OS is installed. Active commit with pivot is finished and return with no error, however, domain xml is not changed

libvirt version
libvirt-1.2.14-1.el7.x86_64
qemu-kvm-rhev-2.2.0-8.el7.x86_64

Steps to reproduce
1. Prepare a running guest with the following xml
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vm1.qcow2'/>  ==> bootable disk
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/test.img'/>  ==> no OS is installed
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>

2. create 3 external disk only snapshot for vdb
#  virsh snapshot-create-as testvm3 --no-metadata --disk-only --diskspec vda,snapshot=no --diskspec vdb,file=/var/lib/libvirt/images/test.s1
Domain snapshot 1428653468 created
#  virsh snapshot-create-as testvm3 --no-metadata --disk-only --diskspec vda,snapshot=no --diskspec vdb,file=/var/lib/libvirt/images/test.s2
Domain snapshot 1428653472 created
#  virsh snapshot-create-as testvm3 --no-metadata --disk-only --diskspec vda,snapshot=no --diskspec vdb,file=/var/lib/libvirt/images/test.s3
Domain snapshot 1428653473 created

3. dump domain xml
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vm1.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test.s3'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/var/lib/libvirt/images/test.s2'/>
        <backingStore type='file' index='2'>
          <format type='qcow2'/>
          <source file='/var/lib/libvirt/images/test.s1'/>
          <backingStore type='file' index='3'>
            <format type='raw'/>
            <source file='/var/lib/libvirt/images/test.img'/>
            <backingStore/>
          </backingStore>
        </backingStore>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>

4. Do active blockcommit
# virsh -k0 blockcommit --active --verbose testvm3 vdb --shallow --pivot
Block Commit: [100 %]
Successfully pivoted

5. check domain xml
# virsh dumpxml testvm3 | grep disk -a10
 <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vm1.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test.s3'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/var/lib/libvirt/images/test.s2'/>
        <backingStore type='file' index='2'>
          <format type='qcow2'/>
          <source file='/var/lib/libvirt/images/test.s1'/>
          <backingStore type='file' index='3'>
            <format type='raw'/>
            <source file='/var/lib/libvirt/images/test.img'/>
            <backingStore/>
          </backingStore>
        </backingStore>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>

6. # virsh -k0 blockcommit --active --verbose testvm3 vdb --shallow --pivot
error: internal error: qemu block name '/var/lib/libvirt/images/test.s2' doesn't match expected '/var/lib/libvirt/images/test.s3'

As blockcommit with pivot always fail due to bz1197592. In order to not block to verify this bz, do pivot after several seconds when guest in mirror phrase

Verified on libvirt-1.2.16-1.el7.x86_64

Tested nfs, gluster backend and block device as virtual disk
1. nfs backend
cd /tmp || exit 1
umount /tmp/images
virsh -k0 destroy testvm3 2>/dev/null
rm -rf images

mkdir -p images || exit 1
mount 10.66.5.63:/var/lib/libvirt/images/nfs /tmp/images

mkdir -p images/a images/b images/c images/d || exit 1

(cd images/a &&
cp /var/lib/libvirt/images/vm1.qcow2 a) || exit 1
sleep 5
(cd images/b &&
qemu-img create -f qcow2 -o backing_file=../a/a,backing_fmt=qcow2 b) || exit 1
(cd images/c &&
qemu-img create -f qcow2 -o backing_file=../b/b,backing_fmt=qcow2 c) || exit 1
(cd images/d &&
qemu-img create -f qcow2 -o backing_file=../c/c,backing_fmt=qcow2 d) || exit 1

emulator='<emulator>/usr/libexec/qemu-kvm</emulator>'

virsh -k0 create /dev/stdin <<EOF
<domain type='kvm'>
<name>testvm3</name>
<memory unit='MiB'>256</memory>
 <vcpu>1</vcpu>
 <os>
   <type arch='x86_64'>hvm</type>
 </os>
 <devices>
   $emulator
   <disk type='file' device='disk'>
     <driver name='qemu' type='qcow2'/>
     <source file='/tmp/images/d/d'/>
     <target dev='vda' bus='virtio'/>
   </disk>
   <graphics type='vnc'/>
 </devices>
</domain>
EOF

sleep 60

virsh -k0 blockcommit --active --verbose testvm3 vda --shallow --wait
sleep 5
virsh blockjob testvm3 vda --pivot
virsh -k0 blockcommit --active --verbose testvm3 vda --shallow --wait
sleep 5
virsh blockjob testvm3 vda --pivot
virsh -k0 blockcommit --active --verbose testvm3 vda --shallow --wait
sleep 5
virsh blockjob testvm3 vda --pivot


virsh -k0 snapshot-create-as --no-metadata testvm3 --reuse-external \
      --disk-only --diskspec vda,file=/tmp/images/b/b
virsh -k0 snapshot-create-as --no-metadata testvm3 --reuse-external \
      --disk-only --diskspec vda,file=/tmp/images/c/c
virsh -k0 snapshot-create-as --no-metadata testvm3 --reuse-external \
      --disk-only --diskspec vda,file=/tmp/images/d/d


virsh -k0 blockcommit --active --verbose testvm3 vda --shallow --wait
sleep 5
virsh blockjob testvm3 vda --pivot
virsh -k0 blockcommit --active --verbose testvm3 vda --shallow --wait
sleep 5
virsh blockjob testvm3 vda --pivot
virsh -k0 blockcommit --active --verbose testvm3 vda --shallow --wait
sleep 5
virsh blockjob testvm3 vda --pivot

All passed, see attached scripts

Created attachment 1039840 [details]
bug verification with gluster backend

Created attachment 1039841 [details]
bug verification with nfs backend

Created attachment 1039843 [details]
bug verification with block device

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html