Bug 1397870

You should use the disk target name "vda" or the index into the backing chain using "vda[1]" (according to the 'index' field in the backing store) to address the individual parts of the backing chain.

The format is documented along with the API http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockCommit

# virsh blockcommit fioo5 vda --base vda[1] --verbose --active --wait
error: internal error: unable to execute QEMU command 'block-commit': Base 'gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/27b0cbcb-4dfd-4eeb-8ab0-8fda54a6d8a4/d4c23ec6-20ce-4a2f-9b32-ca91e65a114a' not found

However this file does exist on checking on mount point - 
[root@rhsdev-grafton4 ~]# mount -t glusterfs 10.70.37.28:/vmstore /mnt/vmstore-mnt
[root@rhsdev-grafton4 ~]# ls -la /mnt/vmstore-mnt/912d9062-3881-479b-a6e5-7b074a252cb6/images/27b0cbcb-4dfd-4eeb-8ab0-8fda54a6d8a4/d4c23ec6-20ce-4a2f-9b32-ca91e65a114a
-rw-rw----. 1 vdsm kvm 18153472 Nov 21 14:39 /mnt/vmstore-mnt/912d9062-3881-479b-a6e5-7b074a252cb6/images/27b0cbcb-4dfd-4eeb-8ab0-8fda54a6d8a4/d4c23ec6-20ce-4a2f-9b32-ca91e65a114a


What am I missing?

I wanted to reproduce it myself but my gluster setup is currently broken.

Could you please add libvirtd debug log containing information when the call failed?

The error message in Comment 2 is from qemu. Libvirt queries qemu for the name of the image as qemu reports it and uses that string to identify the volume files to qemu. It looks like qemu is rejecting those though for some reason.

The debug log will show the query process and other data.

It's very likely that this is a bug in qemu, so the data will help to make it more specific.

Created attachment 1228901 [details]
libvirtd.log

From the debug log the following conversation with qemu takes place. The very long lines are caused by rather long file paths.

Libvirt executes:
{
    "execute": "query-block",
    "id": "libvirt-143"
}
to determine the names for the images that will be used with the block-commit comman.

Qemu returns the following data:
{
    "return": [
        {
            "io-status": "ok",
            "device": "drive-ide0-1-0",
            "locked": false,
            "removable": true,
            "tray_open": false,
            "type": "unknown"
        },
        {
            "io-status": "ok",
            "device": "drive-virtio-disk0",
            "locked": false,
            "removable": false,
            "inserted": {
                "iops_rd": 0,
                "detect_zeroes": "off",
                "image": {
                    "backing-image": {
                        "backing-image": {
                            "virtual-size": 32212254720,
                            "filename": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/27b0cbcb-4dfd-4eeb-8ab0-8fda54a6d8a4/f5098809-e719-404b-8d80-3f824f4333a2",
                            "format": "raw",
                            "actual-size": 12978744832
                        },
                        "backing-filename-format": "raw",
                        "virtual-size": 32212254720,
                        "filename": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/27b0cbcb-4dfd-4eeb-8ab0-8fda54a6d8a4/d4c23ec6-20ce-4a2f-9b32-ca91e65a114a",
                        "cluster-size": 65536,
                        "format": "qcow2",
                        "actual-size": 18153472,
                        "format-specific": {
                            "type": "qcow2",
                            "data": {
                                "compat": "1.1",
                                "lazy-refcounts": false,
                                "refcount-bits": 16,
                                "corrupt": false
                            }
                        },
                        "backing-filename": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/27b0cbcb-4dfd-4eeb-8ab0-8fda54a6d8a4/f5098809-e719-404b-8d80-3f824f4333a2",
                        "dirty-flag": false
                    },
                    "backing-filename-format": "qcow2",
                    "virtual-size": 32212254720,
                    "filename": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/27b0cbcb-4dfd-4eeb-8ab0-8fda54a6d8a4/027a3b37-77d4-4fa9-8173-b1fedba1176c",
                    "cluster-size": 65536,
                    "format": "qcow2",
                    "actual-size": 50003968,
                    "format-specific": {
                        "type": "qcow2",
                        "data": {
                            "compat": "1.1",
                            "lazy-refcounts": false,
                            "refcount-bits": 16,
                            "corrupt": false
                        }
                    },
                    "full-backing-filename": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/27b0cbcb-4dfd-4eeb-8ab0-8fda54a6d8a4/d4c23ec6-20ce-4a2f-9b32-ca91e65a114a",
                    "backing-filename": "d4c23ec6-20ce-4a2f-9b32-ca91e65a114a",
                    "dirty-flag": false
                },
                "iops_wr": 0,
                "ro": false,
                "backing_file_depth": 2,
                "drv": "qcow2",
                "iops": 0,
                "bps_wr": 0,
                "write_threshold": 0,
                "backing_file": "d4c23ec6-20ce-4a2f-9b32-ca91e65a114a",
                "encrypted": false,
                "bps": 0,
                "bps_rd": 0,
                "cache": {
                    "no-flush": false,
                    "direct": true,
                    "writeback": true
                },
                "file": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/27b0cbcb-4dfd-4eeb-8ab0-8fda54a6d8a4/027a3b37-77d4-4fa9-8173-b1fedba1176c",
                "encryption_key_missing": false
            },
            "type": "unknown"
        },
        {
            "io-status": "ok",
            "device": "drive-virtio-disk1",
            "locked": false,
            "removable": false,
            "inserted": {
                "iops_rd": 0,
                "detect_zeroes": "off",
                "image": {
                    "backing-image": {
                        "backing-image": {
                            "virtual-size": 21474836480,
                            "filename": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/b158a429-7459-47e5-8584-9975f187e441",
                            "format": "raw",
                            "actual-size": 16876257280
                        },
                        "backing-filename-format": "raw",
                        "virtual-size": 21474836480,
                        "filename": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/e5d9b44d-032f-4264-9591-138f757af14a",
                        "cluster-size": 65536,
                        "format": "qcow2",
                        "actual-size": 197120,
                        "format-specific": {
                            "type": "qcow2",
                            "data": {
                                "compat": "1.1",
                                "lazy-refcounts": false,
                                "refcount-bits": 16,
                                "corrupt": false
                            }
                        },
                        "backing-filename": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/b158a429-7459-47e5-8584-9975f187e441",
                        "dirty-flag": false
                    },
                    "backing-filename-format": "qcow2",
                    "virtual-size": 21474836480,
                    "filename": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/09b0324d-6c62-4237-8fd4-bc1290efa85a",
                    "cluster-size": 65536,
                    "format": "qcow2",
                    "actual-size": 197120,
                    "format-specific": {
                        "type": "qcow2",
                        "data": {
                            "compat": "1.1",
                            "lazy-refcounts": false,
                            "refcount-bits": 16,
                            "corrupt": false
                        }
                    },
                    "full-backing-filename": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/e5d9b44d-032f-4264-9591-138f757af14a",
                    "backing-filename": "e5d9b44d-032f-4264-9591-138f757af14a",
                    "dirty-flag": false
                },
                "iops_wr": 0,
                "ro": false,
                "backing_file_depth": 2,
                "drv": "qcow2",
                "iops": 0,
                "bps_wr": 0,
                "write_threshold": 0,
                "backing_file": "e5d9b44d-032f-4264-9591-138f757af14a",
                "encrypted": false,
                "bps": 0,
                "bps_rd": 0,
                "cache": {
                    "no-flush": false,
                    "direct": true,
                    "writeback": true
                },
                "file": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/09b0324d-6c62-4237-8fd4-bc1290efa85a",
                "encryption_key_missing": false
            },
            "type": "unknown"
        }
    ],
    "id": "libvirt-143"
}

Libvirt traverses the backing chain as returned and selects the corresponding entry. Libvirt uses the 'filename' field of the corresponding image in the backing chain to identify the image further.

Then libvirt issues the following qemu command: 
{
    "execute": "block-commit",
    "arguments": {
        "device": "drive-virtio-disk1",
        "top": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/09b0324d-6c62-4237-8fd4-bc1290efa85a",
        "base": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/e5d9b44d-032f-4264-9591-138f757af14a"
    },
    "id": "libvirt-144"
}

And qemu replies:
{
    "id": "libvirt-144",
    "error": {
        "class": "GenericError",
        "desc": "Base 'gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/e5d9b44d-032f-4264-9591-138f757af14a' not found"
    }
}

This hints that qemu is not able to properly recognize the qemu URIs which are returned by query-block.

Libvirt plans to use node names instead in the future, but that won't help in this case, since the qemu in question does not auto-generate them.

Reassigning to qemu to see whether the gluster backend URI matching can be fixed somehow.

Jeff,

This is an urgent item for Grafton/RHV, any chance you can give some visibility to this ASAP?

(In reply to Bronce McClain from comment #11)
> Jeff,
> 
> This is an urgent item for Grafton/RHV, any chance you can give some
> visibility to this ASAP?

Hi, yes.  I'm currently investigating this issue now.

I don't think this is an issue with the 7.3 release versions of qemu-rhev - the 7.3 release version worked in my testing, although I was able to reproduce the bug with the 7.2 version listed in the description.  

Are you able to test the whole stack through oVirt using qemu-kvm-rhev-2.6.0-27.el7, to verify this?

I retested with version suggested, and it works. Thanks

Versions used (as upstream testing):
qemu-img-ev-2.6.0-27.1.el7.x86_64
qemu-kvm-tools-ev-2.6.0-27.1.el7.x86_64
qemu-kvm-ev-2.6.0-27.1.el7.x86_64

You can close this bug.

Closing per Comment #16

On further testing with RHEL 7.3 packages, I notice this problem in the following scenario:

1. Create a VM from oVirt
2. Create a snapshot
3. Delete the snapshot - gives error that "libvirtError: internal error: unable to execute QEMU command 'block-commit': Base 'gluster://10.70.37.28/vmstore/imgpath' not found"

Now, restart the VM (i.e shutdown and start on same host)
1. Delete snapshot - works as expected

Seems like a caching related issue is my wild guess.

In Comment 16 when I mentioned it works, I upgraded packages from 7.2, and retried with an existing VM and could not reproduce it.

When the qemu process is not restarted between creating and block-committing the snapshot, hitting the issue again. Re-opening this.

Jeff, any thoughts?

(In reply to Sahina Bose from comment #18)
> On further testing with RHEL 7.3 packages, I notice this problem in the
> following scenario:
> 
> 1. Create a VM from oVirt
> 2. Create a snapshot
> 3. Delete the snapshot - gives error that "libvirtError: internal error:
> unable to execute QEMU command 'block-commit': Base
> 'gluster://10.70.37.28/vmstore/imgpath' not found"
> 
> Now, restart the VM (i.e shutdown and start on same host)
> 1. Delete snapshot - works as expected
> 
> Seems like a caching related issue is my wild guess.
> 
> In Comment 16 when I mentioned it works, I upgraded packages from 7.2, and
> retried with an existing VM and could not reproduce it.
> 
> When the qemu process is not restarted between creating and block-committing
> the snapshot, hitting the issue again. Re-opening this.
> 
> Jeff, any thoughts?

When I reproduced this bug with the 7.2 packages, I created the images with gluster URI backing files via qemu-img, and then performed the block-commit.  This did not work with 7.2, but it did work with the 7.3 packages.

I will try your scenario - creating the snapshot live, and then performing the block-commit, all in the same qemu runtime process.

(In reply to Sahina Bose from comment #18)
> On further testing with RHEL 7.3 packages, I notice this problem in the
> following scenario:
> 
> 1. Create a VM from oVirt
> 2. Create a snapshot
> 3. Delete the snapshot - gives error that "libvirtError: internal error:
> unable to execute QEMU command 'block-commit': Base
> 'gluster://10.70.37.28/vmstore/imgpath' not found"
> 
> Now, restart the VM (i.e shutdown and start on same host)
> 1. Delete snapshot - works as expected
> 
> Seems like a caching related issue is my wild guess.
> 
> In Comment 16 when I mentioned it works, I upgraded packages from 7.2, and
> retried with an existing VM and could not reproduce it.
> 
> When the qemu process is not restarted between creating and block-committing
> the snapshot, hitting the issue again. Re-opening this.
> 
> Jeff, any thoughts?

After trying to reproduce what you are seeing here with QEMU, I don't see it.

Could you post your logs from this new attempt, so I can see the libvirt->QEMU communications?


Here is what I did:

1. started QEMU with an existing gluster qcow2 image

2. created a live snapshot:

{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive-virtio-blk1", "snapshot-file": "gluster://192.168.15.180/gv0/912d9062-3881-479b-a6e5-7b074a252cb6/images/67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/e5d9b44d-032f-4264-9591-283471902874", "format": "qcow2" } }

3. Created some data to test with in the guest, on the new snapshot layer:

# cd /mnt/test

# dd if=/dev/urandom of=test.bin bs=1M count=16
# sha1sum test.bin > test.bin.sha1sum
# sync


3. Performed a block commit of this active layer:


{
    "execute": "block-commit",
    "arguments": {
        "device": "drive-virtio-blk1",
        "base": "gluster://192.168.15.180/gv0/912d9062-3881-479b-a6e5-7b074a252cb6/images/67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/e5d9b44d-032f-4264-9591-138f757af14a",
        "top": "gluster://192.168.15.180/gv0/912d9062-3881-479b-a6e5-7b074a252cb6/images/67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/e5d9b44d-032f-4264-9591-283471902874"
    },
    "id": "test-2"
}


4. The active layer commit was successful:

{"return": {}, "id": "test-2"}
{"timestamp": {"seconds": 1484586065, "microseconds": 235811}, "event": "BLOCK_JOB_READY", "data": {"device": "drive-virtio-blk1", "len": 18743296, "offset": 18743296, "speed": 0, "type": "commit"}}

5. Completed the active layer commit:
{ "execute": "block-job-complete", "arguments": { "device": "drive-virtio-blk1" }}

{"timestamp": {"seconds": 1484586101, "microseconds": 467089}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive-virtio-blk1", "len": 19202048, "offset": 19202048, "speed": 0, "type": "commit"}}

6. Shut system down, start back up and verify the data was written from the active layer to its backing file.


Here is the version of QEMU I was using:
qemu-kvm-rhev-2.3.0-31.el7_2.21

Created attachment 1242132 [details]
2-libvirtd.log

I'm not sure why this is not reproducible on your setup. I'm using the latest upstream qemu from oVirt repo.

Note that the issue is only reproducible for me if I do a live-snap and live-merge with the same qemu process.

(In reply to Sahina Bose from comment #22)
> I'm not sure why this is not reproducible on your setup. I'm using the
> latest upstream qemu from oVirt repo.
> 
> Note that the issue is only reproducible for me if I do a live-snap and
> live-merge with the same qemu process.

I've looked through the logs in your system, and here are some key differences:


From the query-block JSON (just a snippet here):

                "file": "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/27b0cbcb-4dfd-4eeb-8ab0-8fda54a6d8a4/af85a824-ec2e-468c-aa10-8cdac4b6c316",
                "image": {
                    "actual-size": 197120,
                    "backing-filename": "d4c23ec6-20ce-4a2f-9b32-ca91e65a114a",
                    "backing-filename-format": "qcow2",


Why is the backing-filename "d4c23ec6-20ce-4a2f-9b32-ca91e65a114a" here, instead of "gluster://10.70.37.28/vmstore/912d9062-3881-479b-a6e5-7b074a252cb6/images/27b0cbcb-4dfd-4eeb-8ab0-8fda54a6d8a4/d4c23ec6-20ce-4a2f-9b32-ca91e65a114a"?


This is why QEMU cannot find the backing-file.


I noticed when the live snapshot is created, mode=existing is used. How did the backing-file get set to a filename that is not a gluster URI?


When I try to duplicate this using the same versions of QEMU as on your test system (10:2.6.0-27.1.el7), even if I set the backing-file manually in the existing image snapshot via qemu-img, I don't see a similar backing-file value occurring.  I feel like I am missing some information about what happens to that snapshot image, with respect to setting the backing filename outside of the current QEMU process.

oVirt creates snapshots with relative file names, thus the rather weird backing-filename. To achieve this it used to be necessary to pre-create the snapshot image with the qcow2 format and the backing-filename field filled and then use mode=existing. Libvirt did not yet adapt to the option to rewrite the backing-filename directly and thus we don't expose such mechanism to the users yet.

To achieve mode=existing with libvirt you can use the --reuse-external flag for virsh or the appropriate flag for the API. oVirt also uses --without-metadata to avoid recording the snapshot in libvirt's data.

oVirt's explanation for using relative 'backing-filename's is so that they can easily switch access to storage without having to rewrite the backing chain.

(In reply to Peter Krempa from comment #25)
> oVirt creates snapshots with relative file names, thus the rather weird
> backing-filename. To achieve this it used to be necessary to pre-create the
> snapshot image with the qcow2 format and the backing-filename field filled
> and then use mode=existing. Libvirt did not yet adapt to the option to
> rewrite the backing-filename directly and thus we don't expose such
> mechanism to the users yet.
> 
> To achieve mode=existing with libvirt you can use the --reuse-external flag
> for virsh or the appropriate flag for the API. oVirt also uses
> --without-metadata to avoid recording the snapshot in libvirt's data.
> 
> oVirt's explanation for using relative 'backing-filename's is so that they
> can easily switch access to storage without having to rewrite the backing
> chain.

This seems like it needs to be fixed in libvirt or oVirt, somehow.  QEMU can't find the backing-file for the block-commit because the backing-file name that has been set by oVirt in the active layer is different from the base-name oVirt is specifying in the block-commit command.

For qemu-kvm-rhev 7.3+, QEMU's auto-generated node-names should be available, so that may be a possible solution.

Nir, can the backing-file be changed to provide the absolute path from oVirt. (I see there's "full-backing-filename" which already contains the absolute path)

(In reply to Sahina Bose from comment #27)
> Nir, can the backing-file be changed to provide the absolute path from
> oVirt. (I see there's "full-backing-filename" which already contains the
> absolute path)

I don't think so, the backing file should be the name of the volume, assuming that
the volume is in the same directory in file storage. This make it possible to move
disks between storage types (block/file) without rewriting the chain.

I think this should be solved in libvirt/qemu.

It may be possible for QEMU to compare against the full-backing-filename in the case of a protocol URI being specified, if the backing-file field does not match.  I'll run a test here, and if it works in my local test, I'll put together a test brew build to see if it works in the ovirt stack as a proof of concept.

I hope that we can make this work in the same way local images work.

File / block based image:

  qemu image path: /rhev/data-center/pool-uuid/dom-uuid/images/img-uuid/top-uuid
  image backing file: backing-uuid

File and block are the same for ovirt. For block devices, we create a directory
on the host with symlinks to the block devices:

  /rhev/data-center/mnt/blockSD/dom-uuid/images/img-uuid/top-uuid ->
    /dev/dom-uuid/top-uuid

Network disk:

  qemu image path: gluster:gluster-name/dom-uuid/images/img-uuid/top-uuid
  image backing file: backing-uuid

Hopefully qemu can treat the "backing-uuid" as a relative url, and access
the backing file as:

  gluster:gluster-name/dom-uuid/images/img-uuid/backing-uuid

If it works for local (file: url), why not for gluster: urls?

Jeff, do you think this can work?

Could you please try the version in this brew build:

http://download-node-02.eng.bos.redhat.com/brewroot/work/tasks/6038/12426038/


I've tested it with QEMU standalone, replicating what I think is the workflow (snapshot w/mode=existing, change backing file to be relative, then block-commit).


Some notes:

There are limits to my approach with this fix, however.  For instance, if the relative pathname is in a different directory, like so:

# qemu-img info /912d9062-3881-479b-a6e5-7b074a252cb6/images/afed68f7-da67-4892-8e6f-7812d886e488/e5d9b44d-032f-4264-9591-283471902874 

image: /912d9062-3881-479b-a6e5-7b074a252cb6/images/afed68f7-da67-4892-8e6f-7812d886e488/e5d9b44d-032f-4264-9591-283471902874
file format: qcow2
virtual size: 500M (524288000 bytes)
disk size: 193K
cluster_size: 65536
backing file: ../67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/e5d9b44d-032f-4264-9591-138f757af14a


Then the full gluster URI would also need to be specified with the relative path for the full backing filename (note the relative path in the gluster URI), for the base image, e.g.:

gluster://192.168.15.180/gv0/912d9062-3881-479b-a6e5-7b074a252cb6/images/afed68f7-da67-4892-8e6f-7812d886e488/../67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/e5d9b44d-032f-4264-9591-138f757af14a


Is this how oVirt is specifying the URIs with relative pathnames in different directories?

(In reply to Nir Soffer from comment #30)
> I hope that we can make this work in the same way local images work.
> 
> File / block based image:
> 
>   qemu image path:
> /rhev/data-center/pool-uuid/dom-uuid/images/img-uuid/top-uuid
>   image backing file: backing-uuid
> 
> File and block are the same for ovirt. For block devices, we create a
> directory
> on the host with symlinks to the block devices:
> 
>   /rhev/data-center/mnt/blockSD/dom-uuid/images/img-uuid/top-uuid ->
>     /dev/dom-uuid/top-uuid
> 
> Network disk:
> 
>   qemu image path: gluster:gluster-name/dom-uuid/images/img-uuid/top-uuid
>   image backing file: backing-uuid
> 
> Hopefully qemu can treat the "backing-uuid" as a relative url, and access
> the backing file as:
> 
>   gluster:gluster-name/dom-uuid/images/img-uuid/backing-uuid
> 
> If it works for local (file: url), why not for gluster: urls?
> 
> Jeff, do you think this can work?

Nir,

I think so - I believe the test rpm I created for comment #31 should accomplish this (please see the caveat in the comment, however).

(In reply to Jeff Cody from comment #31)
> gluster://192.168.15.180/gv0/912d9062-3881-479b-a6e5-7b074a252cb6/images/
> afed68f7-da67-4892-8e6f-7812d886e488/../67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/
> e5d9b44d-032f-4264-9591-138f757af14a
> 
> 
> Is this how oVirt is specifying the URIs with relative pathnames in
> different directories?

On file storage *all* the volumes are on the same directory. For template volumes,
(used by many images), we create a hardlink to the template volume in the image
directory, so the path is always the volume uuid.

On block storage the symlinks simulate what we have on file storage, so again we
never refer to images in another directory.

Here is example chains for file storage:

# ls -lht /rhev/data-center/mnt/dumbo.tlv.redhat.com\:_voodoo_01/c86e8dab-444b-4c66-b004-f1768f12219b/images/
total 2.2M
-rw-rw----. 1 vdsm kvm 1.0M Jan 23 22:13 24703240-e081-48b2-a209-7244800f6b3d.lease
-rw-r--r--. 1 vdsm kvm  264 Jan 23 22:13 24703240-e081-48b2-a209-7244800f6b3d.meta
-rw-rw----. 1 vdsm kvm 193K Jan 23 22:13 24703240-e081-48b2-a209-7244800f6b3d
-rw-r--r--. 2 vdsm kvm  316 Jan 23 22:12 9c451979-ca52-4e3c-a702-8538b2cee644.meta
-rw-rw----. 2 vdsm kvm 1.0G Jan 23 22:11 9c451979-ca52-4e3c-a702-8538b2cee644
-rw-rw----. 2 vdsm kvm 1.0M Jan 23 22:11 9c451979-ca52-4e3c-a702-8538b2cee644.lease

# qemu-img info 24703240-e081-48b2-a209-7244800f6b3d
image: 24703240-e081-48b2-a209-7244800f6b3d
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 196K
cluster_size: 65536
backing file: 9c451979-ca52-4e3c-a702-8538b2cee644
backing file format: raw
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

Block storage:

Actual lvs:

# lvs 7df95b16-1bd3-4c23-bbbe-b21d403bdcd8
  LV                                   VG                                   Attr       LSize  
  24703240-e081-48b2-a209-7244800f6b3d 7df95b16-1bd3-4c23-bbbe-b21d403bdcd8 -wi-------   1.12g                                                    
  9c451979-ca52-4e3c-a702-8538b2cee644 7df95b16-1bd3-4c23-bbbe-b21d403bdcd8 -wi-------   1.00g                                                    
  ids                                  7df95b16-1bd3-4c23-bbbe-b21d403bdcd8 -wi-ao---- 128.00m                                                    
  inbox                                7df95b16-1bd3-4c23-bbbe-b21d403bdcd8 -wi-a----- 128.00m                                                    
  leases                               7df95b16-1bd3-4c23-bbbe-b21d403bdcd8 -wi-a-----   2.00g                                                    
  master                               7df95b16-1bd3-4c23-bbbe-b21d403bdcd8 -wi-a-----   1.00g                                                    
  metadata                             7df95b16-1bd3-4c23-bbbe-b21d403bdcd8 -wi-a----- 512.00m                                                    
  outbox                               7df95b16-1bd3-4c23-bbbe-b21d403bdcd8 -wi-a----- 128.00m                                                    
  xleases                              7df95b16-1bd3-4c23-bbbe-b21d403bdcd8 -wi-a-----   1.00g                                                    

Image (fake) directory:

# ls -lht /rhev/data-center/mnt/blockSD/7df95b16-1bd3-4c23-bbbe-b21d403bdcd8/images/f3198d8d-a42b-41d4-bfe7-ed0bfd4a54a2/
total 8.0K
lrwxrwxrwx. 1 vdsm kvm 78 Jan 23 23:01 24703240-e081-48b2-a209-7244800f6b3d -> /dev/7df95b16-1bd3-4c23-bbbe-b21d403bdcd8/24703240-e081-48b2-a209-7244800f6b3d
lrwxrwxrwx. 1 vdsm kvm 78 Jan 23 23:01 9c451979-ca52-4e3c-a702-8538b2cee644 -> /dev/7df95b16-1bd3-4c23-bbbe-b21d403bdcd8/9c451979-ca52-4e3c-a702-8538b2cee644

qemu-img:

# qemu-img info /rhev/data-center/mnt/blockSD/7df95b16-1bd3-4c23-bbbe-b21d403bdcd8/images/f3198d8d-a42b-41d4-bfe7-ed0bfd4a54a2/24703240-e081-48b2-a209-7244800f6b3d 
image: /rhev/data-center/mnt/blockSD/7df95b16-1bd3-4c23-bbbe-b21d403bdcd8/images/f3198d8d-a42b-41d4-bfe7-ed0bfd4a54a2/24703240-e081-48b2-a209-7244800f6b3d
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 0
cluster_size: 65536
backing file: 9c451979-ca52-4e3c-a702-8538b2cee644 (actual path: /rhev/data-center/mnt/blockSD/7df95b16-1bd3-4c23-bbbe-b21d403bdcd8/images/f3198d8d-a42b-41d4-bfe7-ed0bfd4a54a2/9c451979-ca52-4e3c-a702-8538b2cee644)
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

(In reply to Jeff Cody from comment #31)
> Could you please try the version in this brew build:
> 
> http://download-node-02.eng.bos.redhat.com/brewroot/work/tasks/6038/12426038/
> 

I tried with the builds from this location and the live merge flows work correctly

> 
> I've tested it with QEMU standalone, replicating what I think is the
> workflow (snapshot w/mode=existing, change backing file to be relative, then
> block-commit).
> 
> 
> Some notes:
> 
> There are limits to my approach with this fix, however.  For instance, if
> the relative pathname is in a different directory, like so:
> 
> # qemu-img info
> /912d9062-3881-479b-a6e5-7b074a252cb6/images/afed68f7-da67-4892-8e6f-
> 7812d886e488/e5d9b44d-032f-4264-9591-283471902874 
> 
> image:
> /912d9062-3881-479b-a6e5-7b074a252cb6/images/afed68f7-da67-4892-8e6f-
> 7812d886e488/e5d9b44d-032f-4264-9591-283471902874
> file format: qcow2
> virtual size: 500M (524288000 bytes)
> disk size: 193K
> cluster_size: 65536
> backing file:
> ../67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/e5d9b44d-032f-4264-9591-138f757af14a
> 
> 
> Then the full gluster URI would also need to be specified with the relative
> path for the full backing filename (note the relative path in the gluster
> URI), for the base image, e.g.:
> 
> gluster://192.168.15.180/gv0/912d9062-3881-479b-a6e5-7b074a252cb6/images/
> afed68f7-da67-4892-8e6f-7812d886e488/../67c8d32e-39b2-48cd-a1cd-1f0fe6d7c153/
> e5d9b44d-032f-4264-9591-138f757af14a
> 
> 
> Is this how oVirt is specifying the URIs with relative pathnames in
> different directories?

(In reply to Sahina Bose from comment #34)
> (In reply to Jeff Cody from comment #31)
> > Could you please try the version in this brew build:
> > 
> > http://download-node-02.eng.bos.redhat.com/brewroot/work/tasks/6038/12426038/
> > 
> 
> I tried with the builds from this location and the live merge flows work
> correctly
> 

Thanks Sahina.  I've sent the patch upstream to qemu-devel:
https://lists.gnu.org/archive/html/qemu-devel/2017-01/msg05520.html

The patch can also be viewed here: 
https://github.com/codyprime/qemu-kvm-jtc/commit/9a93d47ddba5f86d36cbdf884d86e2fa6f5542ae

Per comment 34 and comment 35, commit 418661e0324c1c419552cf24bd4447292e884bdd has fixed this issue.
Check on qemu-kvm-rhev-2.9.0-1.el7.x86_64:
# git log -p block.c
commit 418661e0324c1c419552cf24bd4447292e884bdd
Author: Jeff Cody <jcody>
Date:   Wed Jan 25 20:08:20 2017 -0500

    block: check full backing filename when searching protocol filenames
    
    In bdrv_find_backing_image(), if we are searching an image for a backing
    file that contains a protocol, we currently only compare unmodified
    paths.
    
    However, some management software will change the backing filename to be
    a relative filename in a path.  QEMU is able to handle this fine,
    because internally it will use path_combine to put together the full
    protocol URI.
    
    However, this can lead to an inability to match an image during a QAPI
    command that needs to use bdrv_find_backing_image() to find the image,
    when it is searched by the full URI.
    
    When searching for a protocol filename, if the straight comparison
    fails, this patch will also compare against the full backing filename to
    see if that is a match.
    
    Signed-off-by: Jeff Cody <jcody>
    Message-id: c2d025adca8a2b665189e6f4cf080f44126d0b6b.1485392617.git.jcody
    Reviewed-by: Max Reitz <mreitz>
    Signed-off-by: Max Reitz <mreitz>

diff --git a/block.c b/block.c
index 1dbc060..b8bc2a1 100644
--- a/block.c
+++ b/block.c
@@ -3145,6 +3145,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
     int is_protocol = 0;
     BlockDriverState *curr_bs = NULL;
     BlockDriverState *retval = NULL;
+    Error *local_error = NULL;
 
     if (!bs || !bs->drv || !backing_file) {
         return NULL;
@@ -3165,6 +3166,18 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
                 retval = curr_bs->backing->bs;
                 break;
             }
+            /* Also check against the full backing filename for the image */
+            bdrv_get_full_backing_filename(curr_bs, backing_file_full, PATH_MAX,
+                                           &local_error);
+            if (local_error == NULL) {
+                if (strcmp(backing_file, backing_file_full) == 0) {
+                    retval = curr_bs->backing->bs;
+                    break;
+                }
+            } else {
+                error_free(local_error);
+                local_error = NULL;
+            }
         } else {
             /* If not an absolute filename path, make it relative to the current
              * image's filename path */

So moving to VERIFIED.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392