Bug 1457088

The bug blocks RHEL7.3's new feature BZ1134878 and RHEL7.4's new feature BZ1371758.

If changes are made to libvirt, there are implications for virt-v2v too.

This virt-v2v commit may need to be changed in RHEL 7.4:
https://github.com/libguestfs/libguestfs/commit/ecbf09385033ba0acf14e914927ec81ad20a5308

At the very least we'll need to retest to ensure the libvirt
change doesn't break anything.  If it narrowly only affects
Ceph then we may be fine.

It's probably since commit c7cacb3e that this doesn't work any more.

Of course, "file.filename" was never supposed to be the final interface, but in
theory I think bdrv_file_open() should handle this and parse the "filename" key
into individual options.

(In reply to Richard W.M. Jones from comment #4)
> If changes are made to libvirt, there are implications for virt-v2v too.
> 
> This virt-v2v commit may need to be changed in RHEL 7.4:
> https://github.com/libguestfs/libguestfs/commit/
> ecbf09385033ba0acf14e914927ec81ad20a5308
> 
> At the very least we'll need to retest to ensure the libvirt
> change doesn't break anything.  If it narrowly only affects
> Ceph then we may be fine.

It doesn't only affects ceph, but also iscsi and nbd.

As I test, only iscsi and rbd don't work. Nbd has no problem because it couldn't use file.filename in RHEL7.3.
For iscsi:
# qemu-img info iscsi.qcow2 
image: iscsi.qcow2
file format: qcow2
virtual size: 30G (32212254720 bytes)
disk size: 196K
cluster_size: 65536
backing file: json:{"file.driver":"iscsi", "file.filename":"iscsi://10.66.5.92:3260/iqn.2016-12.com.virttest:emulated-iscsi-noauth.target0/0"}
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

# /usr/libexec/qemu-kvm iscsi.qcow2
qemu-kvm: Could not open backing file: Need all of transport, portal and target options

I have patches that will parse a filename parameter, if present.  Currently, I give precedence to the 'filename' encoded options over passed options.  I am wondering if that is the opposite of how it should be, however.  I'll send patches upstream, and see what qemu-devel has to say.

RFC patches posted: https://lists.nongnu.org/archive/html/qemu-devel/2017-06/msg02724.html

Fix included in qemu-kvm-rhev-2.9.0-12.el7

1. ceph

Reproduce with 

qemu-kvm-rhev-2.9.0-10.el7.x86_64

1). Create snapshot with qemu-img-rhev-2.6.0-28.el7_3.2.x86_64.rpm

# qemu-img create ceph.img -b 'json:{"file.driver":"rbd","file.filename":"rbd:kvm-test/rhel74-virtio.raw:mon_host=10.73.199.64"}' -f qcow2
Formatting 'ceph.img', fmt=qcow2 size=21474836480 backing_file=json:{"file.driver":"rbd",,"file.filename":"rbd:kvm-test/rhel74-virtio.raw:mon_host=10.73.199.64"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

2). boot up snapshot with qemu-kvm-rhev-2.9.0-10.el7.x86_64

qemu-kvm: -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=ceph.img: Could not open backing file: Parameters 'pool' and 'image' are required



Fixed with qemu-kvm-rhev-2.9.0-12.el7.x86_64

1). boot up and login snapshot successfully


cmd:
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=ceph.img \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=03 \

(In reply to Suqin Huang from comment #16)
> 1. ceph
> 
> Reproduce with 
> 
> qemu-kvm-rhev-2.9.0-10.el7.x86_64
> 
> 1). Create snapshot with qemu-img-rhev-2.6.0-28.el7_3.2.x86_64.rpm
> 
> # qemu-img create ceph.img -b
> 'json:{"file.driver":"rbd","file.filename":"rbd:kvm-test/rhel74-virtio.raw:
> mon_host=10.73.199.64"}' -f qcow2
> Formatting 'ceph.img', fmt=qcow2 size=21474836480
> backing_file=json:{"file.driver":"rbd",,"file.filename":"rbd:kvm-test/rhel74-
> virtio.raw:mon_host=10.73.199.64"} encryption=off cluster_size=65536
> lazy_refcounts=off refcount_bits=16
> 
> 2). boot up snapshot with qemu-kvm-rhev-2.9.0-10.el7.x86_64
> 
> qemu-kvm: -drive
> id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,
> file=ceph.img: Could not open backing file: Parameters 'pool' and 'image'
> are required
> 
> 
> 
> Fixed with qemu-kvm-rhev-2.9.0-12.el7.x86_64
> 
> 1). boot up and login snapshot successfully
> 
> 
> cmd:
>     -drive
> id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,
> file=ceph.img \
>     -device
> virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=03 \

Pls try gluster with inet type as https://bugzilla.redhat.com/show_bug.cgi?id=1451557#c3 says. This element is added since qemu-2.9, too.

2. iscsi

1). reproduce with qemu-kvm-rhev-2.9.0-10.el7.x86_64

qemu-kvm: -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=iscsi.img: Could not open backing file: iSCSI: Failed to connect to LUN : Failed to log in to target

2). Fixed in qemu-kvm-rhev-2.9.0-12.el7.x86_64

boot up and login snapshot successfully

> Pls try gluster with inet type as
> https://bugzilla.redhat.com/show_bug.cgi?id=1451557#c3 says. This element is
> added since qemu-2.9, too.

Hi Hanhan,

is it qemu issue? I could not reproduce with qemu-kvm-rhev-2.9.0-10.el7.x86_64

1. Create snapshot 

#qemu-img create -f qcow2 gluster.img -b 'json:{"file.driver":"gluster","file.volume":"gv0","file.path":"rhel74-64-virtio.qcow2","file.logfile":"/tmp/log","file.debug":9,"file.server":[ {"type":"inet","host":"10.73.199.200","port":"24007"}]}' 

Formatting 'gluster.img', fmt=qcow2 size=21474836480 backing_file=json:{"file.driver":"gluster",,"file.volume":"gv0",,"file.path":"rhel74-64-virtio.qcow2",,"file.logfile":"/tmp/log",,"file.debug":9,,"file.server":[ {"type":"inet",,"host":"10.73.199.200",,"port":"24007"}]} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

2. boot up and login the snapshot successfully

    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=gluster.img \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=03 \

According to comment16, comment18 and comment19, update bug status to VERIFIED

(In reply to Suqin Huang from comment #19)
> > Pls try gluster with inet type as
> > https://bugzilla.redhat.com/show_bug.cgi?id=1451557#c3 says. This element is
> > added since qemu-2.9, too.
> 
> Hi Hanhan,
> 
> is it qemu issue? I could not reproduce with
> qemu-kvm-rhev-2.9.0-10.el7.x86_64
> 
> 1. Create snapshot 
> 
> #qemu-img create -f qcow2 gluster.img -b
> 'json:{"file.driver":"gluster","file.volume":"gv0","file.path":"rhel74-64-
> virtio.qcow2","file.logfile":"/tmp/log","file.debug":9,"file.server":[
> {"type":"inet","host":"10.73.199.200","port":"24007"}]}' 
> 
> Formatting 'gluster.img', fmt=qcow2 size=21474836480
> backing_file=json:{"file.driver":"gluster",,"file.volume":"gv0",,"file.path":
> "rhel74-64-virtio.qcow2",,"file.logfile":"/tmp/log",,"file.debug":9,,"file.
> server":[ {"type":"inet",,"host":"10.73.199.200",,"port":"24007"}]}
> encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
> 
> 2. boot up and login the snapshot successfully
> 
>     -drive
> id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,
> file=gluster.img \
>     -device
> virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=03 \

I am sorry. I misunderstood it. It is not a qemu issue.I should try it in libvirt.

Hi Jeff,
When use sheepdog url format:
# qemu-img create -f qcow2 -b 'json:{"file.driver":"sheepdog","file.filename":"sheepdog://10.66.5.92:7000/Alice"}' /var/lib/libvirt/images/sheepdog.qcow2
qemu-img: /var/lib/libvirt/images/sheepdog.qcow2: Parameter 'type' is missing

When using the new format:
# qemu-img create -f qcow2 -b 'json:{"driver": "raw", "file": {"server.host": "10.66.5.92", "server.port": "7000", "tag": "", "driver": "sheepdog", "server.type": "inet", "vdi": "Alice"}}' /var/lib/libvirt/images/sheepdog.qcow2
Formatting '/var/lib/libvirt/images/sheepdog.qcow2', fmt=qcow2 size=5368709120 backing_file=json:{"driver": "raw",, "file": {"server.host": "10.66.5.92",, "server.port": "7000",, "tag": "",, "driver": "sheepdog",, "server.type": "inet",, "vdi": "Alice"}} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

# virsh pool-dumpxml default      
<pool type='dir'>
  <name>default</name>
  <uuid>38795621-2398-4e87-8602-31ae5907db67</uuid>
  <capacity unit='bytes'>0</capacity>
  <allocation unit='bytes'>0</allocation>
  <available unit='bytes'>0</available>
  <source>
  </source>
  <target>
    <path>/var/lib/libvirt/images</path>
  </target>
</pool>

# virsh pool-start default
error: Failed to start pool default
error: invalid argument: missing remote server specification in JSON backing volume definition

I think qemu-img should support sheepdog url format because libvirt in RHEL7.4 cannot parse the new sheepdog format.
What's your idea?

Han Han, sheepdog is not supported in RHEL and therefore is not in the whitelist.

(In reply to Ademar Reis from comment #23)
> Han Han, sheepdog is not supported in RHEL and therefore is not in the
> whitelist.

While not in the whitelist, qemu-img does not use the whitelist and can create / modify all image formats.  However, the use of 'file.filename' in the json is not supported, and it was a bug that it ever actually worked.

Since rbd and iscsi are supported in RHEV, there is a workaround so that existing images could be opened per this BZ, but that is not needed as far as RHEL/RHEV is concerned.

Upstream, sheepdog should probably get the same fix that rbd/iscsi received.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Hi, jeff, I recently used the format of 7.3 and found out:
#  qemu-img create test.qcow2 -b 'json:{"file.driver":"rbd","file.filename":"rbd:rbd/data.raw"}' -f qcow2
qemu-img: test.qcow2: Parameter 'pool' is missing
Could not open backing image to determine size.

is this format obsolete now? If so, I will modify current test cases of rbd.

# qemu-img --version
qemu-img version 2.6.0 (qemu-kvm-rhev-2.6.0-28.el7_3.14), Copyright (c) 2004-2008 Fabrice Bellard
# qemu-img create -f qcow2 -b 'json:{"file.driver":"rbd","file.filename":"rbd:rbd/test.raw"}' test.qcow2
Formatting 'test.qcow2', fmt=qcow2 size=21474836480 backing_file=json:{"file.driver":"rbd",,"file.filename":"rbd:rbd/test.raw"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
# qemu-img info test.qcow2
image: test.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 196K
cluster_size: 65536
backing file: json:{"file.driver":"rbd","file.filename":"rbd:rbd/test.raw"}
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

upgrade to latest qemu-kvm
# qemu-img --version
qemu-img version 2.12.0 (qemu-kvm-rhev-2.12.0-7.el7)
# qemu-img check test.qcow2
qemu-img: Could not open 'test.qcow2': Could not open backing file: Parameter 'pool' is missing

I believe this issue is regressed.

Support for "filename" was dropped upstream in commit bb9f762ff35.  It's been deprecated since commit 91589d9e5ca, v2.10.0.

We brought it back downstream in commit 91589d9e5ca.  However, bdrv_file_open() seems to ignore it anyways.  @parse_filename remains false.

Jeff, please have a closer look.  If this indeed a regression (seems likely), update BZ status accordingly.

(In reply to Markus Armbruster from comment #29)

> Jeff, please have a closer look.  If this indeed a regression (seems
> likely), update BZ status accordingly.

I think it is as well.  I've gone ahead and taken BZ #1610605 for this issue.