Bug 1134878

Summary: libvirt reports json "backing file" is missing
Product: Red Hat Enterprise Linux 7 Reporter: Richard W.M. Jones <rjones>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.1CC: amureini, bmcclain, dyuan, eblake, hhan, jsuchane, libvirt-maint, mzhan, pkrempa, rbalakri, tnisan, tzheng, xuzhang, yanyang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-2.0.0-4.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1134592 Environment:
Last Closed: 2016-11-03 18:09:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1134592, 1367260, 1457088    
Bug Blocks: 910269, 1254282, 1305606, 1251748    

Description Richard W.M. Jones 2014-08-28 11:51:03 UTC
+++ This bug was initially created as a clone of Bug #1134592 +++

Description of problem:

$ qemu-img create -f qcow2 -b 'json: { "file.driver":"ssh", "file.user":"root", "file.host":"avon.home.annexia.org", "file.path":"/vmfs/volumes/datastore1/Fedora 20/Fedora 20-flat.vmdk", "file.host_key_check":"no" }' /tmp/foo

qemu-img file is successful, but:

$ guestfish -a /tmp/foo -i
[...]
libvirt: invalid argument: Backing file 'json: { "file.driver":"ssh", "file.user":"root", "file.host":"avon.home.annexia.org", "file.path":"/vmfs/volumes/datastore1/Fedora 20/Fedora 20-flat.vmdk", "file.host_key_check":"no" }' of image '/tmp/foo' is missing. [code=8 domain=10]

This is presumably because libvirt has trouble parsing the
new qemu 2.1.0 "json:" specification.  TBH I think it shouldn't
even try, it should just ignore them.  Certainly it shouldn't
be giving an error here.

Version-Release number of selected component (if applicable):

libvirt-1.1.3.5-2.fc20.x86_64

How reproducible:

100% - see above.

--- Additional comment from Eric Blake on 2014-08-27 17:37:18 EDT ---

More generically, there are two problems here:
1. Libvirt needs to catch up on the additional formats supported by upstream qemu. (This is one case where a qemu library that parses qcow2 files would be nice - if only it weren't for the fact that no such library exists in part because current qemu aborts on OOM which is inappropriate for library code)

2. Libvirt needs to have sane behavior when a backing format looks like a protocol but for which libvirt does not (yet) know the protocol.  It may still be an error message, but the message should be more like 'json: protocol not yet handled by libvirt', not 'stat of file "json: ..." failed'.  In other words, libvirt should NEVER attempt to stat() any protocol, known or unknown.

--- Additional comment from Richard W.M. Jones on 2014-08-28 07:50:39 EDT ---

I'm going to clone this for RHEL 7.1 since we likely will need it
fixing for virt-v2v access to ESX servers.

Note I really think that if libvirt doesn't know about a protocol
it should NOT error.  It should assume the best.

Comment 2 Richard W.M. Jones 2014-09-09 11:33:17 UTC
How to test:

$ cd /tmp
$ truncate -s 1G test1.img
$ qemu-img create -f qcow2 \
  -b 'json: { "file.driver":"file", "file.filename":"/tmp/test1.img" }' \
  test2.img
$ guestfish -a test2.img run

The original libvirt will give an error like this:

invalid argument: Backing file 'json: { "file.driver":"file", "file.filename":"/tmp/test1.img" }' of image '/tmp/test2.img' is missing. [code=8 domain=10]

Comment 3 Peter Krempa 2014-09-09 15:06:09 UTC
(In reply to Richard W.M. Jones from comment #0)
> +++ This bug was initially created as a clone of Bug #1134592 +++
> 
> Description of problem:
> 
> $ qemu-img create -f qcow2 -b 'json: { "file.driver":"ssh",
> "file.user":"root", "file.host":"avon.home.annexia.org",
> "file.path":"/vmfs/volumes/datastore1/Fedora 20/Fedora 20-flat.vmdk",
> "file.host_key_check":"no" }' /tmp/foo
> 
> qemu-img file is successful, but:
> 
> $ guestfish -a /tmp/foo -i
> [...]
> libvirt: invalid argument: Backing file 'json: { "file.driver":"ssh",
> "file.user":"root", "file.host":"avon.home.annexia.org",
> "file.path":"/vmfs/volumes/datastore1/Fedora 20/Fedora 20-flat.vmdk",
> "file.host_key_check":"no" }' of image '/tmp/foo' is missing. [code=8
> domain=10]
> 
> This is presumably because libvirt has trouble parsing the
> new qemu 2.1.0 "json:" specification.  TBH I think it shouldn't
> even try, it should just ignore them.  Certainly it shouldn't
> be giving an error here.

Upstream's stance on this is that if libvirt doesn't know how to parse a backing file specification we should error out rather than silently accept it. 

> 
> Version-Release number of selected component (if applicable):
> 
> libvirt-1.1.3.5-2.fc20.x86_64
> 
> How reproducible:
> 
> 100% - see above.
> 
> --- Additional comment from Eric Blake on 2014-08-27 17:37:18 EDT ---
> 
> More generically, there are two problems here:
> 1. Libvirt needs to catch up on the additional formats supported by upstream
> qemu. (This is one case where a qemu library that parses qcow2 files would
> be nice - if only it weren't for the fact that no such library exists in
> part because current qemu aborts on OOM which is inappropriate for library
> code)

Eric: Is the JSON backend file format stable enough to actually try to implement it?

If we implement the parser we will also need to implement support for "ssh" network protocol so that we'll be able to parse the protocol and create a source structure describing it. This should bypass all the other checks as the backing file will be known.

Without implementation of ssh storage driver we won't be able to make sure that the remote files have correct permissions though, but we have prior art in this.

> 
> 2. Libvirt needs to have sane behavior when a backing format looks like a
> protocol but for which libvirt does not (yet) know the protocol.  It may
> still be an error message, but the message should be more like 'json:
> protocol not yet handled by libvirt', not 'stat of file "json: ..." failed'.
> In other words, libvirt should NEVER attempt to stat() any protocol, known
> or unknown.

The error message seems sufficient IMHO:
invalid argument: Backing file 'json: { "file.driver":"file", "file.filename":"/tmp/test1.img" }' of image '/tmp/test2.img' is missing.

Comment 4 Eric Blake 2014-09-09 15:18:26 UTC
(In reply to Peter Krempa from comment #3)

> > This is presumably because libvirt has trouble parsing the
> > new qemu 2.1.0 "json:" specification.  TBH I think it shouldn't
> > even try, it should just ignore them.  Certainly it shouldn't
> > be giving an error here.
> 
> Upstream's stance on this is that if libvirt doesn't know how to parse a
> backing file specification we should error out rather than silently accept
> it. 

In the past, upstream has historically treated unknown strings as raw, and handed it to qemu with hopes for the best; but this is not very wise to do (if the string is not actually raw, qemu will be acting on different files than what libvirt expects), so I agree with the stance that libvirt SHOULD be complaining loudly on unknown content so that we then know what needs patching for support.

> > More generically, there are two problems here:
> > 1. Libvirt needs to catch up on the additional formats supported by upstream
> > qemu. (This is one case where a qemu library that parses qcow2 files would
> > be nice - if only it weren't for the fact that no such library exists in
> > part because current qemu aborts on OOM which is inappropriate for library
> > code)
> 
> Eric: Is the JSON backend file format stable enough to actually try to
> implement it?

Yes, the json: format is a stable interface that wraps the QMP command blockdev-add.  It doesn't yet describe all possible block devices that are possible on the command line or hotplug (hence we still have to use HMP fallbacks for hotplug of some block device backends), but for the formats that are supported, the API is stable and strongly typed, with a promise of backwards compatibility.  And I've been reviewing the qemu patches that implement various block drivers to make sure it stays stable.

At any rate, figuring out how to support json: format will be a net win for eventually switching over to QMP hotplug - since we need JSON descriptions of block devices for both reasons (backing files and hotplug), it is shared code that will eventually be reused.

> 
> If we implement the parser we will also need to implement support for "ssh"
> network protocol so that we'll be able to parse the protocol and create a
> source structure describing it. This should bypass all the other checks as
> the backing file will be known.
> 
> Without implementation of ssh storage driver we won't be able to make sure
> that the remote files have correct permissions though, but we have prior art
> in this.

Correct - to date, we have worried about expressing protocols, but not worried about security on the remote side of such protocols.

Comment 8 Richard W.M. Jones 2015-07-14 09:04:23 UTC
Is there a plan to fix this?  Currently virt-v2v prints out an
error naming bug 1134592 and telling people to not use libvirt
as a workaround.

$ virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest
[   0.0] Opening the source -i libvirt -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest
virt-v2v: error: because of libvirt bug 
https://bugzilla.redhat.com/show_bug.cgi?id=1134592 you must set this 
environment variable:

export LIBGUESTFS_BACKEND=direct

and then rerun the virt-v2v command.

Comment 11 Cole Robinson 2016-04-10 19:24:11 UTC
*** Bug 1134592 has been marked as a duplicate of this bug. ***

Comment 15 Han Han 2016-08-18 05:16:02 UTC
Hi Peter,
I noticed that rbd protocol was not added in the pacthes and libvirt-2.0.0-5.el7.x86_64 is not support json backimg from rbd:
# qemu-img create -f qcow2 -b 'json: {"file.driver":"rbd", "file.filename": "rbd:libvirt-pool/vol1:mon_host=10.73.75.52"}' /var/lib/libvirt/images/rbd_json.img
Formatting '/var/lib/libvirt/images/rbd_json.img', fmt=qcow2 size=5368709120 backing_file=json: {"file.driver":"rbd",, "file.filename": "rbd:libvirt-pool/vol1:mon_host=10.73.75.52"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

# virsh attach-disk vm1 /var/lib/libvirt/images/rbd_json.img vdc --subdriver qcow2
Failed to attach disk
internal error: missing parser implementation for JSON backing volume driver 'rbd'

Will we add rbd json backing support? I think it is necessary.

Comment 16 Han Han 2016-08-31 03:11:27 UTC
(In reply to Han Han from comment #15)
> Hi Peter,
> I noticed that rbd protocol was not added in the pacthes and
> libvirt-2.0.0-5.el7.x86_64 is not support json backimg from rbd:
> # qemu-img create -f qcow2 -b 'json: {"file.driver":"rbd", "file.filename":
> "rbd:libvirt-pool/vol1:mon_host=10.73.75.52"}'
> /var/lib/libvirt/images/rbd_json.img
> Formatting '/var/lib/libvirt/images/rbd_json.img', fmt=qcow2 size=5368709120
> backing_file=json: {"file.driver":"rbd",, "file.filename":
> "rbd:libvirt-pool/vol1:mon_host=10.73.75.52"} encryption=off
> cluster_size=65536 lazy_refcounts=off refcount_bits=16
> 
> # virsh attach-disk vm1 /var/lib/libvirt/images/rbd_json.img vdc --subdriver
> qcow2
> Failed to attach disk
> internal error: missing parser implementation for JSON backing volume driver
> 'rbd'
> 
> Will we add rbd json backing support? I think it is necessary.

New a bug BZ1371758 to track the rbd json backing support.

Comment 17 Han Han 2016-09-02 02:25:48 UTC
hi Peter, the bug BZ1372251 may be a test blocker for this bug. Do we need to do attach/detach the json images when verifying?

Comment 18 Peter Krempa 2016-09-05 16:52:23 UTC
attach/detach should work properly with other types of images. BZ1372251 tracks a very specific issue with gluster images, whereas attach/detach should work properly with different types of images.

Comment 22 Ademar Reis 2016-09-07 17:22:05 UTC
See the list of bugs this one "depends on". The QEMU corner cases are: Bug 1365708, Bug 1373710 and Bug 1373786.

Comment 23 Ademar Reis 2016-09-07 17:31:29 UTC
(In reply to Ademar Reis from comment #22)
> See the list of bugs this one "depends on". The QEMU corner cases are: Bug
> 1365708, Bug 1373710 and Bug 1373786.

I'm removing them from the "depends on" list and also removing their TestBlocker keywords. As Peter suggested, they should be handled as standlone issues and should not block the verification of this BZ.

The only BZ currently marked as TestBlocker is the one about glusterfs via json. Which doesn't appear to be a v2v use-case.

Comment 24 Richard W.M. Jones 2016-09-07 18:04:22 UTC
(In reply to Ademar Reis from comment #23)
> (In reply to Ademar Reis from comment #22)
> > See the list of bugs this one "depends on". The QEMU corner cases are: Bug
> > 1365708, Bug 1373710 and Bug 1373786.

Yes I seem to remember we saw bug 1365708, but passing the right type
in the json fixed that.  Don't know why I didn't file a BZ, perhaps
I forgot to or assumed it was expected behavious.  The other two
wouldn't affect virt-v2v.

> I'm removing them from the "depends on" list and also removing their
> TestBlocker keywords. As Peter suggested, they should be handled as
> standlone issues and should not block the verification of this BZ.
> 
> The only BZ currently marked as TestBlocker is the one about glusterfs via
> json. Which doesn't appear to be a v2v use-case.

Correct, glusterfs isn't be used by virt-v2v.

Comment 25 Han Han 2016-09-09 06:57:22 UTC
Hi Peter,
From the discussions above, we know that the BZ1365708, BZ1373710, BZ1373786 will not block the bug. 
So I will verify the bug with following scenarios´╝Ü
1. For file, host_device, https, gluster(tcp), iscsi, nbd, rbd protocols, they are fully supported by libvirt and qemu-kvm-rhev. I will test them with pool-refresh, hot-plug/hot-unplug

2. For host_cdrom, http, ftp, ftps, sheepdog, they are not whitelisted by qemu-kvm program and ssh is no supported by libvirt when attaching. So I will test them with pool-refesh only. And I will create a libvirt TestOnly bug to track these protocols.

Do you have any suggestion about the bug verify?

Comment 26 Han Han 2016-09-09 06:59:16 UTC
Sorry forget to add needinfo for the comment25

Comment 27 Peter Krempa 2016-09-09 07:07:24 UTC
Yes, that looks reasonable to me. A image that has a SSH backing disk can be hotplugged though since libvirt should be able to parse it.

Comment 28 Han Han 2016-09-09 07:17:17 UTC
Due to BZ1140166, ssh json backing disk cannot be hotplugged. So I will only use pool-refresh to check if libvirt can parse it.

Comment 29 Han Han 2016-09-09 09:55:39 UTC
Verify it on libvirt-2.0.0-8.el7.x86_64 qemu-kvm-rhev-2.6.0-22.el7.x86_64

1. Prepare a running VM
# virsh list 
 Id    Name                           State
----------------------------------------------------
 36    V                              running


2. Test on fully supported protocols(by attach/detach, read/write, pool-refresh)
File protocol:
# qemu-img create -f qcow2 -b 'json:{"file.driver":"file", "file.filename":"/tmp/a"}' /var/lib/libvirt/images/file.img
Formatting '/var/lib/libvirt/images/file.img', fmt=qcow2 size=104857600 backing_file=json:{"file.driver":"file",, "file.filename":"/tmp/a"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
# virsh attach-disk V  /var/lib/libvirt/images/file.img vdb --subdriver qcow2                                                                                                
Disk attached successfully
Do some read/write operation on vdb, it works well.
# virsh detach-disk V  vdb                                                   
Disk detached successfully

Host_device protocol:
# qemu-img create -f qcow2 -b 'json:{"file.driver":"host_device", "file.filename":"/dev/sdb"}' /var/lib/libvirt/images/host_device.img
Formatting '/var/lib/libvirt/images/host_device.img', fmt=qcow2 size=10737418240 backing_file=json:{"file.driver":"host_device",, "file.filename":"/dev/sdb"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
# virsh attach-disk V  /var/lib/libvirt/images/host_device.img vdb --subdriver qcow2                                                                                                 
Disk attached successfully
Do some read/write operation on vdb, it works well.
# virsh detach-disk V  vdb                                                          
Disk detached successfully

Https protocol:
Since we use self signed certificate in https, so we use file.sslverify:off to skip the cert-insecure error.
# qemu-img create -f qcow2 -b 'json:{"file.driver":"https", "file.url":"https://lab.work.me/https", "file.sslverify":"off"}' /var/lib/libvirt/images/https.img
Formatting '/var/lib/libvirt/images/https.img', fmt=qcow2 size=524288000 backing_file=json:{"file.driver":"https",, "file.url":"https://lab.work.me/https",, "file.sslverify":"off"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
# virsh attach-disk V  /var/lib/libvirt/images/https.img vdb --subdriver qcow2                                                                                                       
Disk attached successfully
Do some read/write operation on vdb, it works well.
# virsh detach-disk V  vdb                                                    
Disk detached successfully

Glusterfs protocol:
Use glusterfs url format:
# qemu-img create -f qcow2 -b 'json:{"file.driver":"gluster", "file.filename":"gluster://10.66.4.233/gluster-vol1/V"}' /var/lib/libvirt/images/gluster_file.img
Formatting '/var/lib/libvirt/images/gluster_file.img', fmt=qcow2 size=524288000 backing_file=json:{"file.driver":"gluster",, "file.filename":"gluster://10.66.4.233/gluster-vol1/V"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
# virsh attach-disk V  /var/lib/libvirt/images/gluster_file.img vdb --subdriver qcow2                                                                                                
Disk attached successfully
Do some read/write operation on vdb, it works well.
# virsh detach-disk V  vdb                                                    
Disk detached successfully

Iscsi protocol:
# qemu-img create -f qcow2 -b 'json:{"file.driver":"iscsi", "file.filename":"iscsi://10.66.7.179:3260/iqn.2015-05.com.redhat:libvirt.rhevm-test-group3-2/1"}' /var/lib/libvirt/images/iscsi_file.img
Formatting '/var/lib/libvirt/images/iscsi_file.img', fmt=qcow2 size=21474836480 backing_file=json:{"file.driver":"iscsi",, "file.filename":"iscsi://10.66.7.179:3260/iqn.2015-05.com.redhat:libvirt.rhevm-test-group3-2/1"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
# virsh attach-disk V  /var/lib/libvirt/images/iscsi_file.img vdb --subdriver qcow2             
Disk attached successfully
Do some read/write operation on vdb, it works well.
# virsh detach-disk V  vdb                                                    
Disk detached successfully	

Nbd protocol:
# qemu-img create -f qcow2 -b 'json:{"file.driver":"nbd", "file.host":"10.66.6.236", "file.port":"8888"}' /var/lib/libvirt/images/nbd.img
Formatting '/var/lib/libvirt/images/nbd.img', fmt=qcow2 size=524288000 backing_file=json:{"file.driver":"nbd",, "file.host":"10.66.6.236",, "file.port":"8888"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
# virsh attach-disk V  /var/lib/libvirt/images/nbd.img vdb --subdriver qcow2                                                                                                         
Disk attached successfully
Do some read/write operation on vdb, it works well.
# virsh detach-disk V  vdb                                                    
Disk detached successfully	

Pool refresh:
# virsh pool-refresh default
Pool default refreshed


3. Test on partly supported protocols(via pool-refresh)
Create host_cdrom backing image:
# qemu-img create -f qcow2 -b 'json:{"file.driver":"host_cdrom", "file.filename":"/dev/sr0"}' /var/lib/libvirt/images/host_cdrom.img
Formatting '/var/lib/libvirt/images/host_cdrom.img', fmt=qcow2 size=3743416320 backing_file=json:{"file.driver":"host_cdrom",, "file.filename":"/dev/sr0"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

Create http backing image:
# qemu-img create -f qcow2 -b 'json:{"file.driver":"http", "file.url":"http://10.66.6.236:81/tune.qcow2"}' /var/lib/libvirt/images/http.img
Formatting '/var/lib/libvirt/images/http.img', fmt=qcow2 size=104857600 backing_file=json:{"file.driver":"http",, "file.url":"http://10.66.6.236:81/tune.qcow2"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

Create ssh backing image:
# qemu-img create -f qcow2 -b 'json:{"file.driver":"ssh", "file.host":"10.66.6.236", "file.path":"/tmp/ssh", "file.user":"root", "file.port":22, "file.host_key_check":"no"}' /var/lib/libvirt/images/ssh.img
Formatting '/var/lib/libvirt/images/ssh.img', fmt=qcow2 size=104857600 backing_file=json:{"file.driver":"ssh",, "file.host":"10.66.6.236",, "file.path":"/tmp/ssh",, "file.user":"root",, "file.port":22,, "file.host_key_check":"no"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

Create sheepdog backing image:
# qemu-img create -f qcow2 -b 'json:{"file.driver":"sheepdog", "file.filename":"sheepdog://10.66.4.170:7000/Alice"}' /var/lib/libvirt/images/sheepdog.img
Formatting '/var/lib/libvirt/images/sheepdog.img', fmt=qcow2 size=5368709120 backing_file=json:{"file.driver":"sheepdog",, "file.filename":"sheepdog://10.66.4.170:7000/Alice"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

Do pool-refresh:
# virsh pool-refresh default        
Pool default refreshed

# virsh vol-list default|grep img
 file.img             /var/lib/libvirt/images/file.img        
 gluster.img          /var/lib/libvirt/images/gluster.img     
 gluster_file.img     /var/lib/libvirt/images/gluster_file.img
 gluster_multi.img    /var/lib/libvirt/images/gluster_multi.img
 gluster_socket.img   /var/lib/libvirt/images/gluster_socket.img
 host_cdrom.img       /var/lib/libvirt/images/host_cdrom.img  
 host_device.img      /var/lib/libvirt/images/host_device.img 
 http.img             /var/lib/libvirt/images/http.img        
 https.img            /var/lib/libvirt/images/https.img       
 iscsi_file.img       /var/lib/libvirt/images/iscsi_file.img  
 nbd.img              /var/lib/libvirt/images/nbd.img         
 sheepdog.img         /var/lib/libvirt/images/sheepdog.img    
 ssh.img              /var/lib/libvirt/images/ssh.img         

All json backing image can be parsed.

Comment 31 errata-xmlrpc 2016-11-03 18:09:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html