RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1464821 - libvirt is unable to parse unflattened "server" structure in json backing file
Summary: libvirt is unable to parse unflattened "server" structure in json backing file
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: Han Han
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-06-26 01:18 UTC by Han Han
Modified: 2018-05-24 03:33 UTC (History)
7 users (show)

Fixed In Version: libvirt-3.7.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-10 10:50:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:0704 0 None None None 2018-04-10 10:52:02 UTC

Description Han Han 2017-06-26 01:18:57 UTC
Description of problem:
As subject

Version-Release number of selected component (if applicable):
libvirt-3.2.0-15.el7a.x86_64
qemu-kvm-rhev-2.9.0-14.el7.x86_64


How reproducible:
100%

Steps to Reproduce:
# qemu-img create -f qcow2 -b 'json:{"driver": "raw", "file": {"server.host": "10.66.5.92", "server.port": "7000", "tag": "", "driver": "sheepdog", "server.type": "inet", "vdi": "Alice"}}' /var/lib/libvirt/images/sheepdog.qcow2
Formatting '/var/lib/libvirt/images/sheepdog.qcow2', fmt=qcow2 size=5368709120 backing_file=json:{"driver": "raw",, "file": {"server.host": "10.66.5.92",, "server.port": "7000",, "tag": "",, "driver": "sheepdog",, "server.type": "inet",, "vdi": "Alice"}} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

# virsh pool-dumpxml default
<pool type='dir'>
  <name>default</name>
  <uuid>38795621-2398-4e87-8602-31ae5907db67</uuid>
  <capacity unit='bytes'>0</capacity>
  <allocation unit='bytes'>0</allocation>
  <available unit='bytes'>0</available>
  <source>
  </source>
  <target>
    <path>/var/lib/libvirt/images</path>
  </target>
</pool>

# virsh pool-refresh default
error: Failed to refresh pool default
error: invalid argument: missing remote server specification in JSON backing volume definition

Actual results:
As above

Expected results:
Libvirt can parse the unflattened "server" structure.

Additional info:
While the flattened "server" structure works well:
# qemu-img create -f qcow2 -b 'json:{"driver": "raw", "file": {"server":{"host": "10.66.5.92", "port": "7000", "type":"inet"}, "tag": "", "driver": "sheepdog", "vdi": "Alice"}}' /var/lib/libvirt/images/sheepdog.qcow2
Formatting '/var/lib/libvirt/images/sheepdog.qcow2', fmt=qcow2 size=5368709120 backing_file=json:{"driver": "raw",, "file": {"server":{"host": "10.66.5.92",, "port": "7000",, "type":"inet"},, "tag": "",, "driver": "sheepdog",, "vdi": "Alice"}} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
# virsh pool-refresh default
Pool default refreshed

It affects these protocols using 'SocketAddress' structure in qapi/block-core.json in qemu git repo, including nbd and sheepdog.

Comment 2 Han Han 2017-06-28 01:57:18 UTC
NBD example:
➜  ~ qemu-img create -f qcow2 -b 'json:{"file":{"driver":"nbd", "export":"nbd_raw","server":{"type":"inet", "port":"10803"}, "server.host":"xx.xx.xx.xx"}}' /var/lib/libvirt/images/nbd.qcow2                                                                                                        Formatting '/var/lib/libvirt/images/nbd.qcow2', fmt=qcow2 size=10737418240 backing_file=json:{"file":{"driver":"nbd",, "export":"nbd_raw",,"server":{"type":"inet",, "port":"10803"},, "server.host":"xx.xx.xx.xx"}} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
➜  ~ virsh pool-refresh defaulterror: Failed to refresh pool default
error: invalid argument: missing hostname for tcp backing server in JSON backing volume definition

Comment 3 Peter Krempa 2017-07-11 12:33:00 UTC
Fixed upstream by:

commit ffdf5323284b3846c949040397160415a718f6e0
Author: Peter Krempa <pkrempa>
Date:   Mon Jun 26 19:42:06 2017 +0200

    tests: Validate that JSON deflattening fixed nested json pseudo-protocol strings
    
    Sheepdog and possibly others use nested objects for network server and
    thus could be specified in a way that libvirt would not parse.
    
    Validates that https://bugzilla.redhat.com/show_bug.cgi?id=1464821
    is fixed properly.

commit 6d7cdec63df182338032a4998b3ccbf7101cf398
Author: Peter Krempa <pkrempa>
Date:   Tue Jul 11 14:23:08 2017 +0200

    util: storage: Always deflatten JSON pseudo-protocol objects
    
    Now that the JSON deflattener is working sanely we can always attempt
    the deflattening so that we can then parse the tree as expected.

commit 428d175206ebe12be39fcac730ecf509fb806154
Author: Peter Krempa <pkrempa>
Date:   Mon Jun 26 19:37:18 2017 +0200

    util: json: Recursively deflatten objects virJSONValueObjectDeflatten
    
    If a value of the first level object contains more objects needing
    deflattening which would be wrapped in an actual object the function
    would not recurse into them.
    
    By this simple addition we can fully deflatten the objects.

commit d40f4b3e673cda123644c32bc815e7e2f1fc8fcd
Author: Peter Krempa <pkrempa>
Date:   Tue Jun 27 13:48:56 2017 +0200

    util: json: Properly implement JSON deflattening
    
    As it turns out sometimes users pass in an arbitrarily nested structure
    e.g. for the qemu backing chains JSON pseudo protocol. This new
    implementation deflattens now a single object fully even with nested
    keys.
    
    Additionally it's not necessary now to stick with the "file." prefix for
    the properties.

commit 7f1209ad1e23f6beaaec77f18b9aa929e4bca4f7
Author: Peter Krempa <pkrempa>
Date:   Mon Jun 26 18:02:35 2017 +0200

    tests: json: Add test for the deflattening function
    
    Add a few test cases to verify that the old behaviour does not break and
    that new one behaves sanely.

commit f43b7d60d8e03c4b284a7217ea59c2aba5c2ca83
Author: Peter Krempa <pkrempa>
Date:   Mon Jun 26 18:42:07 2017 +0200

    util: json: Don't remove the 'file' subobject when deflattening
    
    Currently the function would deflatten the object by dropping the 'file'
    prefix from the attributes. This does not really scale well or adhere to
    the documentation.
    
    Until we refactor the worker to properly deflatten everything we at
    least simulate it by adding the "file" wrapper object back.

commit de75de7c977b222c3c33f5dc5268d9e07fa97b7e
Author: Peter Krempa <pkrempa>
Date:   Mon Jun 26 16:29:04 2017 +0200

    util: Move JSON object deflattening code to json utility file
    
    The code will become more universal so it makes more sense for it to
    live with the rest of the JSON functions.

Comment 5 Han Han 2017-11-09 06:00:12 UTC
Verify it on libvirt-3.9.0-1.el7.x86_64
1. Prepare sheepdog server according to https://github.com/sheepdog/sheepdog/wiki/Getting-Started

2. Create sheepdog disk
# qemu-img create  sheepdog:172.17.0.2:7000:Alice 5G
Formatting 'sheepdog:172.17.0.2:7000:Alice', fmt=raw size=5368709120

3. Prepare json backing metadata with different nested depth:
# qemu-img create -b 'json:{"driver":"raw","file":{"driver":"sheepdog","server":{"host":"172.17.0.2","port":"7000","type":"inet"},"tag":"","vdi":"Alice"}}' /var/lib/libvirt/images/sheep1.qocw2 -o backing_fmt=raw -f qcow2
Formatting '/var/lib/libvirt/images/sheep1.qocw2', fmt=qcow2 size=5368709120 backing_file=json:{"driver":"raw",,"file":{"driver":"sheepdog",,"server":{"host":"172.17.0.2",,"port":"7000",,"type":"inet"},,"tag":"",,"vdi":"Alice"}} backing_fmt=raw cluster_size=65536 lazy_refcounts=off refcount_bits=16

# qemu-img create -b 'json:{"driver":"raw","file.driver":"sheepdog","file.server":{"host":"172.17.0.2","port":"7000","type":"inet"},"file.tag":"","file.vdi":"Alice"}''' /var/lib/libvirt/images/sheep2.qocw2 -o backing_fmt=raw -f qcow2
Formatting '/var/lib/libvirt/images/sheep2.qocw2', fmt=qcow2 size=5368709120 backing_file=json:{"driver":"raw",,"file.driver":"sheepdog",,"file.server":{"host":"172.17.0.2",,"port":"7000",,"type":"inet"},,"file.tag":"",,"file.vdi":"Alice"} backing_fmt=raw cluster_size=65536 lazy_refcounts=off refcount_bits=16

# qemu-img create -b 'json:{"driver":"raw","file.driver":"sheepdog","file.server.host":"172.17.0.2","file.server.port":"7000","file.server.type":"inet","file.tag":"","file.vdi":"Alice"}' /var/lib/libvirt/images/sheep3.qocw2 -o backing_fmt=raw -f qcow2
Formatting '/var/lib/libvirt/images/sheep3.qocw2', fmt=qcow2 size=5368709120 backing_file=json:{"driver":"raw",,"file.driver":"sheepdog",,"file.server.host":"172.17.0.2",,"file.server.port":"7000",,"file.server.type":"inet",,"file.tag":"",,"file.vdi":"Alice"} backing_fmt=raw cluster_size=65536 lazy_refcounts=off refcount_bits=16

4. Refresh the pool of /var/lib/libvirt/images/
# virsh pool-dumpxml default
<pool type='dir'>
  <name>default</name>
  <uuid>03a38a8d-a7fc-4f6a-be13-b9c4b6935758</uuid>
  <capacity unit='bytes'>107347968000</capacity>
  <allocation unit='bytes'>37366534144</allocation>
  <available unit='bytes'>69981433856</available>
  <source>
  </source>
  <target>
    <path>/var/lib/libvirt/images</path>
    <permissions>
      <mode>0711</mode>
      <owner>0</owner>
      <group>0</group>
      <label>system_u:object_r:virt_image_t:s0</label>
    </permissions>
  </target>
</pool>

# virsh pool-refresh default                        
Pool default refreshed

# virsh vol-list default|grep sheep
 sheep1.qocw2         /var/lib/libvirt/images/sheep1.qocw2    
 sheep2.qocw2         /var/lib/libvirt/images/sheep2.qocw2    
 sheep3.qocw2         /var/lib/libvirt/images/sheep3.qocw2

No error when parsing json backing metadata. Bug fixed.

Comment 9 errata-xmlrpc 2018-04-10 10:50:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0704


Note You need to log in before you can comment on or make changes to this bug.