1509110 – backingStore element disappear after restart libvirtd

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1509110 - backingStore element disappear after restart libvirtd

Summary: backingStore element disappear after restart libvirtd

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	7.5
Hardware:	All
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Peter Krempa
QA Contact:	Meina Li
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-11-03 05:58 UTC by Junxiang Li
Modified:	2018-04-10 11:00 UTC (History)
CC List:	7 users (show)
Fixed In Version:	libvirt-3.9.0-5.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-04-10 10:59:09 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHEA-2018:0704	0	None	None	None	2018-04-10 11:00:18 UTC

Description Junxiang Li 2017-11-03 05:58:17 UTC

Description of problem:
backingStore element disappear after restart libvirtd

Version-Release number of selected component (if applicable):
Both on x86_64 and ppc64le, and the example is on ppc64le
# rpm -q kernel qemu-kvm-rhev libvirt
kernel-3.10.0-734.el7.ppc64le
qemu-kvm-rhev-2.10.0-3.el7.ppc64le
libvirt-3.9.0-1.el7.ppc64le

How reproducible:
100%

Steps to Reproduce:
1.prepare a domain xml:
<domain type='kvm'>
  <name>avocado-vt-vm1</name>
  <uuid>d3370c29-64e5-4104-8131-3342ea422374</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='ppc64le' machine='pseries-rhel7.5.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/avocado/data/avocado-vt/images/jeos-25-64.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'>
      <model name='spapr-pci-host-bridge'/>
      <target index='0'/>
    </controller>
    <controller type='usb' index='0' model='qemu-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:e5:30:c9'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
      <address type='spapr-vio' reg='0x30000000'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
      <address type='spapr-vio' reg='0x30000000'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='keyboard' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='usb'>
      <address type='usb' bus='0' port='2'/>
    </input>
    <graphics type='vnc' port='-1' autoport='yes'>
      <listen type='address'/>
    </graphics>
    <video>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
    <panic model='pseries'/>
  </devices>
</domain>

2.
Define and start the guest

3. Dump the xml
Run 'virsh dumpxml avocado-vt-vm1 > a1.xml'

4. Restart libvirtd
Run 'systemctl restart libvirtd'

5. Dump the xml again
Run 'virsh dumpxml avocado-vt-vm1 > a2.xml'

Actual results:
The backingStore element disappear after restart libvirtd

Expected results:
The backingStore element still exits.


Additional info:

Comment 3 Peter Krempa 2017-11-08 14:30:57 UTC

So the very important but missing fact is that the termination backing store element (the empty one) vanishes:

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/base.qcow2'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>

This is true for single layer and for nested images.

Comment 4 Peter Krempa 2017-11-13 12:19:29 UTC

Fixed upstream:

commit 19448a2561699807c075e071a45df292efd01f6b 
Author: Peter Krempa <pkrempa>
Date:   Wed Nov 8 15:42:51 2017 +0100

    conf: Properly parse <backingStore/>
    
    The terminator would not be parsed properly since the XPath selector was
    looking for an populated element, and also the code did not bother
    assigning the terminating virStorageSourcePtr to the backingStore
    property of the parent.
    
    Some tests would catch it if there wasn't bigger fallout from the change
    to backing store termination in a693fdba0111. Fix them properly now.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1509110

Comment 7 Meina Li 2017-11-24 09:03:30 UTC

Hi,Peter

I verified it on libvirt-3.9.0-2.el7.x86_64/qemu-kvm-rhev-2.10.0-6.el7.x86_64,the result seems not expected on raw source file. Could you help to check it again? Thanks.

Steps:
1. Prepare vm with the following xml and start vm:
…
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/lmn.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/libvirt/images/demo.img'/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
...
# virsh start lmn
Domain lmn started

2. Check the backingStore element in dumpxml:
# virsh dumpxml lmn
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/lmn.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/libvirt/images/demo.img'/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
 ...

Comment 8 Peter Krempa 2017-11-24 09:55:40 UTC

Hmm right. The difference is that now that we require an explicit terminator in the data structures to format the element in the XML.

This configuration does not add the terminating data structure since raw images don't go through the detection code which would add it normally.

Comment 9 Meina Li 2017-11-27 09:02:09 UTC

Hi,Peter
About your response, I still have two questions confused:
(1) Is it different with rhel7.4 for raw image source file in design? I can test the case with raw source file in rhel7.4, and I think it should be the same result now. Or it is a another bug in this version.
(2) In this bug, if raw images don't go through the detection code, why modify the test xml which include raw source file simultaneously? For example:
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/dev/null'/>
      <target dev='sdg' bus='scsi'/>
      <readonly/>
      <shareable/>
      <wwn>0x5000c50015ea71aa</wwn>
      <alias name='scsi0-0-0-6'/>
      <address type='drive' controller='0' bus='0' target='0' unit='6'/>
    </disk>
Can you help me to analyze if my understanding is wrong, thank you very much.


Verified successfully on qcow2 image source file:
Test env components:
libvirt-3.9.0-2.el7.x86_64
Test steps:
Scenario 1: start vm with file/block disk and check backingStore element.
1. Prepare vm with the following xml and start vm:
…
      <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/lmn.qcow2'/>
      <target dev='sda' bus='ide'/>
      <wwn>0x5000c50015ea71aa</wwn>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/test.img'/>
      <target dev='sdb' bus='scsi'/>
      <readonly/>
      <shareable/>
      <wwn>0x5000c50015ea71aa</wwn>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
     </disk>
     <disk type='block' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source dev='/dev/Host/test.img'/>
      <target dev='vde' bus='virtio'/>
      <alias name='virtio-disk4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
...
# virsh start lmn
Domain lmn started
2. Check the backingStore element in dumpxml:
# virsh dumpxml lmn
...
        <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/lmn.qcow2'/>
      <backingStore/>
      <target dev='sda' bus='ide'/>
      <wwn>0x5000c50015ea71aa</wwn>
 ...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/test.img'/>
      <backingStore/>
      <target dev='sdb' bus='scsi'/>
    ...
    <disk type='block' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source dev='/dev/Host/test.img'/>
      <backingStore/>
      <target dev='vde' bus='virtio'/>
 ...
3. Restart libvirtd and check vm's dumpxml, it is same with step2.
Scenario 2: check backingStore element after blockcopy.
1. Directly use the vm started in scenario 1.
2. Do blockcopy and check vm's dumpxml.
# virsh blockcopy lmn vda /tmp/test1/lmn1.py --blockdev --transient-job
Block Copy started
# virsh blockcopy lmn vde --blockdev /dev/HostVG/vol1 --transient-job
Block Copy started
# virsh dumpxml lmn
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/lmn.qcow2'/>
      <backingStore/>
      <mirror type='file' file='/tmp/test1/lmn.py' format='qcow2' job='copy' ready='yes'>
        <format type='qcow2'/>
        <source file='/tmp/test1/lmn.py'/>
      </mirror>
  ...
    <disk type='block' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source dev='/dev/Host/test.img'/>
      <backingStore/>
      <mirror type='block' job='copy' ready='yes'>
        <format type='qcow2'/>
        <source dev='/dev/HostVG/vol1'/>
      </mirror>
...
3. Restart libvirtd and check vm's dumpxml, it  same with step2.
Scenario 3: check backingStore element after snapshot.
1. Directly use the vm started in scenario 1.
2. Do snapshot and check vm's dumpxml.
# virsh snapshot-create-as lmn vol1 --disk-only
Domain snapshot vol1 created
# virsh snapshot-create-as lmn vol2 --disk-only
Domain snapshot vol2 created
# virsh dumpxml lmn
…
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/lmn.vol2'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/var/lib/libvirt/images/lmn.vol1'/>
        <backingStore type='file' index='2'>
          <format type='qcow2'/>
          <source file='/var/lib/libvirt/images/lmn.qcow2'/>
          <backingStore/>
        </backingStore>
      </backingStore>
      <target dev='sda' bus='ide'/>
…
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/dev/Host/test.vol2'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/dev/Host/test.vol1'/>
        <backingStore type='block' index='2'>
          <format type='qcow2'/>
          <source dev='/dev/Host/test.img'/>
          <backingStore/>
        </backingStore>
      </backingStore>
...
3. Restart libvirtd and check vm's dumpxml, it  same with step2.

Comment 10 Peter Krempa 2017-11-27 09:28:46 UTC

(In reply to Meina Li from comment #9)
> Hi,Peter
> About your response, I still have two questions confused:
> (1) Is it different with rhel7.4 for raw image source file in design? I can
> test the case with raw source file in rhel7.4, and I think it should be the
> same result now. Or it is a another bug in this version.

No, it was a bug in the refactoring I've done. I've posted patches that fix it:

https://www.redhat.com/archives/libvir-list/2017-November/msg01001.html

> (2) In this bug, if raw images don't go through the detection code, why
> modify the test xml which include raw source file simultaneously? For

That was a mistake. The code which formats <backingStore/> was relying on default value of some variables to format it. Thus by adding an explicit terminator element into the chain we stopped formatting it.

It was missed during review.

> example:
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='raw' cache='none'/>
>       <source file='/dev/null'/>
>       <target dev='sdg' bus='scsi'/>
>       <readonly/>
>       <shareable/>
>       <wwn>0x5000c50015ea71aa</wwn>
>       <alias name='scsi0-0-0-6'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='6'/>
>     </disk>
> Can you help me to analyze if my understanding is wrong, thank you very much.

Comment 11 Peter Krempa 2017-12-01 14:54:20 UTC

I've fixed it upstream by:

commit adcc31bb89e47ef642cbcfcff372131db7cd8d8b
Author: Peter Krempa <pkrempa>
Date:   Fri Nov 24 12:59:59 2017 +0100

    qemu: domain: Fix backing store terminator for non-backing local files
    
    Raw local files do not pass through the backing store detector and thus
    the code did not allocate the required backing store terminator for
    them. Previously the terminating element would be formatted into the XML
    since the default values used for the metadata allowed that. This is a
    regression since a693fdba0111ff which was not detected in the review.
    
    This patch also reverts all the changes in the test files.

commit 24e47ee2b93312742f73a785b9ca788df4677749
Author: Peter Krempa <pkrempa>
Date:   Fri Nov 24 12:04:14 2017 +0100

    qemu: process: Move handling of non-backing files into qemuDomainDetermineDiskChain
    
    Until now we would skip loading of the backing chain for files which
    don't support backing chains only when starting up the VM. Move the
    check from qemuProcessPrepareHostStorage with some adaptations so that's
    always applied.

commit b19710b9b6039728620a8d557a322ef139529226
Author: Peter Krempa <pkrempa>
Date:   Fri Nov 24 12:09:53 2017 +0100

    qemu: domain: Refactor control flow in qemuDomainDetermineDiskChain
    
    Split out clearing of the backing chain prior to other code since it
    will be required later and optimize few layers of nested conditions and
    loops.

Comment 14 Meina Li 2017-12-13 10:26:42 UTC

Verified successfully on raw image source file
Test env components:
libvirt-3.9.0-6.el7.x86_64

Test steps:
Prepare a guest with the following xml:
... <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/rhel7.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/HostVG/vol1'/>
      <target dev='sda' bus='scsi'/>
      <wwn>0x5000c50015ea71aa</wwn>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw'/>
      <source protocol='gluster' name='gv0/vol1'>
        <host name='10.66.4.164' port='24007'/>
      </source>
      <target dev='sdb' bus='ide'/>
      <wwn>0x5000c50015ea71aa</wwn>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>...

1. Start guest with file/block/network disk and check backingStore element.
# virsh dumpxml rhel7 | grep "backingStore" -a1
      <source file='/var/lib/libvirt/images/rhel7.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
--
      <source dev='/dev/HostVG/vol1'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
--
      </source>
      <backingStore/>
      <target dev='sdb' bus='ide'/>
# systemctrl restart libvirtd
# virsh dumpxml rhel7 | grep "backingStore" -a1
---->The same as above.

2. Do blockcopy for guest and check backingStore element.
# virsh blockcopy rhel7 sda /var/lib/libvirt/images/test.img --wait --verbose --transient-job 
Block Copy: [100 %]
Now in mirroring phase
# virsh blockcopy rhel7 sdb /var/lib/libvirt/images/test1.img --wait --verbose --transient-job 
Block Copy: [100 %]
Now in mirroring phase
# virsh dumpxml rhel7 | grep "backingStore" -a1
      <source file='/var/lib/libvirt/images/rhel7.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
--
      <source dev='/dev/HostVG/vol1'/>
      <backingStore/>
      <mirror type='file' file='/var/lib/libvirt/images/test.img' format='raw' job='copy' ready='yes'>
--
      </source>
      <backingStore/>
      <mirror type='file' file='/var/lib/libvirt/images/test1.img' format='raw' job='copy' ready='yes'>
# systemctrl restart libvirtd
# virsh dumpxml rhel7 | grep "backingStore" -a1
---->The same as above.

3. Hotplug disk device and check backingStore element.
# cat test.xml
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/test.img'/>
      <target dev='vdb' bus='virtio'/>
    </disk>
# virsh attach-device rhel7 test.xml 
Device attached successfully
# virsh dumpxml rhel7 | grep "backingStore" -a1
      <source file='/var/lib/libvirt/images/rhel7.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
--
      <source file='/var/lib/libvirt/images/test.img'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
--
      <source dev='/dev/HostVG/vol1'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
--
      </source>
      <backingStore/>
      <target dev='sdb' bus='ide'/>
# systemctrl restart libvirtd
# virsh dumpxml rhel7 | grep "backingStore" -a1
---->The same as above.

Move it to Verified.

Comment 15 Meina Li 2018-01-17 11:29:55 UTC

Hi,Peter

For <backingStore> element, it can output correctly after guest start for network disk.

For rhel7.5, as above test for gluster, it can output after guest start; but for the other network disk type(iscsi and rbd), it can't output <backingStore> element.

Version: libvirt-3.9.0-7.el7.x86_64
Result after guest start:
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='threads'/>
      <source file='/var/lib/libvirt/images/lmn.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads'/>
      <source protocol='rbd' name='lmn/vol1'>
        <host name='**IP**'/>
      </source>                           ---------><backingStore> not show up
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads'/>
      <source protocol='gluster' name='gv0/test1.img'>
        <host name='**IP**' port='24007'/>
      </source>
      <backingStore/>
      <target dev='vdc' bus='virtio'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </disk>
 
So, this is a regression question, please help to check it again,thank you very much.

Comment 16 Peter Krempa 2018-01-22 15:06:30 UTC

That certainly is not a regression. Libvirt wasn't ever able to traverse backing chains to iSCSI or RBD disks. We report the rest of the chain as unknown by not putting the </backingStore> terminator in the chain for the disk.

Comment 20 errata-xmlrpc 2018-04-10 10:59:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0704

Note You need to log in before you can comment on or make changes to this bug.