RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1866400 - RFE: provide API which allows to take memory snapshot in sync with storage when storage is outsourced (e.g. using vhost-user-blk)
Summary: RFE: provide API which allows to take memory snapshot in sync with storage wh...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: libvirt
Version: unspecified
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: beta
: ---
Assignee: Peter Krempa
QA Contact: yisun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-05 13:50 UTC by Peter Krempa
Modified: 2022-11-15 10:37 UTC (History)
9 users (show)

Fixed In Version: libvirt-8.2.0-1.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-11-15 10:03:03 UTC
Type: Feature Request
Target Upstream Version: 8.2.0
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker LIBVIRTAT-12981 0 None None None 2022-09-02 03:32:47 UTC
Red Hat Issue Tracker LIBVIRTAT-12982 0 None None None 2022-09-02 03:30:49 UTC
Red Hat Issue Tracker LIBVIRTAT-12983 0 None None None 2022-06-30 10:04:14 UTC
Red Hat Product Errata RHSA-2022:8003 0 None None None 2022-11-15 10:03:48 UTC

Description Peter Krempa 2020-08-05 13:50:09 UTC
Description of problem:
qemu allows bypass of the block layer e.g. when vhost-user-blk is used. In that situation libvirt doesn't control the storage backend.

If a user then wants to create a full VM snapshot (including memory) the memory snapshot is still taken by libvirt, while something else has to orchestrate the disk snapshot.

We should provide means which will allow to synchronize such an operation.

Comment 6 RHEL Program Management 2022-02-05 07:27:20 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Comment 8 Peter Krempa 2022-03-22 09:40:07 UTC
commit ccffb60a9d8d855871a1b4fa79223351dbf9d606
Author: Peter Krempa <pkrempa>
Date:   Wed Mar 9 17:05:14 2022 +0100

    kbase: Introduce 'snapshots' page and describe the new 'manual' snapshot
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit 75aaf806c9081fe5de04525426e1f12685089b58
Author: Peter Krempa <pkrempa>
Date:   Fri Mar 4 14:37:13 2022 +0100

    qemuSnapshotCreateActiveExternal: Implement manual snapshot mode
    
    In case we are snapshotting at least one 'manual' disk we will pause the
    VM and keep it paused.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1866400
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit a1465e661e7629acfee6f2b6a6268d071c413a3b
Author: Peter Krempa <pkrempa>
Date:   Fri Mar 4 15:50:19 2022 +0100

    conf: snapshot: Introduce 'manual' mode for snapshot of a disk
    
    The idea of the manual mode is to allow a synchronized snapshot in cases
    when the storage is outsourced to an unmanaged storage provider which
    requires cooperation with snapshotting.
    
    The mode will instruct the hypervisor to pause along when the other
    components are snapshotted and the 'manual' disk can be snapshotted
    along. This increases latency of the snapshot but allows them in
    otherwise impossible situations.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

v8.1.0-201-gccffb60a9d

Comment 9 yisun 2022-04-08 10:38:16 UTC
Test with:
libvirt-8.2.0-1.el9.x86_64

POSTIVE TESTS:
===========================================
create snapshot with cmd
===========================================
1. have a running vm with two disks vda and vdb
[root@dell-per730-61 ~]# virsh domstate vm1
runnin

[root@dell-per730-61 ~]# virsh domblklist vm1
 Target   Source
--------------------------------------------------------
 vda      /var/lib/libvirt/images/jeos-27-x86_64.qcow2
 vdb      /var/lib/libvirt/images/vdb.qcow2

2. create external snapshot with vdb=manual, vda=external and dump mem snapshot
[root@dell-per730-61 ~]# virsh snapshot-create-as vm1 snap1 --diskspec vda,snapshot=external --diskspec vdb,snapshot=manual --memspec file=/tmp/vm1.mem
Domain snapshot snap1 created

3. check:
3.1 vda pointing to new image layer - jeos-27-x86_64.snap1
[root@dell-per730-61 ~]# virsh domblklist vm1 | grep vda
 vda      /var/lib/libvirt/images/jeos-27-x86_64.snap1

3.2 vdb pointing to origial image - vdb.qcow2
[root@dell-per730-61 ~]# virsh domblklist vm1 | grep vdb
 vdb      /var/lib/libvirt/images/vdb.qcow2

3.3 check snapshot xml has correct content
[root@dell-per730-61 ~]# virsh snapshot-dumpxml vm1 snap1 | egrep "[vda|vdb|vdc].*snapshot" 
<domainsnapshot>
    <disk name='vda' snapshot='external' type='file'>
    <disk name='vdb' snapshot='manual'/>
</domainsnapshot>

3.4 vm now in PAUSED status
[root@dell-per730-61 ~]# virsh domstate vm1
paused

3.5 the vm can be resumed
[root@dell-per730-61 ~]# virsh domstate vm1
running

3.6 check the snapshot metadata can be removed
[root@dell-per730-61 ~]# virsh snapshot-delete vm1 snap1 --metadata
Domain snapshot snap1 deleted

3.7 mem dumpted:
[root@dell-per730-61 ~]# ll -h /tmp/vm1.mem 
-rw-------. 1 root root 411M Apr  8 05:14 /tmp/vm1.mem



===========================================
create snapshot with xml
===========================================
1. have a running vm with 3 disks
[root@dell-per730-61 ~]# virsh domstate vm1
running

[root@dell-per730-61 ~]# virsh domblklist vm1
 Target   Source
--------------------------------------------------------
 vda      /var/lib/libvirt/images/jeos-27-x86_64.qcow2
 vdb      /var/lib/libvirt/images/vdb.qcow2
 vdc      /var/lib/libvirt/images/vdc.qcow2

2. prepare the snapshot xml as follow:
[root@dell-per730-61 ~]# cat snapshot.xml 
<domainsnapshot>
    <name>snap1</name>
    <memory file='/tmp/vm1.mem'/>
    <disks>
      <disk name='vda' snapshot='external'/>
      <disk name='vdb' snapshot='manual'/>
      <disk name='vdc' snapshot='no'/>
    </disks>
</domainsnapshot>

3. create snapshot
[root@dell-per730-61 ~]# virsh snapshot-create vm1 snapshot.xml 
Domain snapshot snap1 created from 'snapshot.xml'

4. check:
4.1 vda pointing to new image layer, vdb and vdc still pointing to original images
[root@dell-per730-61 ~]# virsh domblklist vm1
 Target   Source
--------------------------------------------------------
 vda      /var/lib/libvirt/images/jeos-27-x86_64.snap1
 vdb      /var/lib/libvirt/images/vdb.qcow2
 vdc      /var/lib/libvirt/images/vdc.qcow2

4.2 vm now in PAUSED status
[root@dell-per730-61 ~]# virsh domstate vm1
paused

4.3 check snapshot xml has correct content
[root@dell-per730-61 ~]# virsh snapshot-dumpxml vm1 snap1 | egrep "[vda|vdb|vdc].*snapshot" 
<domainsnapshot>
    <disk name='vda' snapshot='external' type='file'>
    <disk name='vdb' snapshot='manual'/>
    <disk name='vdc' snapshot='no'/>
</domainsnapshot>

4.4 mem dumped:
[root@dell-per730-61 ~]# ll -h /tmp/vm1.mem 
-rw-------. 1 root root 414M Apr  8 05:28 /tmp/vm1.mem

4.5 vm can be resumed:
[root@dell-per730-61 ~]# virsh resume vm1
Domain 'vm1' resumed

[root@dell-per730-61 ~]# virsh domstate vm1
running



===========================================
set defualt snapshot mode in disk xmls
===========================================
1. edit vm's xml and try to set snapshot=manual with vdb
[root@dell-per730-61 ~]# virsh edit vm1
...
    <disk type='file' device='disk' snapshot='manual'>
...

error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng
Extra element devices in interleave
Element domain failed to validate content

Failed. Try again? [y,n,i,f,?]: 

2. check the domaincommon.rng, there is no <value>manual</value>
[root@dell-per730-61 schemas]# cat domaincommon.rng | grep   '<define name="snapshot"' -A8
  <define name="snapshot">
    <attribute name="snapshot">
      <choice>
        <value>no</value>
        <value>internal</value>
        <value>external</value>
      </choice>
    </attribute>
  </define>

3. [root@dell-per730-61 schemas]# rpm -q --whatprovides /usr/share/libvirt/schemas/domaincommon.rng 
libvirt-libs-8.2.0-1.el9.x86_64

NEGATIVE TESTS:
1. Mixed manual and internal snapshots:
[root@dell-per730-61 ~]# virsh snapshot-create-as vm1 snap1 --diskspec vdb,snapshot=manual --diskspec vda,snapshot=internal --memspec file=/tmp/vm1.mem
error: unsupported configuration: internal and full system snapshots require all disks to be selected for snapshot

2. When vm is shutoff:
[root@dell-per730-61 ~]# virsh snapshot-create-as vm1 snap1 --diskspec vdb,snapshot=manual --diskspec vda,snapshot=external --memspec file=/tmp/vm1.mem
error: XML error: memory state cannot be saved with offline or disk-only snapshot

Comment 10 yisun 2022-04-08 10:40:48 UTC
Hi Peter, 
Seems the domaincommon.rng was not updated within the commits, I cannot set vm's xml with <disk...snapshot='manual'> with virsh edit cmd, pls help to check.

Steps as follow:
===========================================
set defualt snapshot mode in disk xmls
===========================================
1. edit vm's xml and try to set snapshot=manual with vdb
[root@dell-per730-61 ~]# virsh edit vm1
...
    <disk type='file' device='disk' snapshot='manual'>
...

error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng
Extra element devices in interleave
Element domain failed to validate content

Failed. Try again? [y,n,i,f,?]: 

2. check the domaincommon.rng, there is no <value>manual</value>
[root@dell-per730-61 schemas]# cat domaincommon.rng | grep   '<define name="snapshot"' -A8
  <define name="snapshot">
    <attribute name="snapshot">
      <choice>
        <value>no</value>
        <value>internal</value>
        <value>external</value>
      </choice>
    </attribute>
  </define>

3. [root@dell-per730-61 schemas]# rpm -q --whatprovides /usr/share/libvirt/schemas/domaincommon.rng 
libvirt-libs-8.2.0-1.el9.x86_64

Comment 13 Peter Krempa 2022-05-23 12:03:13 UTC
(In reply to yisun from comment #10)
> Hi Peter, 
> Seems the domaincommon.rng was not updated within the commits, I cannot set
> vm's xml with <disk...snapshot='manual'> with virsh edit cmd, pls help to
> check.

[...]

>     <disk type='file' device='disk' snapshot='manual'>
> 
> error: XML document failed to validate against schema: Unable to validate

Indeed. I've posted patches:

https://listman.redhat.com/archives/libvir-list/2022-May/231687.html

Comment 14 yisun 2022-05-27 11:28:51 UTC
For the rng issue mentioned above, the patch is ok. Since it's a minor fix, and after editing the rng file in test env, all things are good. So set this bz to VERIFIED.

Tested with
libvirt-8.3.0-1.el9.x86_64
Same steps as comment 9, and for the rng related part, steps as follow:

[root@dell-per740xd-09 ~]# virsh domstate vm1
running

[root@dell-per740xd-09 ~]# virsh dumpxml vm1 | grep snapshot -a10
...
    <disk type='file' device='disk' snapshot='external'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/jeos-27-x86_64.qcow2' index='2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk' snapshot='manual'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vdb.qcow2' index='1'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

[root@dell-per740xd-09 ~]# virsh snapshot-create-as vm1 snap1 --memspec file=/tmp/vm1.mem
Domain snapshot snap1 created

[root@dell-per740xd-09 ~]# virsh domstate vm1
paused

[root@dell-per740xd-09 ~]# virsh dumpxml vm1 | grep snapshot -a10
...
    <disk type='file' device='disk' snapshot='external'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/jeos-27-x86_64.snap1' index='3'/>
      <backingStore type='file' index='2'>
        <format type='qcow2'/>
        <source file='/var/lib/libvirt/images/jeos-27-x86_64.qcow2'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk' snapshot='manual'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vdb.qcow2' index='1'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>
...

[root@dell-per740xd-09 ~]# ll /tmp/vm1.mem 
-rw-------. 1 root root 450097674 May 27 07:22 /tmp/vm1.mem

[root@dell-per740xd-09 ~]# virsh resume vm1
Domain 'vm1' resumed

[root@dell-per740xd-09 ~]# virsh domblklist vm1
 Target   Source
--------------------------------------------------------
 vda      /var/lib/libvirt/images/jeos-27-x86_64.snap1
 vdb      /var/lib/libvirt/images/vdb.qcow2

Comment 16 errata-xmlrpc 2022-11-15 10:03:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: libvirt security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8003


Note You need to log in before you can comment on or make changes to this bug.