1) We tried to create a snapshot: virsh snapshot-create-as --domain 'ixdc-2' --name 'live-backup-20171207062146' --description '2017-12-07 06:21:46' --atomic \ --diskspec vda,file='/home/datacenter/storage/ixdc-2/ixdc-2.vda.4SRAd2' \ --disk-only Output: error: internal error: unable to execute QEMU command 'transaction': Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open back 2) Since it didn't work, we saved the VM: virsh save ixdc-2 /mnt/datacenter/tmp/ixdc-2.save 3) And then repeated the first command (virsh snapshot-create-as) which was now successful: [h9.imatronix.com] : Snapshot created. 4) But then, we couldn't restore the VM (probably because we created a snapshot after 'save'): error: Failed to restore domain from /mnt/datacenter/tmp/ixdc-2.save error: (domain_definition):1: Start tag expected, '<' not found Reading XML form stdin... Please, let's start with the question from step 4: is it possible to save, create a snapshot and then restore? === More INFO === Host is a Linux Centos 7 server with: libvirt-1.2.17-13.el7_2.5.x86_64 qemu-kvm-ev-2.3.0-31.el7_2.10.1.x86_64 qemu-kvm-common-ev-2.3.0-31.el7_2.10.1.x86_64
Could you please attach first 100kiB of /mnt/datacenter/tmp/ixdc-2.save ? The file might got corrupted somehow. It'd be useful to see the XML saved in the save file.
Created attachment 1367991 [details] Memory snapshot
We can reproduce the problem with another VM. We do: virsh save mydomain /mnt/datacenter/tmp/mydomain.save virsh resume /mnt/datacenter/tmp/mydomain.save And get: error: Failed to restore domain from /mnt/datacenter/tmp/mydomain.save error: (domain_definition):1: Start tag expected, '<' not found Reading XML form stdin... --- More details --- # virsh -v 1.2.17 # rpm -qa | grep libvi libvirt-daemon-driver-network-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-kvm-1.2.17-13.el7_2.5.x86_64 libvirt-1.2.17-13.el7_2.5.x86_64 libvirt-python-1.2.17-2.el7.x86_64 libvirt-daemon-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-qemu-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-storage-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-interface-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-config-nwfilter-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-config-network-1.2.17-13.el7_2.5.x86_64 php-libvirt-0.5.2-1.el7.x86_64 libvirt-client-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-secret-1.2.17-13.el7_2.5.x86_64 libvirt-daemon-driver-lxc-1.2.17-13.el7_2.5.x86_64
I guess we should upgrade to libvirt x86_64 3.2.0-14.el7_4.5 and retry.
Same error is thrown doing a virsh save/restore on another machine with newer libvirt versions: # virsh -v 2.0.0 # rpm -qa | grep libvirt php-libvirt-0.4.8-1.el7.x86_64 libvirt-python-1.2.17-2.el7.x86_64 libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-2.0.0-10.el7_3.5.x86_64 # rpm -qa | grep qemu ipxe-roms-qemu-20130517-8.gitc4bce43.el7_2.1.noarch qemu-kvm-ev-2.3.0-31.el7_2.10.1.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 qemu-img-ev-2.3.0-31.el7_2.10.1.x86_64 qemu-kvm-common-ev-2.3.0-31.el7_2.10.1.x86_64
We upgraded to latest libvirt version and the same problem persist. # virsh -v 3.2.0 # rpm -qa | grep libvirt php-libvirt-0.4.8-1.el7.x86_64 libvirt-python-1.2.17-2.el7.x86_64 libvirt-libs-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-nwfilter-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-nodedev-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-storage-disk-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-storage-scsi-3.2.0-14.el7_4.5.x86_64 libvirt-client-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-network-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-qemu-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-secret-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-config-nwfilter-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-lxc-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-storage-mpath-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-storage-logical-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-storage-3.2.0-14.el7_4.5.x86_64 libvirt-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-storage-core-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-interface-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-config-network-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-storage-rbd-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-driver-storage-gluster-3.2.0-14.el7_4.5.x86_64 libvirt-daemon-kvm-3.2.0-14.el7_4.5.x86_64
This bug was caused because of a libvirt-hook script that was writing to stdout. If a libvirt-hook script writes to stdout (echo) during a 'restore' event, this bug is triggered. Workaround is to avoid writing to stdout on libvirt-hooks.
The intent of the hook during restore is to be able to change the XML file so if the script needs to return a valid XML file in that case, or no output at all. Closing since the behaviour is expected in this case.