Bug 1525734

Summary:

Failed to restore domain (Start tag expected, '<' not found)

Product:

[Community] Virtualization Tools

Reporter:

Christopher Pereira <kripper>

Component:

libvirt

Assignee:

Libvirt Maintainers <libvirt-maint>

Status:

CLOSED NOTABUG

QA Contact:

Severity:

urgent

Docs Contact:

Priority:

unspecified

Version:

unspecified

CC:

kripper, libvirt-maint, pkrempa, rbalakri

Target Milestone:

---

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2018-03-19 05:29:22 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
Memory snapshot	none

Description Christopher Pereira 2017-12-13 23:55:33 UTC

1) We tried to create a snapshot:

	virsh snapshot-create-as --domain 'ixdc-2' --name 'live-backup-20171207062146' --description '2017-12-07 06:21:46' --atomic \
			--diskspec vda,file='/home/datacenter/storage/ixdc-2/ixdc-2.vda.4SRAd2' \
			--disk-only
		
Output:

	error: internal error: unable to execute QEMU command 'transaction': Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open back

2) Since it didn't work, we saved the VM:

	virsh save ixdc-2 /mnt/datacenter/tmp/ixdc-2.save
	
3) And then repeated the first command (virsh snapshot-create-as) which was now successful:

	[h9.imatronix.com] : Snapshot created.

4) But then, we couldn't restore the VM (probably because we created a snapshot after 'save'):

error: Failed to restore domain from /mnt/datacenter/tmp/ixdc-2.save
error: (domain_definition):1: Start tag expected, '<' not found
Reading XML form stdin...

Please, let's start with the question from step 4: is it possible to save, create a snapshot and then restore?

=== More INFO ===

Host is a Linux Centos 7 server with:

libvirt-1.2.17-13.el7_2.5.x86_64
qemu-kvm-ev-2.3.0-31.el7_2.10.1.x86_64
qemu-kvm-common-ev-2.3.0-31.el7_2.10.1.x86_64

Comment 1 Peter Krempa 2017-12-14 13:36:57 UTC

Could you please attach first 100kiB of /mnt/datacenter/tmp/ixdc-2.save ? The file might got corrupted somehow. It'd be useful to see the XML saved in the save file.

Comment 2 Christopher Pereira 2017-12-14 13:54:46 UTC

Created attachment 1367991 [details]
Memory snapshot

Comment 3 Christopher Pereira 2017-12-14 14:03:47 UTC

We can reproduce the problem with another VM.

We do:

virsh save mydomain /mnt/datacenter/tmp/mydomain.save
virsh resume /mnt/datacenter/tmp/mydomain.save

And get:

error: Failed to restore domain from /mnt/datacenter/tmp/mydomain.save
error: (domain_definition):1: Start tag expected, '<' not found
Reading XML form stdin...

--- More details ---

# virsh -v
1.2.17

# rpm -qa | grep libvi
libvirt-daemon-driver-network-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-kvm-1.2.17-13.el7_2.5.x86_64
libvirt-1.2.17-13.el7_2.5.x86_64
libvirt-python-1.2.17-2.el7.x86_64
libvirt-daemon-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-qemu-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-storage-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-interface-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-config-nwfilter-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-config-network-1.2.17-13.el7_2.5.x86_64
php-libvirt-0.5.2-1.el7.x86_64
libvirt-client-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-secret-1.2.17-13.el7_2.5.x86_64
libvirt-daemon-driver-lxc-1.2.17-13.el7_2.5.x86_64

Comment 4 Christopher Pereira 2017-12-14 14:09:39 UTC

I guess we should upgrade to libvirt x86_64 3.2.0-14.el7_4.5 and retry.

Comment 5 Christopher Pereira 2017-12-14 14:24:14 UTC

Same error is thrown doing a virsh save/restore on another machine with newer libvirt versions:

# virsh -v
2.0.0

# rpm -qa | grep libvirt
php-libvirt-0.4.8-1.el7.x86_64
libvirt-python-1.2.17-2.el7.x86_64
libvirt-daemon-2.0.0-10.el7_3.5.x86_64
libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64
libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64
libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64
libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64
libvirt-client-2.0.0-10.el7_3.5.x86_64
libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64
libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64
libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64
libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64
libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64
libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64
libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64
libvirt-2.0.0-10.el7_3.5.x86_64

# rpm -qa | grep qemu
ipxe-roms-qemu-20130517-8.gitc4bce43.el7_2.1.noarch
qemu-kvm-ev-2.3.0-31.el7_2.10.1.x86_64
libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64
qemu-img-ev-2.3.0-31.el7_2.10.1.x86_64
qemu-kvm-common-ev-2.3.0-31.el7_2.10.1.x86_64

Comment 6 Christopher Pereira 2017-12-15 04:17:31 UTC

We upgraded to latest libvirt version and the same problem persist.

# virsh -v
3.2.0

# rpm -qa | grep libvirt
php-libvirt-0.4.8-1.el7.x86_64
libvirt-python-1.2.17-2.el7.x86_64
libvirt-libs-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-nwfilter-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-nodedev-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7_4.5.x86_64
libvirt-client-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-network-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-qemu-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-secret-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-config-nwfilter-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-lxc-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-storage-3.2.0-14.el7_4.5.x86_64
libvirt-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-storage-core-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-interface-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-config-network-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-storage-rbd-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7_4.5.x86_64
libvirt-daemon-kvm-3.2.0-14.el7_4.5.x86_64

Comment 7 Christopher Pereira 2018-03-18 06:13:49 UTC

This bug was caused because of a libvirt-hook script that was writing to stdout.
If a libvirt-hook script writes to stdout (echo) during a 'restore' event, this bug is triggered.

Workaround is to avoid writing to stdout on libvirt-hooks.

Comment 8 Peter Krempa 2018-03-19 05:29:22 UTC

The intent of the hook during restore is to be able to change the XML file so if the script needs to return a valid XML file in that case, or no output at all.

Closing since the behaviour is expected in this case.