Created attachment 1265790 [details] Error messages during guest boot Description of problem: After changing host disks, I copied a qcow2 fedora25 image to the new disk, then recreated a VM to use it. The VM does not boot (see attached picture), complains about I/O errors. Worse yet, after the first boot, the disk is corrupt and can't be read anymore. Trying to boot the guest again results in the following message: Error starting domain: internal error: process exited while connecting to monitor: 2017-03-23T13:13:53.997667Z qemu-system-x86_64: -drive file=/home/ddd/.local/share/libvirt/images/fedora25-qxl.qcow2,format=qcow2,if=none,id=drive-virtio-disk0: qcow2: Image is corrupt; cannot be opened read/write I repeated the copy twice, got the same result every time. Version-Release number of selected component (if applicable): qemu.x86_64 2:2.7.1.4.fc25 updates How reproducible: Always (three boots) Steps to Reproduce: 1. Copy guest image from its source (cp /mnt/old-fedora25/home/ddd/.local/share/libvirt/images/fedora25-qxl.qcow2 home/ddd/.local/share/libvirt/images/fedora25-qxl.qcow2) 2. Create a guest. See the XML at end of the report. 3. Boot the guest (I used virt-manager). Actual results: Guest does not boot, complains about I/O errors. After that boot, the guest disk image is corrupt. Expected results: a) If there is a corruption, it should be detected before the first boot. Here, the qcow2 file corruption appears caused by qemu b) Since the image used to boot on the other disk, the guest should boot from the new disk as well. Additional info: Dump of guest configuration: <domain type='kvm'> <name>fedora25-qxl</name> <uuid>45c4340b-9a6b-415b-bc5f-e4ca425981ea</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Westmere</model> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/ddd/.local/share/libvirt/images/fedora25-qxl.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:fa:47:19'/> <source bridge='virbr0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'> <listen type='address'/> <image compression='off'/> </graphics> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <redirdev bus='usb' type='spicevmc'> <address type='usb' bus='0' port='2'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <address type='usb' bus='0' port='3'/> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> <rng model='virtio'> <backend model='random'>/dev/urandom</backend> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </rng> </devices> </domain>
Unclear .. did you copy a running VM's disk? You cannot do that reliably and the data corruption you see is expected. If you're copying a shut down qcow2 image, then the only conclusion can be that the original image is corrupt, and therefore so is the copy.
No, the other file is not for a running VM, it's the previous disk that used to boot in that system. As far as I can tell, the file is not seen as "corrupt" for the first boot. It is then seen as "corrupt" after the boot. Therefore, the boot process corrupts it. Is there a utility I could run on the "before" / "after" files that would tell me about the corruption?
There's 'qemu-img check', but I don't know how exhaustive it is. Worth a shot though
There are a few ways to tell if a qcow2 file is corrupt. Provided you use them exactly as I say, none of them should modify the original file: (1) qemu-img check as described by Cole. (2) qemu-img convert -f qcow2 original.qcow2 -O raw /var/tmp/test.raw Convert the original file to a raw file. This reads every sector of the original file, so should uncover any possible qcow2 problems. (Unfortunately qemu-img convert cannot convert to /dev/null, so you need sufficient space to write the result even though you won't be needing it). (3) guestfish --ro -a original.qcow2 -i This opens the disk, protected by an overlay (--ro option). You can do things like listing files and directories. (4) virt-ls -lR -a original.qcow2 --checksum With the --checksum flag (which can also be omitted) this will read and compute a checksum of every file present in the original qcow2.
(In reply to Richard W.M. Jones from comment #4) > (4) virt-ls -lR -a original.qcow2 --checksum Missed a bit ... that should be: virt-ls -lR -a original.qcow2 --checksum /
> As far as I can tell, the file is not seen as "corrupt" for the first boot. > It is then seen as "corrupt" after the boot. Therefore, the boot process > corrupts it. Not necessarily - QEMU might find the corruption during the first boot (it does some self checks) and _then_ mark the whole image as needing repair. This blocks subsequent boots. > Is there a utility I could run on the "before" / "after" files that would tell > me about the corruption? Yes, "qemu-img check file.qcow2" will check for invalid metadata.
So the results of the experiments are that: - Converting the file gives me a raw file that is slightly smaller than the qcow2 file, which I find surprising because that guest is relatively fresh, and has 12.7G available. Maybe a side effect of using BTRFS as the main filesystem? ddd@muse images> qemu-img convert -f qcow2 fedora25-qxl.qcow2 -O raw fedora25-qxl.raw ddd@muse images> ls -lh fedora25-qxl.raw fedora25-qxl.qcow2 -rw-------. 1 ddd ddd 21G Mar 24 15:21 fedora25-qxl.qcow2 -rw-r--r--. 1 ddd ddd 20G Mar 24 15:53 fedora25-qxl.raw - qemu-img check does indeed find some errors. Not sure if that can help you in any way. What kind of operation could cause that? Could a reboot of the host without stopping the guests do that (i.e. does the "shutdown" sequence properly shutdown guests too, or does it just kill them in mid-flight?) Leaked cluster 163766 refcount=1 reference=0 Leaked cluster 163767 refcount=1 reference=0 Leaked cluster 163768 refcount=1 reference=0 Leaked cluster 163769 refcount=1 reference=0 Leaked cluster 163770 refcount=1 reference=0 Leaked cluster 163771 refcount=1 reference=0 Leaked cluster 163772 refcount=1 reference=0 Leaked cluster 163773 refcount=1 reference=0 Leaked cluster 163774 refcount=1 reference=0 Leaked cluster 163775 refcount=1 reference=0 Leaked cluster 163776 refcount=1 reference=0 Leaked cluster 163777 refcount=1 reference=0 Leaked cluster 163778 refcount=1 reference=0 Leaked cluster 163779 refcount=1 reference=0 Leaked cluster 163780 refcount=1 reference=0 Leaked cluster 163781 refcount=1 reference=0 Leaked cluster 163782 refcount=1 reference=0 Leaked cluster 163783 refcount=1 reference=0 Leaked cluster 163784 refcount=1 reference=0 Leaked cluster 163785 refcount=1 reference=0 Leaked cluster 163786 refcount=1 reference=0 Leaked cluster 163787 refcount=1 reference=0 Leaked cluster 163788 refcount=1 reference=0 Leaked cluster 163789 refcount=1 reference=0 Leaked cluster 163790 refcount=1 reference=0 Leaked cluster 163791 refcount=1 reference=0 Leaked cluster 163792 refcount=1 reference=0 Leaked cluster 163793 refcount=1 reference=0 Leaked cluster 163794 refcount=1 reference=0 Leaked cluster 163795 refcount=1 reference=0 Leaked cluster 163796 refcount=1 reference=0 Leaked cluster 163797 refcount=1 reference=0 Leaked cluster 163798 refcount=1 reference=0 Leaked cluster 163799 refcount=1 reference=0 Leaked cluster 163800 refcount=1 reference=0 Leaked cluster 163801 refcount=1 reference=0 Leaked cluster 163802 refcount=1 reference=0 Leaked cluster 163803 refcount=1 reference=0 Leaked cluster 163804 refcount=1 reference=0 Leaked cluster 163805 refcount=1 reference=0 Leaked cluster 163806 refcount=1 reference=0 Leaked cluster 163807 refcount=1 reference=0 Leaked cluster 163808 refcount=1 reference=0 Leaked cluster 163809 refcount=1 reference=0 Leaked cluster 163810 refcount=1 reference=0 Leaked cluster 163811 refcount=1 reference=0 Leaked cluster 163812 refcount=1 reference=0 Leaked cluster 163813 refcount=1 reference=0 Leaked cluster 163814 refcount=1 reference=0 Leaked cluster 163815 refcount=1 reference=0 Leaked cluster 163816 refcount=1 reference=0 Leaked cluster 163817 refcount=1 reference=0 Leaked cluster 163818 refcount=1 reference=0 Leaked cluster 163819 refcount=1 reference=0 Leaked cluster 163820 refcount=1 reference=0 Leaked cluster 163821 refcount=1 reference=0 Leaked cluster 163822 refcount=1 reference=0 Leaked cluster 163823 refcount=1 reference=0 Leaked cluster 163824 refcount=1 reference=0 Leaked cluster 163825 refcount=1 reference=0 Leaked cluster 163826 refcount=1 reference=0 Leaked cluster 163827 refcount=1 reference=0 Leaked cluster 163828 refcount=1 reference=0 Leaked cluster 163829 refcount=1 reference=0 Leaked cluster 163830 refcount=1 reference=0 Leaked cluster 163831 refcount=1 reference=0 Leaked cluster 163832 refcount=1 reference=0 Leaked cluster 163833 refcount=1 reference=0 Leaked cluster 163834 refcount=1 reference=0 Leaked cluster 163835 refcount=1 reference=0 Leaked cluster 163836 refcount=1 reference=0 Leaked cluster 163837 refcount=1 reference=0 Leaked cluster 163838 refcount=1 reference=0 Leaked cluster 163839 refcount=1 reference=0 ERROR cluster 163868 refcount=0 reference=1 ERROR cluster 163869 refcount=0 reference=1 ERROR cluster 172062 refcount=0 reference=1 ERROR cluster 180255 refcount=0 reference=1 ERROR cluster 188448 refcount=0 reference=1 ERROR cluster 196641 refcount=0 reference=1 ERROR cluster 196642 refcount=0 reference=1 ERROR cluster 204835 refcount=0 reference=1 ERROR cluster 213028 refcount=0 reference=1 ERROR cluster 221221 refcount=0 reference=1 ERROR cluster 229414 refcount=0 reference=1 ERROR cluster 229415 refcount=0 reference=1 ERROR cluster 237608 refcount=0 reference=1 ERROR cluster 245801 refcount=0 reference=1 ERROR cluster 253994 refcount=0 reference=1 ERROR cluster 262187 refcount=0 reference=1 ERROR cluster 262188 refcount=0 reference=1 ERROR cluster 270381 refcount=0 reference=1 ERROR cluster 278574 refcount=0 reference=1 ERROR cluster 286767 refcount=0 reference=1 ERROR cluster 294960 refcount=0 reference=1 ERROR cluster 294961 refcount=0 reference=1 ERROR cluster 303154 refcount=0 reference=1 ERROR cluster 311347 refcount=0 reference=1 ERROR cluster 319540 refcount=0 reference=1 ERROR cluster 327733 refcount=0 reference=1 ERROR OFLAG_COPIED L2 cluster: l1_index=20 l1_entry=80000002801d0000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=21 l1_entry=80000002a01e0000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=22 l1_entry=80000002c01f0000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=23 l1_entry=80000002e0200000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=24 l1_entry=8000000300220000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=25 l1_entry=8000000320230000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=26 l1_entry=8000000340240000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=27 l1_entry=8000000360250000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=28 l1_entry=8000000380270000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=29 l1_entry=80000003a0280000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=30 l1_entry=80000003c0290000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=31 l1_entry=80000003e02a0000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=32 l1_entry=80000004002c0000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=33 l1_entry=80000004202d0000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=34 l1_entry=80000004402e0000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=35 l1_entry=80000004602f0000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=36 l1_entry=8000000480310000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=37 l1_entry=80000004a0320000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=38 l1_entry=80000004c0330000 refcount=0 ERROR OFLAG_COPIED L2 cluster: l1_index=39 l1_entry=80000004e0340000 refcount=0 46 errors were found on the image. Data may be corrupted, or further writes to the image may corrupt it. 16356 leaked clusters were found on the image. This means waste of disk space, but no harm to data. 147456/327680 = 45.00% allocated, 0.00% fragmented, 0.00% compressed clusters Image end offset: 21478375424}}} The second suggestion from Rick Jones, to convert the file, works without reporting any error: {{{ ddd@muse images> qemu-img convert -f qcow2 fedora25-qxl.qcow2 -O raw fedora25-qxl.raw ddd@muse images> ls -lh fedora25-qxl.raw fedora25-qxl.qcow2 -rw-------. 1 ddd ddd 21G Mar 24 15:21 fedora25-qxl.qcow2 -rw-r--r--. 1 ddd ddd 20G Mar 24 15:53 fedora25-qxl.raw
Not sure what caused the initial issue here but doesn't seem to be much indication of a qemu bug, so closing.