Hi Dafna,

could you please be more specific on the reproducer case?

with packages:
libvirt-0.10.2-15.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.351.el6.x86_64

I'm not able to reproduce the issue with following steps:

$ virsh start RHEL_nightly
Domain RHEL_nightly started

$ virsh list
 Id    Name                           State
----------------------------------------------------
 2     RHEL_nightly                   running

$ virsh suspend 2
Domain 2 suspended

$ virsh list
 Id    Name                           State
----------------------------------------------------
 2     RHEL_nightly                   paused

$ virsh resume 2
Domain 2 resumed

$ virsh list
 Id    Name                           State
----------------------------------------------------
 2     RHEL_nightly                   running

Please try the steps above and try upgrading the packages to current newest versions or provide better reproducer steps.

Thanks

still reproduced with 
libvirt-0.10.2-15.el6.x86_64 
qemu-kvm-rhev-0.12.1.2-2.348.el6.x86_64

I am using rhevm (meaning vdsm is installed and all commands are sent through engine -> vdsm) 

1. run a vm -> suspend the vm -> resume the vm. 

ERROR showing in vdsm log comes from libvirt: 

Thread-1370::ERROR::2013-01-14 17:46:44,298::vm::680::vm.Vm::(_startUnderlyingVm) vmId=`213d5348-4215-4f79-b313-81b23af4f502`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 642, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/libvirtvm.py", line 1472, in _run
    self._connection.restore(fname)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 83, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 3155, in restore
    if ret == -1: raise libvirtError ('virDomainRestore() failed', conn=self)
libvirtError: internal error Child process (lzop -dc) unexpected exit status 2


1 I ran the same vm after it exited when failing restore:  

root@gold-vdsc ~]# virsh -r list
 Id    Name                           State
----------------------------------------------------
 4     DESKTOP                        running

as you can see, we have one vm shown as running in virsh while 2 qemu pid's for the same exact vm:  

[root@gold-vdsc ~]# ps -elf |grep qemu
6 S qemu     23501     1  1  80   0 - 266960 poll_s 17:46 ?       00:00:00 /usr/libexec/qemu-kvm -name DESKTOP -S -M rhel6.4.0 -cpu Conroe -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -uuid 213d5348-4215-4f79-b313-81b23af4f502 -smbios type=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=6Server-6.4.0.3.el6,serial=58BF6CC0-8ED1-11E0-A8E7-CFC2C2C2C2D6,uuid=213d5348-4215-4f79-b313-81b23af4f502 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/DESKTOP.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2013-01-14T03:46:40,driftfix=slew -no-shutdown -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/afcde1c5-6022-4077-ab06-2beed7e5e404/4230e713-9cc4-40ba-9c96-876ccca30a9d/images/5b2ce745-d7a8-469d-889a-4f23ccf623f9/8b8e5554-fc54-4452-a513-e213a69fd354,if=none,id=drive-virtio-disk0,format=raw,serial=5b2ce745-d7a8-469d-889a-4f23ccf623f9,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:23:61:5a,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/DESKTOP.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/DESKTOP.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -spice port=5900,tls-port=5901,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -incoming fd:27 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
6 S qemu     23749     1 10  80   0 - 266894 poll_s 17:47 ?       00:00:00 /usr/libexec/qemu-kvm -name DESKTOP -S -M rhel6.4.0 -cpu Conroe -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -uuid 213d5348-4215-4f79-b313-81b23af4f502 -smbios type=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=6Server-6.4.0.3.el6,serial=58BF6CC0-8ED1-11E0-A8E7-CFC2C2C2C2D6,uuid=213d5348-4215-4f79-b313-81b23af4f502 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/DESKTOP.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2013-01-14T03:47:25,driftfix=slew -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/afcde1c5-6022-4077-ab06-2beed7e5e404/4230e713-9cc4-40ba-9c96-876ccca30a9d/images/5b2ce745-d7a8-469d-889a-4f23ccf623f9/8b8e5554-fc54-4452-a513-e213a69fd354,if=none,id=drive-virtio-disk0,format=raw,serial=5b2ce745-d7a8-469d-889a-4f23ccf623f9,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:23:61:5a,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/DESKTOP.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/DESKTOP.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -spice port=5902,tls-port=5903,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
0 S root     23788  7099  0  80   0 - 25812 pipe_w 17:47 pts/0    00:00:00 grep qemu

Hi Dafna,

I can not reproduce this issue in rhevm with the packages:
# rpm -qa libvirt qemu-kvm-rhev vdsm
vdsm-4.10.2-1.1.el6.x86_64
libvirt-0.10.2-15.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.351.el6.x86_64

run a vm -> suspend the vm -> run the vm.
successfully without error.

and # ps aux | grep qemu
qemu     28531  0.0  0.0      0     0 ?        Z<   05:23   0:00 [python] <defunct>
qemu     29978  1.2  3.2 1221752 260068 ?      Sl   05:33   0:00 /usr/libexec/qemu-kvm -name libing -S -M rhel6.3.0 -cpu Penryn -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -uuid c2525d68-9c29-4548-b78d-3ed326cb1378 -smbios type=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=6Server-6.4.0.3.el6,serial=0FBD2800-7BD9-11E1-0000-E839354BFEEA_e8:39:35:4b:fe:ea,uuid=c2525d68-9c29-4548-b78d-3ed326cb1378 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/libing.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2013-01-14T22:33:50,driftfix=slew -no-shutdown -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/4005fe5d-b136-4342-b478-1f6a3bc38f1a/e0dfdb68-4f68-40ca-8c54-7506c6c88127/images/b8455d40-bbf4-4da8-82af-51ac274d0e96/fe25752f-1067-4bd5-b29c-67a0605be546,if=none,id=drive-virtio-disk0,format=raw,serial=b8455d40-bbf4-4da8-82af-51ac274d0e96,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=31,id=hostnet0,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:a8:7a:8b,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/libing.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/libing.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -spice port=5900,tls-port=5901,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -incoming fd:29 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
root     30071  0.0  0.0 103248   852 pts/0    S+   05:34   0:00 grep qemu

only one process, seems working normally.

Did I miss some steps?

Thanks, 
EricLee

I can see a difference in qemu: 
qemu-kvm-rhev-0.12.1.2-2.348.el6.x86_64

my kernel is 2.6.32-353.el6.x86_64

I am using iscsi storage (perhaps its a file/block issue).

(In reply to comment #5)
> I can see a difference in qemu: 
> qemu-kvm-rhev-0.12.1.2-2.348.el6.x86_64
> 
> my kernel is 2.6.32-353.el6.x86_64
> 
> I am using iscsi storage (perhaps its a file/block issue).

Yes, I think that's the root reason.

I can reproduce it with iscsi storage in rhevm.

when resume the suspend gust will fail:
# cat /var/log/vdsm/vdsm.log
.....
Thread-3212::ERROR::2013-01-15 07:23:22,089::vm::680::vm.Vm::(_startUnderlyingVm) vmId=`963a1200-881b-4b39-89ae-1c05fcfae2db`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 642, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/libvirtvm.py", line 1427, in _run
    self._connection.restore(fname)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 83, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 3155, in restore
    if ret == -1: raise libvirtError ('virDomainRestore() failed', conn=self)
libvirtError: internal error Child process (lzop -dc) unexpected exit status 2
....

Will get two processes after restart the same guest:
# ps aux | grep qemu
qemu      2619  1.2  6.4 1068164 521320 ?      Sl   07:23   0:00 /usr/libexec/qemu-kvm -name bug -S -M rhel6.3.0 -cpu Penryn -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -uuid 963a1200-881b-4b39-89ae-1c05fcfae2db -smbios type=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=6Server-6.4.0.3.el6,serial=0FBD2800-7BD9-11E1-0000-E839354BFEEA_e8:39:35:4b:fe:ea,uuid=963a1200-881b-4b39-89ae-1c05fcfae2db -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/bug.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2013-01-15T00:23:16,driftfix=slew -no-shutdown -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/99cd5a84-fd8e-40aa-a0f8-ddc78ff95c83/d6414725-3778-46c1-9215-fdd8d52bfdde/images/c7e4f16d-9004-4a39-9a38-9b87535675a0/6b6331ef-1b4e-445f-8b5b-f129b96100ae,if=none,id=drive-virtio-disk0,format=raw,serial=c7e4f16d-9004-4a39-9a38-9b87535675a0,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -netdev tap,fd=31,id=hostnet0,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:a8:7a:09,bus=pci.0,addr=0x3,bootindex=1 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/bug.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/bug.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -spice port=5900,tls-port=5901,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -incoming fd:29 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
qemu      2881 27.8  0.3 1068280 29008 ?       Sl   07:23   0:01 /usr/libexec/qemu-kvm -name bug -S -M rhel6.3.0 -cpu Penryn -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -uuid 963a1200-881b-4b39-89ae-1c05fcfae2db -smbios type=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=6Server-6.4.0.3.el6,serial=0FBD2800-7BD9-11E1-0000-E839354BFEEA_e8:39:35:4b:fe:ea,uuid=963a1200-881b-4b39-89ae-1c05fcfae2db -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/bug.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2013-01-15T00:23:57,driftfix=slew -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/99cd5a84-fd8e-40aa-a0f8-ddc78ff95c83/d6414725-3778-46c1-9215-fdd8d52bfdde/images/c7e4f16d-9004-4a39-9a38-9b87535675a0/6b6331ef-1b4e-445f-8b5b-f129b96100ae,if=none,id=drive-virtio-disk0,format=raw,serial=c7e4f16d-9004-4a39-9a38-9b87535675a0,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:a8:7a:09,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/bug.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/bug.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -spice port=5902,tls-port=5903,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
root      2918  0.0  0.0 103248   852 pts/0    S+   07:24   0:00 grep qemu
qemu     28531  0.0  0.0      0     0 ?        Z<   05:23   0:00 [python] <defunct>

Dafna, I've created a scratch build for you:

http://brewweb.devel.redhat.com/brew/taskinfo?taskID=5284793

I think it should fix your problem - can you please confirm that? And if it doesn't, please attach new full debug logs (there are some patches within the build that extend debugging in crucial parts). Thanks!

the vm is still killed when trying to restore. 
but the qemu pid is killed as well: 

[root@gold-vdsc ~]# ps -elf |grep qemu
5 Z qemu     12853 10320  0  75  -5 -     0 exit   11:23 ?        00:00:00 [python] <defunct>
0 S root     23200  6074  0  80   0 - 25812 pipe_w 11:42 pts/0    00:00:00 grep qemu


will attach full logs

Created attachment 683504 [details]
logs

Cool, so I think I know where the problem is now:

When using compressed image, here are the steps libvirt do when resuming:

1) Read compressed image header (where domain XML is stored among other run-time info)
2) Start decompression binary, which is fed on stdin with compressed image. The stdout, where decompressed data are expected to be written, is connected to qemu process
3) Start qemu process
4) If any above step fail, and domain is transient, remove it from internal list of domains.

The problem here was, if the decompression binary failed after we started qemu process, we removed appropriate domain from the internal list, so 'virsh list' didn't report it anymore, however - did not kill qemu process. The scratch build I've created is introducing logging of decompression binary's stderr (among other features/fixes). And we can see result yet:

2013-01-20 09:33:16.257+0000: 10048: debug : qemuDomainSaveImageStartVM:5018 : Decompression binary stderr: lzop: <stdin>: warning: ignoring trailing garbage in lzop file

Which is weird, because the file decompression binary is reading from was created by the very same binary. Or maybe - is something appending data to saved image (maybe vdsm is storing some data there)?

The proper solution may be - ignore errors from decompression binary. Qemu process will fail with "load of migration failed" if there are some unrecoverable problems anyway.

Patches proposed upstream:

https://www.redhat.com/archives/libvir-list/2013-January/msg01639.html

Patch fixing this bug has been pushed upstream:

commit 93e5a1432d1304fafde4b2186cef63692f171c57
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Jan 28 15:13:27 2013 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Tue Jan 29 09:51:47 2013 +0100

    qemu: Destroy domain on decompression binary error
    
    https://bugzilla.redhat.com/show_bug.cgi?id=894723
    
    Currently, if qemuProcessStart() succeeds, but it's decompression
    binary that returns nonzero status, we don't kill the qemu process,
    but remove it from internal domain list, leaving the qemu process
    hanging around totally uncontrolled.


Although, I think the rest of patches in the set is worth backporting as well, as they catch stderr of decompression binary so if we kill it, we know at least what was the error. However, upstream is currently frozen as we are very close to upstream release, which is scheduled on Wed Jan 23 (tomorrow). So I am not moving to POST yet.

And one more thing - Ayal/Dafna the path you are passing to libvirt to save a domain onto - is it really a path on a filesystem or could it be a bare iSCSI partition? The difference would be - on a partition, decompression binary can't know where is the end of compressed data and where do uninitialized blocks start. That would explain appending some rubbish at the end of saved image.

Okay, patches has been pushed upstream now:


commit 137229bf4aec7647f3f04033ad390bcc872bd7e1
Author:     Michal Privoznik <mprivozn>
AuthorDate: Thu Jan 17 11:59:23 2013 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Tue Feb 5 15:45:21 2013 +0100

    qemu: Catch stderr of image compression binary
    
    If a compression binary prints something to stderr, currently
    it is discarded. However, it can contain useful data from
    debugging POV, so we should catch it.

commit cc6c425f94f4285261d4d12534f1944372f533a6
Author:     Michal Privoznik <mprivozn>
AuthorDate: Thu Jan 17 11:42:00 2013 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Tue Feb 5 15:45:21 2013 +0100

    qemu: Catch stderr of image decompression binary
    
    If a decompression binary prints something to stderr, currently
    it is discarded. However, it can contain useful data from
    debugging POV, so we should catch it.

commit 1f25194ad1e044d2fe192e871081dd570102a62a
Author:     Michal Privoznik <mprivozn>
AuthorDate: Thu Jan 17 11:09:39 2013 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Tue Feb 5 15:45:21 2013 +0100

    virFileWrapperFd: Switch to new virCommandDoAsyncIO
    
    Commit 34e8f63a32f83 introduced support for catching errors from
    libvirt iohelper. However, at those times there wasn't such fancy
    API as virCommandDoAsyncIO(), so everything has to be implemented
    on our own. But since we do have the API now, we can use it and
    drop our implementation then.

commit f0154959b3f8c3213c611883d04da1a5bac81df9
Author:     Michal Privoznik <mprivozn>
AuthorDate: Wed Jan 16 18:55:06 2013 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Tue Feb 5 15:45:21 2013 +0100

    tests: Create test for virCommandDoAsyncIO
    
    This is just a basic test, so we don't break virCommand in the
    future. A "Hello world\n" string is written to commanhelper,
    which copies input to stdout and stderr where we read it from.
    Then the read strings are compared with expected values.

commit 39c77fe586baccd0a4a9862e8cf7c78ac7af3494
Author:     Michal Privoznik <mprivozn>
AuthorDate: Wed Jan 16 11:58:00 2013 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Tue Feb 5 15:45:21 2013 +0100

    Introduce event loop to commandtest
    
    This is just preparing environment for the next patch, which is
    going to need an event loop.

commit 68fb755002da73db4dad1f2ec41bfa317855c206
Author:     Michal Privoznik <mprivozn>
AuthorDate: Wed Jan 16 11:33:17 2013 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Tue Feb 5 15:45:21 2013 +0100

    virCommand: Introduce virCommandDoAsyncIO
    
    Currently, if we want to feed stdin, or catch stdout or stderr of a
    virCommand we have to use virCommandRun(). When using virCommandRunAsync()
    we have to register FD handles by hand. This may lead to code duplication.
    Hence, introduce an internal API, which does this automatically within
    virCommandRunAsync(). The intended usage looks like this:
    
        virCommandPtr cmd = virCommandNew*(...);
        char *buf = NULL;
    
        ...
    
        virCommandSetOutputBuffer(cmd, &buf);
        virCommandDoAsyncIO(cmd);
    
        if (virCommandRunAsync(cmd, NULL) < 0)
            goto cleanup;
    
        ...
    
        if (virCommandWait(cmd, NULL) < 0)
            goto cleanup;
    
        /* @buf now contains @cmd's stdout */
        VIR_DEBUG("STDOUT: %s", NULLSTR(buf));
    
        ...
    
    cleanup:
        VIR_FREE(buf);
        virCommandFree(cmd);
    
    Note, that both stdout and stderr buffers may change until virCommandWait()
    returns.

v1.0.2-49-g137229b

(In reply to comment #18)
> Patch fixing this bug has been pushed upstream:
> 
> commit 93e5a1432d1304fafde4b2186cef63692f171c57
> Author:     Michal Privoznik <mprivozn>
> AuthorDate: Mon Jan 28 15:13:27 2013 +0100
> Commit:     Michal Privoznik <mprivozn>
> CommitDate: Tue Jan 29 09:51:47 2013 +0100
> 
>     qemu: Destroy domain on decompression binary error
>     
>     https://bugzilla.redhat.com/show_bug.cgi?id=894723
>     
>     Currently, if qemuProcessStart() succeeds, but it's decompression
>     binary that returns nonzero status, we don't kill the qemu process,
>     but remove it from internal domain list, leaving the qemu process
>     hanging around totally uncontrolled.
> 
> 
> Although, I think the rest of patches in the set is worth backporting as
> well, as they catch stderr of decompression binary so if we kill it, we know
> at least what was the error. However, upstream is currently frozen as we are
> very close to upstream release, which is scheduled on Wed Jan 23 (tomorrow).
> So I am not moving to POST yet.
> 
> And one more thing - Ayal/Dafna the path you are passing to libvirt to save
> a domain onto - is it really a path on a filesystem or could it be a bare
> iSCSI partition? The difference would be - on a partition, decompression
> binary can't know where is the end of compressed data and where do
> uninitialized blocks start. That would explain appending some rubbish at the
> end of saved image.

It can be either a file or an LV.

Ayal et Dafna,

I don't think you should use a LV or a partition directly. Fortunately, you are using a compression which is wise enough to see a garbage at the end of compressed data. Problem is, when you are using a LV directly - even for storing linear data - you cannot simply tell where does data end and where do uninitialized block start. That's why you should create a filesystem on the top of LV. If you were not using compression, qemu would be very impressed about rubbish migration data when restoring a domain.

(In reply to comment #22)
> Ayal et Dafna,
> 
> I don't think you should use a LV or a partition directly. Fortunately, you
> are using a compression which is wise enough to see a garbage at the end of
> compressed data. Problem is, when you are using a LV directly - even for
> storing linear data - you cannot simply tell where does data end and where
> do uninitialized block start. That's why you should create a filesystem on
> the top of LV. If you were not using compression, qemu would be very
> impressed about rubbish migration data when restoring a domain.

It's not fortunate, it's deliberate for exactly this reason.  Managing a single file file system is a big overhead (error flows are ridiculous for this use case).

Compression solves this problem very neatly.
Only thing is, that we need to make sure virt-qe has this use case in their testing.
Preferably have some unit tests as well.

I don't think so. The decompression binary exits with non-zero status when something goes bad during decompression process, e.g a garbage appended to the end of compressed data is found. From the binary POV it's mangled data. And when libvirt detects the binary hasn't exited cleanly, it just kills the domain which is being resumed. We cannot let a domain continue with corrupted memory, right? And corrupted memory it is - there is not special exit code (well, for the lzop binary you are using) to distinguish:

a) garbage at EOF = hopefully safe to continue, if the binary just hadn't supplied the garbage to the qemu

b) all the other error states, e.g. corrupted image = unsafe to continue.

And I don't think libvirt should parse the stderr of the decompression binary just to tell which case we are dealing with.

IOW, Dafna, you will still be unable to resume a domain if decompression binary doesn't exit cleanly, but libvirt will not leave any leaked qemu process behind.

Dan, I may be missing something here.  Isn't this the way it has been working for years?

(In reply to comment #25)
> Dan, I may be missing something here.  Isn't this the way it has been
> working for years?

I have not read through this bug, but I suspect that it was tickled by

  http://gerrit.ovirt.org/5928
  Move option 'save_image_format' to qemu.conf

as before it, we have not been using compression at all.

(In reply to comment #26)
> (In reply to comment #25)
> > Dan, I may be missing something here.  Isn't this the way it has been
> > working for years?
> 
> I have not read through this bug, but I suspect that it was tickled by
> 
>   http://gerrit.ovirt.org/5928
>   Move option 'save_image_format' to qemu.conf
> 
> as before it, we have not been using compression at all.

Ok, then qemu just knows where to stop reading according to the amount of memory? or have we been having silent memory corruptions?
According to Michal, we should not be using block devices, only files when suspending to disk (i.e. create a FS on the LV for a single file we're going to save the state in).
I totally disagree with this requirement as compression works with block devices in other places and does not require a file system, that is the point of adding a size variable.  It should just ignore the rest of the device.  If it cannot ignore it, then it should just zero out the rest of the LV.

Turns out there's a race, so we need another patch to prevent the race:

commit 3178df9afa45cf9d0694536f7fcefd0384def488
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Feb 8 15:17:44 2013 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Wed Feb 13 09:54:19 2013 +0100

    virCommand: Don't misuse the eventloop for async IO
    
    Currently, if a command wants to do asynchronous IO, a callback
    is registered in the libvirtd eventloop to handle writes and
    reads. However, there's a race in virCommandWait. The eventloop
    may already be executing the callback, while virCommandWait is
    mangling internal state of virCommand. To deal with it, we need
    to either introduce locking or spawn a separate thread where we
    poll() on stdio from child. The former, however, requires to
    unlock all mutexes held, as the event loop may execute other
    callbacks which tries to lock one of the mutexes, deadlock and
    thus never wake us up. So it's safer to spawn a separate thread.

v1.0.2-152-g3178df9

The patch is pushed upstream, so no changes to the bug are needed really. I am adding this comment just for transcription sakes.

(In reply to comment #27)
> 
> Ok, then qemu just knows where to stop reading according to the amount of
> memory? or have we been having silent memory corruptions?

I doubt that we have such corruptions (we hibernate into block devices since rhev-2.2, someone should have noticed already). Be if we cannot restore a hibernated VM from a block volume, we must issue a quick fix.

Okay, so from my discussion with Ayal, we decided to solve this issue in libvirt. So I've just posted patch upstream:

https://www.redhat.com/archives/libvir-list/2013-February/msg01062.html

Fixed upstream by v1.0.2-209-g0eeedf5:

commit 0eeedf52e7cb6a1794796856ac077d17a7d7def4
Author: Michal Privoznik <mprivozn>
Date:   Tue Feb 19 18:07:58 2013 +0100

    qemu: Run lzop with '--ignore-warn'
    
    Currently, if lzop decompression binary produces a warning, it
    doesn't exit with zero status but 2 instead. Terrifying, but
    true. However, warnings may be ignored using '--ignore-warn'
    command line argument.  Moreover, in which case, the exit status
    will be zero.

*** Bug 910706 has been marked as a duplicate of this bug. ***

Verified pass with package: libvirt-0.10.2-19.el6

1. Prepare a iscsi storage domain in RHEVM.
2. Run a vm on the iscsi storage domain.
3. Suspend the vm, # ps aux | grep qemu there is only one qemu process, after suspend completed, can not catch qemu process, and the state of the vm on RHEVM is Suspend.
4. reRun the vm, and # ps aux | grep qemu there is only one qemu process too.
5. Shut Off the vm.
6. reRun the vm.

All above steps passed.

So setting VERIFIED.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1581.html