Bug 984793

Summary: libvirtd crashed when start a guest with <seclabel> element in xml
Product: Red Hat Enterprise Linux 6 Reporter: EricLee <bili>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.5CC: acathrow, ajia, cwei, dyuan, gsun, honzhang, jiahu, jmiao, mprivozn, mzhan, shyu, xuzhang, ydu, zhwang
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-0.10.2-21.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-21 09:05:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 920205    
Attachments:
Description Flags
libvirtd crash log
none
backtrace while restore saved file
none
libvirtd log while restoring saved file
none
libvirtd_rhevm_log
none
vdsm_rhevm_log none

Description EricLee 2013-07-16 04:18:01 UTC 
Description of problem:
libvirtd crashed when start a guest with <seclabel> element in xml

Version-Release number of selected component (if applicable):
kernel-2.6.32-396.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.378.el6.x86_64
libvirt-0.10.2-20.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. # virsh dumpxml kvm-rhel6.4-x86_64-qcow2-virtio
<domain type='kvm'>
  <name>kvm-rhel6.4-x86_64-qcow2-virtio</name>
  <uuid>6cb8b900-9e5d-a8b6-d05c-510206d121c7</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.3.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/kvm-rhel6.4-x86_64-qcow2.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:48:88:34'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/test.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <graphics type='spice' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'/>
</domain>

2. # virsh start kvm-rhel6.4-x86_64-qcow2-virtio
error: Failed to start domain kvm-rhel6.4-x86_64-qcow2-virtio
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the hypervisor
error: Failed to reconnect to the hypervisor

Actual results：
As steps

Expected results:
Should not crash.

Additional info:
The same xml and same image in libvirt-0.10.2-19.el6 will not crash.

bt and t a a bt info:
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f0e84303700 (LWP 19020)]
0x00007f0e8b74b497 in virSecurityManagerGenLabel (mgr=<value optimized out>, vm=<value optimized out>) at security/security_manager.c:344
344                    goto cleanup;
(gdb) bt
#0  0x00007f0e8b74b497 in virSecurityManagerGenLabel (mgr=<value optimized out>, vm=<value optimized out>) at security/security_manager.c:344
#1  0x0000000000485f79 in qemuProcessStart (conn=0x7f0e64000990, driver=0x7f0e780bb400, vm=0x7f0e7818e710, migrateFrom=0x0, stdin_fd=-1, stdin_path=0x0, snapshot=0x0,
    vmop=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, flags=1) at qemu/qemu_process.c:3596
#2  0x000000000046b3ce in qemuDomainObjStart (conn=0x7f0e64000990, driver=0x7f0e780bb400, vm=0x7f0e7818e710, flags=<value optimized out>) at qemu/qemu_driver.c:5796
#3  0x000000000046b9d2 in qemuDomainStartWithFlags (dom=0x7f0e740cc090, flags=0) at qemu/qemu_driver.c:5853
#4  0x00007f0e8b6a7930 in virDomainCreate (domain=0x7f0e740cc090) at libvirt.c:8319
#5  0x000000000043ff92 in remoteDispatchDomainCreate (server=<value optimized out>, client=<value optimized out>, msg=<value optimized out>, rerr=0x7f0e84302b80,
    args=<value optimized out>, ret=<value optimized out>) at remote_dispatch.h:1066
#6  remoteDispatchDomainCreateHelper (server=<value optimized out>, client=<value optimized out>, msg=<value optimized out>, rerr=0x7f0e84302b80, args=<value optimized out>,
    ret=<value optimized out>) at remote_dispatch.h:1044
#7  0x00007f0e8b6f2422 in virNetServerProgramDispatchCall (prog=0x168aa40, server=0x1681ff0, client=0x1687200, msg=0x1689c70) at rpc/virnetserverprogram.c:431
#8  virNetServerProgramDispatch (prog=0x168aa40, server=0x1681ff0, client=0x1687200, msg=0x1689c70) at rpc/virnetserverprogram.c:304
#9  0x00007f0e8b6f370e in virNetServerProcessMsg (srv=<value optimized out>, client=0x1687200, prog=<value optimized out>, msg=0x1689c70) at rpc/virnetserver.c:170
#10 0x00007f0e8b6f3dac in virNetServerHandleJob (jobOpaque=<value optimized out>, opaque=0x1681ff0) at rpc/virnetserver.c:191
#11 0x00007f0e8b6168bc in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:144
#12 0x00007f0e8b6161a9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#13 0x0000003d94a07851 in start_thread () from /lib64/libpthread.so.0
#14 0x0000003d946e890d in clone () from /lib64/libc.so.6
(gdb) t a a bt

Thread 11 (Thread 0x7f0e85705700 (LWP 19018)):
#0  0x0000003d94a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f0e8b616386 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f0e8b616953 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f0e8b6161a9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x0000003d94a07851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003d946e890d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f0e84d04700 (LWP 19019)):
#0  0x0000003d94a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f0e8b616386 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f0e8b616953 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f0e8b6161a9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x0000003d94a07851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003d946e890d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f0e84303700 (LWP 19020)):
#0  0x00007f0e8b74b497 in virSecurityManagerGenLabel (mgr=<value optimized out>, vm=<value optimized out>) at security/security_manager.c:344
#1  0x0000000000485f79 in qemuProcessStart (conn=0x7f0e64000990, driver=0x7f0e780bb400, vm=0x7f0e7818e710, migrateFrom=0x0, stdin_fd=-1, stdin_path=0x0, snapshot=0x0,
    vmop=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, flags=1) at qemu/qemu_process.c:3596
#2  0x000000000046b3ce in qemuDomainObjStart (conn=0x7f0e64000990, driver=0x7f0e780bb400, vm=0x7f0e7818e710, flags=<value optimized out>) at qemu/qemu_driver.c:5796
#3  0x000000000046b9d2 in qemuDomainStartWithFlags (dom=0x7f0e740cc090, flags=0) at qemu/qemu_driver.c:5853
#4  0x00007f0e8b6a7930 in virDomainCreate (domain=0x7f0e740cc090) at libvirt.c:8319
#5  0x000000000043ff92 in remoteDispatchDomainCreate (server=<value optimized out>, client=<value optimized out>, msg=<value optimized out>, rerr=0x7f0e84302b80,
    args=<value optimized out>, ret=<value optimized out>) at remote_dispatch.h:1066
#6  remoteDispatchDomainCreateHelper (server=<value optimized out>, client=<value optimized out>, msg=<value optimized out>, rerr=0x7f0e84302b80, args=<value optimized out>,
    ret=<value optimized out>) at remote_dispatch.h:1044
#7  0x00007f0e8b6f2422 in virNetServerProgramDispatchCall (prog=0x168aa40, server=0x1681ff0, client=0x1687200, msg=0x1689c70) at rpc/virnetserverprogram.c:431
#8  virNetServerProgramDispatch (prog=0x168aa40, server=0x1681ff0, client=0x1687200, msg=0x1689c70) at rpc/virnetserverprogram.c:304
#9  0x00007f0e8b6f370e in virNetServerProcessMsg (srv=<value optimized out>, client=0x1687200, prog=<value optimized out>, msg=0x1689c70) at rpc/virnetserver.c:170
#10 0x00007f0e8b6f3dac in virNetServerHandleJob (jobOpaque=<value optimized out>, opaque=0x1681ff0) at rpc/virnetserver.c:191
#11 0x00007f0e8b6168bc in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:144
#12 0x00007f0e8b6161a9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#13 0x0000003d94a07851 in start_thread () from /lib64/libpthread.so.0
#14 0x0000003d946e890d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f0e83902700 (LWP 19021)):
#0  0x0000003d94a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f0e8b616386 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f0e8b616953 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f0e8b6161a9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x0000003d94a07851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003d946e890d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f0e82f01700 (LWP 19022)):
#0  0x0000003d94a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x00007f0e8b616386 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f0e8b616953 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f0e8b6161a9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x0000003d94a07851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003d946e890d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f0e82500700 (LWP 19023)):
#0  0x0000003d94a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f0e8b616386 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f0e8b616953 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f0e8b6161a9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x0000003d94a07851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003d946e890d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f0e81aff700 (LWP 19024)):
#0  0x0000003d94a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f0e8b616386 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f0e8b616953 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f0e8b6161a9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x0000003d94a07851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003d946e890d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f0e810fe700 (LWP 19025)):
#0  0x0000003d94a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f0e8b616386 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f0e8b616953 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f0e8b6161a9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x0000003d94a07851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003d946e890d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f0e806fd700 (LWP 19026)):
#0  0x0000003d94a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f0e8b616386 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f0e8b616953 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f0e8b6161a9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x0000003d94a07851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003d946e890d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f0e7fcfc700 (LWP 19027)):
#0  0x0000003d94a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f0e8b616386 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f0e8b616953 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f0e8b6161a9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x0000003d94a07851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003d946e890d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f0e8b597860 (LWP 19017)):
---Type <return> to continue, or q <return> to quit---
#0  0x0000003d946df253 in poll () from /lib64/libc.so.6
#1  0x00007f0e8b603dec in virEventPollRunOnce () at util/event_poll.c:615
#2  0x00007f0e8b603027 in virEventRunDefaultImpl () at util/event.c:247
#3  0x00007f0e8b6f2f4d in virNetServerRun (srv=0x1681ff0) at rpc/virnetserver.c:748
#4  0x0000000000423947 in main (argc=<value optimized out>, argv=<value optimized out>) at libvirtd.c:1228
Comment 1 EricLee 2013-07-16 04:21:33 UTC 
Hi all,

This bug should be a regression as libvirt-0.10.2-19.el6 can not reproduce, can I set the regression/block keywords?

And for libvirtd.log, please see next comment.

Thanks,
EricLee
Comment 2 EricLee 2013-07-16 04:23:00 UTC 
Created attachment 774006 [details]
libvirtd crash log
Comment 4 Alex Jia 2013-07-16 05:18:47 UTC 
I haven't found the issue on libvirt upstream, maybe, some patches need to been backported into libvirt rhel.

In addition, I think this question is introduced by memory leaks. 

==21463== 1,220 (288 direct, 932 indirect) bytes in 1 blocks are definitely lost in loss record 95 of 105
==21463==    at 0x4A04A28: calloc (vg_replace_malloc.c:467)
==21463==    by 0x4C83C9B: virAllocN (memory.c:128)
==21463==    by 0x4C96AAB: virObjectNew (virobject.c:100)
==21463==    by 0x4D0171E: virGetConnect (datatypes.c:109)
==21463==    by 0x4D1AE25: do_open (libvirt.c:1078)
==21463==    by 0x4D1C07A: virConnectOpenAuth (libvirt.c:1407)
==21463==    by 0x40C6F5: vshReconnect (virsh.c:340)
==21463==    by 0x40D28F: vshCommandRun (virsh.c:1645)
==21463==    by 0x410B4A: main (virsh.c:3093)
Comment 5 Alex Jia 2013-07-16 05:53:23 UTC 
(In reply to Alex Jia from comment #4)
> I haven't found the issue on libvirt upstream, maybe, some patches need to
> been backported into libvirt rhel.

Can successfully start a guest.

> 
> In addition, I think this question is introduced by memory leaks. 
> 
> ==21463== 1,220 (288 direct, 932 indirect) bytes in 1 blocks are definitely
> lost in loss record 95 of 105
> ==21463==    at 0x4A04A28: calloc (vg_replace_malloc.c:467)
> ==21463==    by 0x4C83C9B: virAllocN (memory.c:128)
> ==21463==    by 0x4C96AAB: virObjectNew (virobject.c:100)
> ==21463==    by 0x4D0171E: virGetConnect (datatypes.c:109)
> ==21463==    by 0x4D1AE25: do_open (libvirt.c:1078)
> ==21463==    by 0x4D1C07A: virConnectOpenAuth (libvirt.c:1407)
> ==21463==    by 0x40C6F5: vshReconnect (virsh.c:340)
> ==21463==    by 0x40D28F: vshCommandRun (virsh.c:1645)
> ==21463==    by 0x410B4A: main (virsh.c:3093)

It should be another question, the libvirt raises memory leaks with error path (after the libvirtd crashing ).
Comment 6 Alex Jia 2013-07-16 06:59:25 UTC 
Not sure whether need to backport the following patches into libvirt rhel:


commit 2ce63c161111c6d813130f850639d1548d80c3fe
Author: Peter Krempa <pkrempa>
Date:   Tue Jul 2 18:34:58 2013 +0200

    selinux: Always generate imagelabel
    
    The imagelabel SELinux label was only generated when relabeling was
    enabled. This prohibited labeling of files created by libvirt that need
    to be labeled even if relabeling is turned off.
    
    The only codepath this change has direct impact on is labeling of FDs
    passed to qemu which is always safe in current state.



commit 48b49a631a0e6ab58376325c8301d5e52152a221
Author: Daniel P. Berrange <berrange>
Date:   Wed Feb 6 12:40:41 2013 +0000

    Serialize execution of security manager APIs
    
    Add locking to virSecurityManagerXXX APIs, so that use of the
    security drivers is internally serialized. This avoids the need
    to rely on the global driver locks to achieve serialization
    
    Signed-off-by: Daniel P. Berrange <berrange>
Comment 7 Jincheng Miao 2013-07-16 09:15:08 UTC 
I am curious about the patch :

commit 8d68cbeaa8a64759323da1d64526e2a941eb93e1
Author:     Michal Privoznik <mprivozn>
AuthorDate: Tue Apr 2 17:49:34 2013 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Wed Apr 3 10:19:46 2013 +0200

@@ -437,6 +437,19 @@ int virSecurityManagerGenLabel(virSecurityManagerPtr mgr,
         return ret;
 
     virObjectLock(mgr);
+    for (i = 0; vm->nseclabels; i++) {

Look here, the condition is vm->nseclabels. It is no related to loop variable i. That means if vm->nseclabels is true, the loop will not stop.
I guess it should be "for (i = 0; i < vm->nseclabels; i++)" 
either "for (i = 0; vm->seclabels[i]; i++)"

+        for (j = 0; sec_managers[j]; j++)
+            if (STREQ(vm->seclabels[i]->model, sec_managers[j]->drv->name))
+                break;
+
+        if (!sec_managers[j]) {
+            virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
+                           _("Unable to find security driver for label %s"),
+                           vm->seclabels[i]->model);
+            goto cleanup;
+        }
+    }
+

And I will test it.
Comment 8 Jincheng Miao 2013-07-16 09:18:53 UTC 
After I change the condition, it is ok to start a guest with 
<seclabel type='dynamic' model='selinux' relabel='yes'/>.
Comment 9 Alex Jia 2013-07-16 10:00:51 UTC 
(In reply to Jincheng Miao from comment #7)
> I am curious about the patch :
> 
> commit 8d68cbeaa8a64759323da1d64526e2a941eb93e1
> Author:     Michal Privoznik <mprivozn>
> AuthorDate: Tue Apr 2 17:49:34 2013 +0200
> Commit:     Michal Privoznik <mprivozn>
> CommitDate: Wed Apr 3 10:19:46 2013 +0200
> 

It seems it's not a root reason, with same patch, the libvirt upstream is fine.
Comment 10 Michal Privoznik 2013-07-16 10:04:11 UTC 
So we need to backport this commit:

commit ea151935bb5cfacc0290f60960f8d515ca52ab29
Author:     Guido Günther <agx>
AuthorDate: Wed Apr 3 22:08:58 2013 +0200
Commit:     Guido Günther <agx>
CommitDate: Wed Apr 3 22:57:31 2013 +0200

    security_manager: fix comparison
    
    otherwise we crash later on if we don't find a match like:
    
     #0  0xb72c2b4f in virSecurityManagerGenLabel (mgr=0xb8e42d20, vm=0xb8ef40c0) at security/security_manager.c:424
     #1  0xb18811f3 in qemuProcessStart (conn=conn@entry=0xb8eed880, driver=driver@entry=0xb8e3b1e0, vm=vm@entry=0xb8ef58f0,
         migrateFrom=migrateFrom@entry=0xb18f6088 "stdio", stdin_fd=18,
         stdin_path=stdin_path@entry=0xb8ea7798 "/var/lib/jenkins/jobs/libvirt-tck-build/workspace/tck.img", snapshot=snapshot@entry=0x0,
         vmop=vmop@entry=VIR_NETDEV_VPORT_PROFILE_OP_RESTORE, flags=flags@entry=2) at qemu/qemu_process.c:3364
     #2  0xb18d6cb2 in qemuDomainSaveImageStartVM (conn=conn@entry=0xb8eed880, driver=driver@entry=0xb8e3b1e0, vm=0xb8ef58f0, fd=fd@entry=0xb6bf3f98,
         header=header@entry=0xb6bf3fa0, path=path@entry=0xb8ea7798 "/var/lib/jenkins/jobs/libvirt-tck-build/workspace/tck.img",
         start_paused=start_paused@entry=false) at qemu/qemu_driver.c:4843
     #3  0xb18d7eeb in qemuDomainRestoreFlags (conn=conn@entry=0xb8eed880,
         path=path@entry=0xb8ea7798 "/var/lib/jenkins/jobs/libvirt-tck-build/workspace/tck.img", dxml=dxml@entry=0x0, flags=flags@entry=0)
         at qemu/qemu_driver.c:4962
     #4  0xb18d8123 in qemuDomainRestore (conn=0xb8eed880, path=0xb8ea7798 "/var/lib/jenkins/jobs/libvirt-tck-build/workspace/tck.img")
         at qemu/qemu_driver.c:4987
     #5  0xb718d186 in virDomainRestore (conn=0xb8eed880, from=0xb8ea87d8 "/var/lib/jenkins/jobs/libvirt-tck-build/workspace/tck.img") at libvirt.c:2768
     #6  0xb7736363 in remoteDispatchDomainRestore (args=<optimized out>, rerr=0xb6bf41f0, client=0xb8eedaf0, server=<optimized out>, msg=<optimized out>)
         at remote_dispatch.h:4679
     #7  remoteDispatchDomainRestoreHelper (server=0xb8e1a3e0, client=0xb8eedaf0, msg=0xb8ee72c8, rerr=0xb6bf41f0, args=0xb8ea8968, ret=0xb8ef5330)
         at remote_dispatch.h:4661
     #8  0xb720db01 in virNetServerProgramDispatchCall (msg=0xb8ee72c8, client=0xb8eedaf0, server=0xb8e1a3e0, prog=0xb8e216b0)
         at rpc/virnetserverprogram.c:439
     #9  virNetServerProgramDispatch (prog=0xb8e216b0, server=server@entry=0xb8e1a3e0, client=0xb8eedaf0, msg=0xb8ee72c8) at rpc/virnetserverprogram.c:305
     #10 0xb7206e97 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0xb8e1a3e0) at rpc/virnetserver.c:162
     #11 virNetServerHandleJob (jobOpaque=0xb8ea7720, opaque=0xb8e1a3e0) at rpc/virnetserver.c:183
     #12 0xb70f9f78 in virThreadPoolWorker (opaque=opaque@entry=0xb8e1a540) at util/virthreadpool.c:144
     #13 0xb70f94a5 in virThreadHelper (data=0xb8e0e558) at util/virthreadpthread.c:161
     #14 0xb705d954 in start_thread (arg=0xb6bf4b70) at pthread_create.c:304
     #15 0xb6fd595e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
    
    This unbreaks libvirt-tck's domain/100-transient-save-restore.t with
    qemu:///session and selinux compiled in but disabled.
    
    Introduced by 8d68cbeaa8a64759323da1d64526e2a941eb93e1
Comment 12 Xuesong Zhang 2013-07-16 10:16:17 UTC 
hi, 

I met one issue with build libvirt-0.10.2-20.el6.x86_64, the libvirtd log info and backtrace info seems same like this bug.

The steps is:
1. save one running VM.
# virsh save a a.save
Domain a saved to a.save
2. then restore the VM with that save file in step1.
# virsh restore a.save 
error: Failed to restore domain from a.save
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the hypervisor
error: Failed to reconnect to the hypervisor



As you can see the libvirtd is crashed while restore the saved file. I attached the libvirtd log and backtrace info for your reference, it seems like that the 2 issues are caused by same reason.
Comment 13 Xuesong Zhang 2013-07-16 10:17:17 UTC 
Created attachment 774165 [details]
backtrace while restore saved file
Comment 14 Xuesong Zhang 2013-07-16 10:18:01 UTC 
Created attachment 774166 [details]
libvirtd log while restoring saved file
Comment 15 Michal Privoznik 2013-07-16 10:22:51 UTC 
Moving to POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2013-July/msg00448.html
Comment 16 Hu Jianwei 2013-07-17 06:28:01 UTC 
Created attachment 774628 [details]
libvirtd_rhevm_log

guest migration failed
Comment 17 Hu Jianwei 2013-07-17 06:28:56 UTC 
Created attachment 774629 [details]
vdsm_rhevm_log

guest migration failed
Comment 18 Hu Jianwei 2013-07-17 06:31:40 UTC 
I met migration fail on libvirt-0.10.2-20.el6.x86_64 in rhevm,so can not verify this bug, after checking the logs, maybe this migration fail issue is same as bug 984793,
https://bugzilla.redhat.com/show_bug.cgi?id=984793#c12

Hosts packages version:
libvirt-0.10.2-20.el6.x86_64
libvirt-client-0.10.2-20.el6.x86_64
libvirt-debuginfo-0.10.2-20.el6.x86_64
libvirt-devel-0.10.2-20.el6.x86_64
libvirt-lock-sanlock-0.10.2-20.el6.x86_64
libvirt-python-0.10.2-20.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.355.el6_4.6.x86_64
qemu-kvm-rhev-debuginfo-0.12.1.2-2.355.el6_4.2.x86_64
qemu-kvm-rhev-tools-0.12.1.2-2.355.el6_4.2.x86_64
spice-client-0.8.2-15.el6.x86_64
spice-glib-0.14-7.el6.x86_64
spice-gtk-0.14-7.el6.x86_64
spice-gtk-python-0.14-7.el6.x86_64
spice-server-0.12.0-12.el6.x86_64
spice-vdagent-0.12.0-4.el6.x86_64
vdsm-4.10.2-23.0.el6ev.x86_64
vdsm-cli-4.10.2-23.0.el6ev.noarch
vdsm-python-4.10.2-23.0.el6ev.x86_64
vdsm-xmlrpc-4.10.2-23.0.el6ev.noarch

Also, I attached the logs, libvirtd_rhevm_log and vdsm_rhevm_log.
Comment 19 Hu Jianwei 2013-07-17 06:40:10 UTC 
Please ignore comment 18，that's wrong comment.
Comment 20 Hu Jianwei 2013-07-17 06:41:51 UTC 
Also, ignore comment 16,17.
Comment 21 Xuesong Zhang 2013-07-17 12:34:37 UTC 
As for comment 12, the root reason is caused by seclabel after debug. 

Here is the detail description about comment 12.

1. define and start one guest without seclabel.
2. then we can start and destroy the guest successfully, no libvirtd crash. Check the dumpxml, then find:
guest is shutoff status: didn't contains any seclabel.
guest is running status: libvirtd will add the seclabel automatically.

3. save the guest, after save, the guest is in shutoff status. saved file contains the running guest status, so the saved file contans seclabel info.

4. restore the guest, libvirtd crashed.

5. while resotre the guest, if adding option "--xml", add dumpxml file which didn't contain seclabel, the guest will be restored and libvirtd will not crashed.

From the step 4 and step 5, we can see, the restore issue is also caused by seclabel. The same reason with this bug.
Comment 23 yanbing du 2013-07-19 07:29:16 UTC 
Verify this bug with libvirt-0.10.2-21.el6.x86_64.
1. Prepare a guest with <seclabel> element:
...
 <seclabel type='dynamic' model='selinux' relabel='yes'/>
...
2. Start the guest
# virsh start test
Domain test started

3. Check the guest xml 
# virsh dumpxml test |grep seclabel -A 3
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c73,c933</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c73,c933</imagelabel>
  </seclabel>
</domain>

4. Save and restore the guest
# virsh save test /tmp/test.save

Domain test saved to /tmp/test.save

# virsh restore /tmp/test.save 
Domain restored from /tmp/test.save

# virsh domstate test
running
Comment 25 errata-xmlrpc 2013-11-21 09:05:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1581.html