Bug 1161831

Summary: screenshot after qemu-attach crashes libvirtd
Product: Red Hat Enterprise Linux 7 Reporter: Luyao Huang <lhuang>
Component: libvirtAssignee: Ján Tomko <jtomko>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.1CC: dyuan, mzhan, rbalakri
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: All   
Whiteboard:
Fixed In Version: libvirt-1.2.13-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-19 05:55:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Luyao Huang 2014-11-08 09:36:33 UTC
Description of problem:
libvirt will crash when use screenshot to a vm which use qemu-attach connect to libvirt

Version-Release number of selected component (if applicable):
libvirt-1.2.8-6.el7.x86_64

How reproducible:
100%

Steps to Reproduce:

1.# /usr/libexec/qemu-kvm -m 512 -hda /var/lib/libvirt/images/test6.img -net nic -net tap,vlan=0,ifname=tap0,script=no --daemonize -monitor unix:/tmp/demo,server,nowait -vnc 127.0.0.1:2 -name sdsd2

2.# virsh qemu-attach 22094
Domain sdsd2 attached to pid 22094

3.# virsh screenshot sdsd2
error: could not take a screenshot of sdsd2
error: End of file while reading data: Input/output error
error: Failed to reconnect to the hyperviso

Actual results:
libvirtd crash

Expected results:
fix it

Additional info:

(gdb) t a a bt

Thread 11 (Thread 0x7f114a34d700 (LWP 24159)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f5440, 
    m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2  0x00007f115a39d75b in virThreadPoolWorker (
    opaque=opaque@entry=0x7f115c2d7a60) at util/virthreadpool.c:104
#3  0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
    at util/virthread.c:197
#4  0x00007f1157bf7df3 in start_thread (arg=0x7f114a34d700)
    at pthread_create.c:308
#5  0x00007f115750e05d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 10 (Thread 0x7f115ae66880 (LWP 24155)):
#0  0x00007f1157503a8d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f115a3625a1 in poll (__timeout=5000, __nfds=13, 
    __fds=<optimized out>) at /usr/include/bits/poll2.h:46
#2  virEventPollRunOnce () at util/vireventpoll.c:643
#3  0x00007f115a361092 in virEventRunDefaultImpl () at util/virevent.c:308
#4  0x00007f115aee96ad in virNetServerRun (srv=srv@entry=0x7f115c2f52c0)
    at rpc/virnetserver.c:1139
---Type <return> to continue, or q <return> to quit---
#5  0x00007f115aeb6548 in main (argc=<optimized out>, argv=<optimized out>)
    at libvirtd.c:1507

Thread 9 (Thread 0x7f114ab4e700 (LWP 24158)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f5440, 
    m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2  0x00007f115a39d75b in virThreadPoolWorker (
    opaque=opaque@entry=0x7f115c2d8090) at util/virthreadpool.c:104
#3  0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
    at util/virthread.c:197
#4  0x00007f1157bf7df3 in start_thread (arg=0x7f114ab4e700)
    at pthread_create.c:308
#5  0x00007f115750e05d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 8 (Thread 0x7f1148349700 (LWP 24163)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f54d8, 
    m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2  0x00007f115a39d77b in virThreadPoolWorker (
---Type <return> to continue, or q <return> to quit---
    opaque=opaque@entry=0x7f115c2d7a60) at util/virthreadpool.c:104
#3  0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
    at util/virthread.c:197
#4  0x00007f1157bf7df3 in start_thread (arg=0x7f1148349700)
    at pthread_create.c:308
#5  0x00007f115750e05d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 7 (Thread 0x7f1146b46700 (LWP 24166)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f54d8, 
    m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2  0x00007f115a39d77b in virThreadPoolWorker (
    opaque=opaque@entry=0x7f115c2d8090) at util/virthreadpool.c:104
#3  0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
    at util/virthread.c:197
#4  0x00007f1157bf7df3 in start_thread (arg=0x7f1146b46700)
    at pthread_create.c:308
#5  0x00007f115750e05d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 6 (Thread 0x7f1147347700 (LWP 24165)):
---Type <return> to continue, or q <return> to quit---
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f54d8, 
    m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2  0x00007f115a39d77b in virThreadPoolWorker (
    opaque=opaque@entry=0x7f115c2d7a60) at util/virthreadpool.c:104
#3  0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
    at util/virthread.c:197
#4  0x00007f1157bf7df3 in start_thread (arg=0x7f1147347700)
    at pthread_create.c:308
#5  0x00007f115750e05d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 5 (Thread 0x7f114b34f700 (LWP 24157)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f5440, 
    m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2  0x00007f115a39d75b in virThreadPoolWorker (
    opaque=opaque@entry=0x7f115c2d7a60) at util/virthreadpool.c:104
#3  0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
    at util/virthread.c:197
#4  0x00007f1157bf7df3 in start_thread (arg=0x7f114b34f700)
---Type <return> to continue, or q <return> to quit---
    at pthread_create.c:308
#5  0x00007f115750e05d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 4 (Thread 0x7f1147b48700 (LWP 24164)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f54d8, 
    m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2  0x00007f115a39d77b in virThreadPoolWorker (
    opaque=opaque@entry=0x7f115c2d8090) at util/virthreadpool.c:104
#3  0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
    at util/virthread.c:197
#4  0x00007f1157bf7df3 in start_thread (arg=0x7f1147b48700)
    at pthread_create.c:308
#5  0x00007f115750e05d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 3 (Thread 0x7f1148b4a700 (LWP 24162)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f54d8, 
    m=m@entry=0x7f115c2f5418) at util/virthread.c:153
---Type <return> to continue, or q <return> to quit---
#2  0x00007f115a39d77b in virThreadPoolWorker (
    opaque=opaque@entry=0x7f115c2d8090) at util/virthreadpool.c:104
#3  0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
    at util/virthread.c:197
#4  0x00007f1157bf7df3 in start_thread (arg=0x7f1148b4a700)
    at pthread_create.c:308
#5  0x00007f115750e05d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 2 (Thread 0x7f114934b700 (LWP 24161)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f5440, 
    m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2  0x00007f115a39d75b in virThreadPoolWorker (
    opaque=opaque@entry=0x7f115c2d7a60) at util/virthreadpool.c:104
#3  0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
    at util/virthread.c:197
#4  0x00007f1157bf7df3 in start_thread (arg=0x7f114934b700)
    at pthread_create.c:308
#5  0x00007f115750e05d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

---Type <return> to continue, or q <return> to quit---
Thread 1 (Thread 0x7f1149b4c700 (LWP 24160)):
#0  __strcmp_sse42 () at ../sysdeps/x86_64/multiarch/strcmp-sse42.S:164
#1  0x00007f115a50d0c9 in virSecuritySELinuxSetFileconHelper (
    path=0x7f1124000e50 "/var/cache/libvirt/qemu/qemu.screendump.m4QkFu", 
    tcon=0x0, optional=<optimized out>) at security/security_selinux.c:890
#2  0x00007f115a509513 in virSecurityManagerSetSavedStateLabel (
    mgr=0x7f113c10d630, vm=vm@entry=0x7f1138000cf0, 
    savefile=savefile@entry=0x7f1124000e50 "/var/cache/libvirt/qemu/qemu.screendump.m4QkFu") at security/security_manager.c:547
#3  0x00007f115a506476 in virSecurityStackSetSavedStateLabel (
    mgr=<optimized out>, vm=0x7f1138000cf0, 
    savefile=0x7f1124000e50 "/var/cache/libvirt/qemu/qemu.screendump.m4QkFu")
    at security/security_stack.c:351
#4  0x00007f115a509513 in virSecurityManagerSetSavedStateLabel (
    mgr=0x7f113c1680a0, vm=0x7f1138000cf0, 
    savefile=0x7f1124000e50 "/var/cache/libvirt/qemu/qemu.screendump.m4QkFu")
    at security/security_manager.c:547
#5  0x00007f11432ff94f in qemuDomainScreenshot (dom=<optimized out>, 
    st=0x7f11240009f0, screen=<optimized out>, flags=<optimized out>)
    at qemu/qemu_driver.c:3858
#6  0x00007f115a425b10 in virDomainScreenshot (
    domain=domain@entry=0x7f1124000930, stream=stream@entry=0x7f11240009f0, 
    screen=0, flags=0) at libvirt.c:3171
---Type <return> to continue, or q <return> to quit---
#7  0x00007f115aec8833 in remoteDispatchDomainScreenshot (
    server=<optimized out>, ret=0x7f11240008e0, args=0x7f1124000900, 
    rerr=0x7f1149b4bc80, msg=<optimized out>, client=0x7f115c2f6340)
    at remote_dispatch.h:7412
#8  remoteDispatchDomainScreenshotHelper (server=<optimized out>, 
    client=0x7f115c2f6340, msg=<optimized out>, rerr=0x7f1149b4bc80, 
    args=0x7f1124000900, ret=0x7f11240008e0) at remote_dispatch.h:7379
#9  0x00007f115a498ff2 in virNetServerProgramDispatchCall (msg=0x7f115c303db0, 
    client=0x7f115c2f6340, server=0x7f115c2f52c0, prog=0x7f115c300d20)
    at rpc/virnetserverprogram.c:437
#10 virNetServerProgramDispatch (prog=0x7f115c300d20, 
    server=server@entry=0x7f115c2f52c0, client=0x7f115c2f6340, 
    msg=0x7f115c303db0) at rpc/virnetserverprogram.c:307
#11 0x00007f115aee81fd in virNetServerProcessMsg (msg=<optimized out>, 
    prog=<optimized out>, client=<optimized out>, srv=0x7f115c2f52c0)
    at rpc/virnetserver.c:172
#12 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f115c2f52c0)
    at rpc/virnetserver.c:193
#13 0x00007f115a39d6c5 in virThreadPoolWorker (
    opaque=opaque@entry=0x7f115c2d8090) at util/virthreadpool.c:145
#14 0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
    at util/virthread.c:197
#15 0x00007f1157bf7df3 in start_thread (arg=0x7f1149b4c700)
---Type <return> to continue, or q <return> to quit---
    at pthread_create.c:308
#16 0x00007f115750e05d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113



Seems libvirtd crash reason is:

a qemu-attach vm have a wrong selinux label:

  <seclabel type='static' model='selinux' relabel='yes'>
    <label>system_u:system_r:virtd_t:s0-s0:c0.c1023</label>
  </seclabel>
  <seclabel type='static' model='dac' relabel='yes'>
    <label>system_u:system_r:virtd_t:s0-s0:c0.c1023</label>
  </seclabel>

and donnot have a image label.
When use sreenshot, libvirtd call virSecuritySELinuxSetFileconHelper function
and tcon = NULL.And try to pass NULL as the first parameter to strcmp().
Libvirt should add a check when try to use strcmp() function.

Comment 1 Ján Tomko 2014-11-11 08:08:00 UTC
Upstream patch:
https://www.redhat.com/archives/libvir-list/2014-November/msg00308.html

Comment 3 Ján Tomko 2014-12-04 16:25:08 UTC
v1 of the fix by Luyao Huang:
https://www.redhat.com/archives/libvir-list/2014-December/msg00009.html

The first patch has been pushed as:
commit f8c1fb3d2e38f181912544e956af068acde0e900
Author:     Luyao Huang <lhuang>
AuthorDate: 2014-12-01 17:54:35 +0800
Commit:     Martin Kletzander <mkletzan>
CommitDate: 2014-12-01 12:04:38 +0100

    qemu: Make pid available for security managers in qemuProcessAttach
    
    There are some small issue in qemuProcessAttach:
    
    1.Fix virSecurityManagerGetProcessLabel always get pid = 0,
    move 'vm->pid = pid' before call virSecurityManagerGetProcessLabel.
    
    2.Use virSecurityManagerGenLabel to get image label.
    
    3.Fix always set selinux label for other security driver label.
    
    Signed-off-by: Luyao Huang <lhuang>

git describe: v1.2.10-221-gf8c1fb3

v2 of the second patch from the series on the list:
https://www.redhat.com/archives/libvir-list/2014-December/msg00207.html

Comment 4 Ján Tomko 2014-12-11 09:46:45 UTC
Fixed upstream by:
commit c7c96647e903f50273977d1514d3a2a8f713b6e7
Author:     Luyao Huang <lhuang>
AuthorDate: 2014-12-09 16:33:57 +0800
Commit:     Ján Tomko <jtomko>
CommitDate: 2014-12-11 10:29:43 +0100

    dac: Add a new func to get DAC label of a running process
    
    When using qemuProcessAttach to attach a qemu process,
    the DAC label is not filled correctly.
    
    Introduce a new function to get the uid:gid from the system
    and fill the label.
    
    This fixes the daemon crash when 'virsh screenshot' is called:
    https://bugzilla.redhat.com/show_bug.cgi?id=1161831
    
    It also fixes qemu-attach after the prerequisite of this patch
    (commit f8c1fb3) was pushed out of order.
    
    Signed-off-by: Luyao Huang <lhuang>
    Signed-off-by: Ján Tomko <jtomko>

git describe: v1.2.11-rc2-1-gc7c9664

Comment 6 vivian zhang 2015-04-28 09:47:16 UTC
I can produce this bug with build libvirt-1.2.8-6.el7.x86_64

verify it with build
libvirt-1.2.14-1.el7.x86_64

verify steps

1. # /usr/libexec/qemu-kvm -hdb /var/lib/libvirt/images/new.img -monitor unix:/tmp/demo,server,nowait -name new -vnc 127.0.0.1:2

2.# ps aux |grep new
root     16819 19.9  0.2 589964 19276 pts/4    Sl+  17:38   0:03 /usr/libexec/qemu-kvm -hdb /var/lib/libvirt/images/new.img -monitor unix:/tmp/demo,server,nowait -name new -vnc 127.0.0.1:2
root     16828  0.0  0.0 112640   960 pts/3    S+   17:38   0:00 grep --color=auto new

3. # virsh qemu-attach 16819
Domain new attached to pid 16819


4. # virsh list
 Id    Name                           State
----------------------------------------------------
 44    new                            running


5. check dac label with dumpxml, dac label can be get correct when using root 

# virsh dumpxml new

...
 <seclabel type='static' model='selinux' relabel='yes'>
    <label>unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0-s0:c0.c1023</imagelabel>
  </seclabel>
  <seclabel type='static' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>
  </seclabel>
</domain>
...

6. check libvirtd process, and do screenshot. libvirt does not crash
# ps aux |grep libvirtd
root     10010  0.0  0.4 1247060 32308 ?       Ssl  Apr24   1:53 /usr/sbin/libvirtd
root     16945  0.0  0.0 112644   960 pts/3    S+   17:44   0:00 grep --color=auto libvirtd

# virsh screenshot new /tmp/new.ppm
Screenshot saved to /tmp/new.ppm, with type of image/x-portable-pixmap

# ps aux |grep libvirtd
root     10010  0.0  0.4 1247508 32764 ?       Ssl  Apr24   1:53 /usr/sbin/libvirtd
root     16958  0.0  0.0 112644   964 pts/3    S+   17:44   0:00 grep --color=auto libvirtd
[root@server 1.2.14-1.el7]#

Comment 8 vivian zhang 2015-05-22 08:42:20 UTC
I can produce this with build libvirt-1.2.8-6.el7.x86_64

verify this with build libvirt-1.2.15-2.el7.x86_64

steps:

1. # /usr/libexec/qemu-kvm -hdb /var/lib/libvirt/images/new.img -monitor unix:/tmp/demo,server,nowait -name ef
VNC server running on `::1:5900'

2. # ps aux |grep ef
root        93  0.0  0.0      0     0 ?        S<   May21   0:00 [deferwq]
root      2431 51.6  0.3 623652 28528 pts/1    Sl+  16:39   0:05 /usr/libexec/qemu-kvm -hdb /var/lib/libvirt/images/new.img -monitor unix:/tmp/demo,server,nowait -name ef

3. # virsh qemu-attach 2431
Domain ef attached to pid 2431

4. # virsh list
 Id    Name                           State
----------------------------------------------------
 17    ef                             running

5.# virsh screenshot ef
Screenshot saved to ef-2015-05-22-16:40:48.ppm, with type of image/x-portable-pixmap

check libvirtd does not crashed

move to verified

Comment 10 errata-xmlrpc 2015-11-19 05:55:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html