Bug 1369988 - libvirtd crashed when use vcpuinfo to a running guest with vcpus hotplug table
Summary: libvirtd crashed when use vcpuinfo to a running guest with vcpus hotplug table
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
Assignee: Peter Krempa
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-25 02:10 UTC by Luyao Huang
Modified: 2016-08-26 15:37 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2016-08-26 15:37:22 UTC
Embargoed:

Attachments (Terms of Use)

Description Luyao Huang 2016-08-25 02:10:08 UTC 
Description of problem:
libvirtd crashed when use vcpuinfo to a running guest with vcpus hotplug table

Version-Release number of selected component (if applicable):
v2.1.0-209-ge3229f6

How reproducible:
100%

Steps to Reproduce:
1. Add this in guest xml:

  <vcpu placement='static' current='7'>10</vcpu>
  <vcpus>
    <vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='1' enabled='yes' hotpluggable='yes' order='8'/>
    <vcpu id='2' enabled='yes' hotpluggable='yes' order='3'/>
    <vcpu id='3' enabled='no' hotpluggable='yes' order='4'/>
    <vcpu id='4' enabled='yes' hotpluggable='yes' order='5'/>
    <vcpu id='5' enabled='yes' hotpluggable='yes' order='6'/>
    <vcpu id='6' enabled='yes' hotpluggable='yes' order='9'/>
    <vcpu id='7' enabled='no' hotpluggable='yes' order='2'/>
    <vcpu id='8' enabled='no' hotpluggable='yes'/>
    <vcpu id='9' enabled='yes' hotpluggable='yes' order='7'/>
  </vcpus>


2. start guest:

# virsh start r7
Domain r7 started

3. show vcpuinfo :

# virsh vcpuinfo r7
error: Disconnected from qemu:///system due to I/O error
error: End of file while reading data: Input/output error


Actual results:
libvirtd crashed when use vcpuinfo to a running guest with vcpus hotplug table

Expected results:

no error

Additional info:

valgrind:

==14569== Thread 10:
==14569== Invalid write of size 4
==14569==    at 0x22086422: qemuDomainHelperGetVcpus (qemu_driver.c:1485)
==14569==    by 0x22086BE4: qemuDomainGetVcpus (qemu_driver.c:5402)
==14569==    by 0x55CB110: virDomainGetVcpus (libvirt-domain.c:7763)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpus (remote.c:2380)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpusHelper (remote_dispatch.h:6202)
==14569==    by 0x5634391: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==14569==    by 0x5634391: virNetServerProgramDispatch (virnetserverprogram.c:307)
==14569==    by 0x15C16C: virNetServerProcessMsg (virnetserver.c:148)
==14569==    by 0x15C16C: virNetServerHandleJob (virnetserver.c:169)
==14569==    by 0x5520A70: virThreadPoolWorker (virthreadpool.c:167)
==14569==    by 0x551FDF7: virThreadHelper (virthread.c:206)
==14569==    by 0x82E4DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==14569==    by 0x85F073C: clone (in /usr/lib64/libc-2.17.so)
==14569==  Address 0xe4d44e8 is 24 bytes before a block of size 80 alloc'd
==14569==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==14569==    by 0x54B2D23: virAlloc (viralloc.c:144)
==14569==    by 0x54CF7E3: virLastErrorObject (virerror.c:243)
==14569==    by 0x54D13C8: virResetLastError (virerror.c:415)
==14569==    by 0x55CB084: virDomainGetVcpus (libvirt-domain.c:7740)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpus (remote.c:2380)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpusHelper (remote_dispatch.h:6202)
==14569==    by 0x5634391: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==14569==    by 0x5634391: virNetServerProgramDispatch (virnetserverprogram.c:307)
==14569==    by 0x15C16C: virNetServerProcessMsg (virnetserver.c:148)
==14569==    by 0x15C16C: virNetServerHandleJob (virnetserver.c:169)
==14569==    by 0x5520A70: virThreadPoolWorker (virthreadpool.c:167)
==14569==    by 0x551FDF7: virThreadHelper (virthread.c:206)
==14569==    by 0x82E4DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==14569==    by 0x85F073C: clone (in /usr/lib64/libc-2.17.so)
==14569== 
==14569== Invalid write of size 4
==14569==    at 0x22086426: qemuDomainHelperGetVcpus (qemu_driver.c:1486)
==14569==    by 0x22086BE4: qemuDomainGetVcpus (qemu_driver.c:5402)
==14569==    by 0x55CB110: virDomainGetVcpus (libvirt-domain.c:7763)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpus (remote.c:2380)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpusHelper (remote_dispatch.h:6202)
==14569==    by 0x5634391: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==14569==    by 0x5634391: virNetServerProgramDispatch (virnetserverprogram.c:307)
==14569==    by 0x15C16C: virNetServerProcessMsg (virnetserver.c:148)
==14569==    by 0x15C16C: virNetServerHandleJob (virnetserver.c:169)
==14569==    by 0x5520A70: virThreadPoolWorker (virthreadpool.c:167)
==14569==    by 0x551FDF7: virThreadHelper (virthread.c:206)
==14569==    by 0x82E4DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==14569==    by 0x85F073C: clone (in /usr/lib64/libc-2.17.so)
==14569==  Address 0xe4d44ec is 20 bytes before a block of size 80 alloc'd
==14569==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==14569==    by 0x54B2D23: virAlloc (viralloc.c:144)
==14569==    by 0x54CF7E3: virLastErrorObject (virerror.c:243)
==14569==    by 0x54D13C8: virResetLastError (virerror.c:415)
==14569==    by 0x55CB084: virDomainGetVcpus (libvirt-domain.c:7740)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpus (remote.c:2380)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpusHelper (remote_dispatch.h:6202)
==14569==    by 0x5634391: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==14569==    by 0x5634391: virNetServerProgramDispatch (virnetserverprogram.c:307)
==14569==    by 0x15C16C: virNetServerProcessMsg (virnetserver.c:148)
==14569==    by 0x15C16C: virNetServerHandleJob (virnetserver.c:169)
==14569==    by 0x5520A70: virThreadPoolWorker (virthreadpool.c:167)
==14569==    by 0x551FDF7: virThreadHelper (virthread.c:206)
==14569==    by 0x82E4DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==14569==    by 0x85F073C: clone (in /usr/lib64/libc-2.17.so)
==14569== 
==14569== Invalid write of size 8
==14569==    at 0x2206B282: qemuGetProcessInfo (qemu_driver.c:1433)
==14569==    by 0x22086443: qemuDomainHelperGetVcpus (qemu_driver.c:1488)
==14569==    by 0x22086BE4: qemuDomainGetVcpus (qemu_driver.c:5402)
==14569==    by 0x55CB110: virDomainGetVcpus (libvirt-domain.c:7763)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpus (remote.c:2380)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpusHelper (remote_dispatch.h:6202)
==14569==    by 0x5634391: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==14569==    by 0x5634391: virNetServerProgramDispatch (virnetserverprogram.c:307)
==14569==    by 0x15C16C: virNetServerProcessMsg (virnetserver.c:148)
==14569==    by 0x15C16C: virNetServerHandleJob (virnetserver.c:169)
==14569==    by 0x5520A70: virThreadPoolWorker (virthreadpool.c:167)
==14569==    by 0x551FDF7: virThreadHelper (virthread.c:206)
==14569==    by 0x82E4DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==14569==    by 0x85F073C: clone (in /usr/lib64/libc-2.17.so)
==14569==  Address 0xe4d44f0 is 16 bytes before a block of size 80 alloc'd
==14569==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==14569==    by 0x54B2D23: virAlloc (viralloc.c:144)
==14569==    by 0x54CF7E3: virLastErrorObject (virerror.c:243)
==14569==    by 0x54D13C8: virResetLastError (virerror.c:415)
==14569==    by 0x55CB084: virDomainGetVcpus (libvirt-domain.c:7740)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpus (remote.c:2380)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpusHelper (remote_dispatch.h:6202)
==14569==    by 0x5634391: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==14569==    by 0x5634391: virNetServerProgramDispatch (virnetserverprogram.c:307)
==14569==    by 0x15C16C: virNetServerProcessMsg (virnetserver.c:148)
==14569==    by 0x15C16C: virNetServerHandleJob (virnetserver.c:169)
==14569==    by 0x5520A70: virThreadPoolWorker (virthreadpool.c:167)
==14569==    by 0x551FDF7: virThreadHelper (virthread.c:206)
==14569==    by 0x82E4DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==14569==    by 0x85F073C: clone (in /usr/lib64/libc-2.17.so)
==14569== 
==14569== Invalid write of size 4
==14569==    at 0x2206B28E: qemuGetProcessInfo (qemu_driver.c:1436)
==14569==    by 0x22086443: qemuDomainHelperGetVcpus (qemu_driver.c:1488)
==14569==    by 0x22086BE4: qemuDomainGetVcpus (qemu_driver.c:5402)
==14569==    by 0x55CB110: virDomainGetVcpus (libvirt-domain.c:7763)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpus (remote.c:2380)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpusHelper (remote_dispatch.h:6202)
==14569==    by 0x5634391: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==14569==    by 0x5634391: virNetServerProgramDispatch (virnetserverprogram.c:307)
==14569==    by 0x15C16C: virNetServerProcessMsg (virnetserver.c:148)
==14569==    by 0x15C16C: virNetServerHandleJob (virnetserver.c:169)
==14569==    by 0x5520A70: virThreadPoolWorker (virthreadpool.c:167)
==14569==    by 0x551FDF7: virThreadHelper (virthread.c:206)
==14569==    by 0x82E4DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==14569==    by 0x85F073C: clone (in /usr/lib64/libc-2.17.so)
==14569==  Address 0xe4d44f8 is 8 bytes before a block of size 80 alloc'd
==14569==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==14569==    by 0x54B2D23: virAlloc (viralloc.c:144)
==14569==    by 0x54CF7E3: virLastErrorObject (virerror.c:243)
==14569==    by 0x54D13C8: virResetLastError (virerror.c:415)
==14569==    by 0x55CB084: virDomainGetVcpus (libvirt-domain.c:7740)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpus (remote.c:2380)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpusHelper (remote_dispatch.h:6202)
==14569==    by 0x5634391: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==14569==    by 0x5634391: virNetServerProgramDispatch (virnetserverprogram.c:307)
==14569==    by 0x15C16C: virNetServerProcessMsg (virnetserver.c:148)
==14569==    by 0x15C16C: virNetServerHandleJob (virnetserver.c:169)
==14569==    by 0x5520A70: virThreadPoolWorker (virthreadpool.c:167)
==14569==    by 0x551FDF7: virThreadHelper (virthread.c:206)
==14569==    by 0x82E4DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==14569==    by 0x85F073C: clone (in /usr/lib64/libc-2.17.so)
==14569== 
==14569== Invalid write of size 1
==14569==    at 0x4C2DEA5: memset (vg_replace_strmem.c:1224)
==14569==    by 0x54B5576: UnknownInlinedFun (string3.h:84)
==14569==    by 0x54B5576: virBitmapToDataBuf (virbitmap.c:762)
==14569==    by 0x22086478: qemuDomainHelperGetVcpus (qemu_driver.c:1503)
==14569==    by 0x22086BE4: qemuDomainGetVcpus (qemu_driver.c:5402)
==14569==    by 0x55CB110: virDomainGetVcpus (libvirt-domain.c:7763)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpus (remote.c:2380)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpusHelper (remote_dispatch.h:6202)
==14569==    by 0x5634391: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==14569==    by 0x5634391: virNetServerProgramDispatch (virnetserverprogram.c:307)
==14569==    by 0x15C16C: virNetServerProcessMsg (virnetserver.c:148)
==14569==    by 0x15C16C: virNetServerHandleJob (virnetserver.c:169)
==14569==    by 0x5520A70: virThreadPoolWorker (virthreadpool.c:167)
==14569==    by 0x551FDF7: virThreadHelper (virthread.c:206)
==14569==    by 0x82E4DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==14569==    by 0x85F073C: clone (in /usr/lib64/libc-2.17.so)
==14569==  Address 0xe50da8b is 6 bytes after a block of size 21 alloc'd
==14569==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==14569==    by 0x54B2D8C: virAllocN (viralloc.c:191)
==14569==    by 0x14AD00: remoteDispatchDomainGetVcpus (remote.c:2377)
==14569==    by 0x14AD00: remoteDispatchDomainGetVcpusHelper (remote_dispatch.h:6202)
==14569==    by 0x5634391: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==14569==    by 0x5634391: virNetServerProgramDispatch (virnetserverprogram.c:307)
==14569==    by 0x15C16C: virNetServerProcessMsg (virnetserver.c:148)
==14569==    by 0x15C16C: virNetServerHandleJob (virnetserver.c:169)
==14569==    by 0x5520A70: virThreadPoolWorker (virthreadpool.c:167)
==14569==    by 0x551FDF7: virThreadHelper (virthread.c:206)
==14569==    by 0x82E4DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==14569==    by 0x85F073C: clone (in /usr/lib64/libc-2.17.so)
==14569== 
==14569== Invalid write of size 1
==14569==    at 0x54B55A8: virBitmapToDataBuf (virbitmap.c:771)
==14569==    by 0x22086478: qemuDomainHelperGetVcpus (qemu_driver.c:1503)
==14569==    by 0x22086BE4: qemuDomainGetVcpus (qemu_driver.c:5402)
==14569==    by 0x55CB110: virDomainGetVcpus (libvirt-domain.c:7763)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpus (remote.c:2380)
==14569==    by 0x14AD22: remoteDispatchDomainGetVcpusHelper (remote_dispatch.h:6202)
==14569==    by 0x5634391: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==14569==    by 0x5634391: virNetServerProgramDispatch (virnetserverprogram.c:307)
==14569==    by 0x15C16C: virNetServerProcessMsg (virnetserver.c:148)
==14569==    by 0x15C16C: virNetServerHandleJob (virnetserver.c:169)
==14569==    by 0x5520A70: virThreadPoolWorker (virthreadpool.c:167)
==14569==    by 0x551FDF7: virThreadHelper (virthread.c:206)
==14569==    by 0x82E4DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==14569==    by 0x85F073C: clone (in /usr/lib64/libc-2.17.so)
==14569==  Address 0xe50da8b is 6 bytes after a block of size 21 alloc'd
==14569==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==14569==    by 0x54B2D8C: virAllocN (viralloc.c:191)
==14569==    by 0x14AD00: remoteDispatchDomainGetVcpus (remote.c:2377)
==14569==    by 0x14AD00: remoteDispatchDomainGetVcpusHelper (remote_dispatch.h:6202)
==14569==    by 0x5634391: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==14569==    by 0x5634391: virNetServerProgramDispatch (virnetserverprogram.c:307)
==14569==    by 0x15C16C: virNetServerProcessMsg (virnetserver.c:148)
==14569==    by 0x15C16C: virNetServerHandleJob (virnetserver.c:169)
==14569==    by 0x5520A70: virThreadPoolWorker (virthreadpool.c:167)
==14569==    by 0x551FDF7: virThreadHelper (virthread.c:206)
==14569==    by 0x82E4DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==14569==    by 0x85F073C: clone (in /usr/lib64/libc-2.17.so)
==14569== 


gdb:

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fbf76863700 (LWP 14344)]
0x00007fbf841741d7 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007fbf841741d7 in raise () from /lib64/libc.so.6
#1  0x00007fbf841758c8 in abort () from /lib64/libc.so.6
#2  0x00007fbf841b3f07 in __libc_message () from /lib64/libc.so.6
#3  0x00007fbf841bb503 in _int_free () from /lib64/libc.so.6
#4  0x00007fbf84269562 in xdr_bytes () from /lib64/libc.so.6
#5  0x00007fbf87e945d5 in xdr_remote_domain_get_vcpus_ret (xdrs=0x7fbf76862b50, objp=0x7fbf50000b80) at remote/remote_protocol.c:2116
#6  0x00007fbf84268d65 in xdr_free () from /lib64/libc.so.6
#7  0x00007fbf8727347a in virNetServerProgramDispatchCall (msg=0x7fbf89703240, client=0x7fbf897032b0, server=<optimized out>, prog=0x7fbf896fee00) at rpc/virnetserverprogram.c:488
#8  virNetServerProgramDispatch (prog=0x7fbf896fee00, server=server@entry=0x7fbf896e9a80, client=0x7fbf897032b0, msg=0x7fbf89703240) at rpc/virnetserverprogram.c:307
#9  0x00007fbf87e9b16d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7fbf896e9a80) at rpc/virnetserver.c:148
#10 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7fbf896e9a80) at rpc/virnetserver.c:169
#11 0x00007fbf8715fa71 in virThreadPoolWorker (opaque=opaque@entry=0x7fbf896de620) at util/virthreadpool.c:167
#12 0x00007fbf8715edf8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#13 0x00007fbf8450edc5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007fbf8423673d in clone () from /lib64/libc.so.6
Comment 1 Peter Krempa 2016-08-26 15:37:22 UTC 
Fixed upstream:

commit 9b6e947b015026bc7bca9acc4283808459c4efd2
Author: Peter Krempa <pkrempa>
Date:   Thu Aug 25 14:53:06 2016 -0400

    qemu: driver: Fix qemuDomainHelperGetVcpus for sparse vcpu topologies
    
    ce43cca0e refactored the helper to prepare it for sparse topologies but
    forgot to fix the iterator used to fill the structures. This would
    result into a weirdly sparse populated array and possible out of bounds
    access and crash once sparse vcpu topologies were allowed.
Note You need to log in before you can comment on or make changes to this bug.