Bug 997906

Summary:	CVE-2013-5651 Running numatune with invalid nodeset parameter crash libvirtd. [rhel-7.0]
Product:	Red Hat Enterprise Linux 7	Reporter:	Peter Krempa <pkrempa>
Component:	libvirt	Assignee:	Peter Krempa <pkrempa>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	high	Docs Contact:
Priority:	medium
Version:	7.0	CC:	acathrow, ajia, dyuan, eblake, gsun, hliu, lsu, mzhan, pkrempa, pmatouse, zsong
Target Milestone:	rc
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	libvirt-1.1.1-3.el7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	997367	Environment:
Last Closed:	2014-06-13 12:09:59 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	997367
Bug Blocks:	1006493

Description Peter Krempa 2013-08-16 13:15:54 UTC

+++ This bug was initially created as a clone of Bug #997367 +++

Description of problem:
Running numatune with invalid nodeset parameter crash libvirtd.

Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux Server release 6.5 Beta
libvirt-1.1.1-2.el7

How reproducible:
always

Steps to Reproduce:
1. make sure a domain 'foo' is shut off.

2. change nodeset to a very large number.
# virsh numatune foo --nodeset 1000000000
Actual result:
error: Unable to change numa parameters
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the hypervisor
error: Failed to reconnect to the hypervisor

Expected result:
An error message or empty line without crashing libvirtd

3. check if libvirtd is crashed.
# ps aux | grep libvirtd

Additional Info:

# virsh nodeinfo
CPU model:           x86_64
CPU(s):              24
CPU frequency:       1596 MHz
CPU socket(s):       1
Core(s) per socket:  6
Thread(s) per core:  2
NUMA cell(s):        2
Memory size:         32834484 KiB

libvirtd backtrace:
Thread 11 (Thread 0x7f5244dac700 (LWP 10403)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x00007f525128bb46 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f525128c113 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f525128b969 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x00007f524fdb69d1 in start_thread (arg=0x7f5244dac700) at pthread_create.c:301
#5  0x00007f524f6fca8d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 10 (Thread 0x7f52425a8700 (LWP 10407)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x00007f525128bb46 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f525128c113 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f525128b969 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x00007f524fdb69d1 in start_thread (arg=0x7f52425a8700) at pthread_create.c:301
#5  0x00007f524f6fca8d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 9 (Thread 0x7f52411a6700 (LWP 10409)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x00007f525128bb46 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f525128c113 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f525128b969 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x00007f524fdb69d1 in start_thread (arg=0x7f52411a6700) at pthread_create.c:301
#5  0x00007f524f6fca8d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 8 (Thread 0x7f52407a5700 (LWP 10410)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x00007f525128bb46 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f525128c113 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f525128b969 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x00007f524fdb69d1 in start_thread (arg=0x7f52407a5700) at pthread_create.c:301
#5  0x00007f524f6fca8d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 7 (Thread 0x7f52443ab700 (LWP 10404)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x00007f525128bb46 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f525128c113 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f525128b969 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x00007f524fdb69d1 in start_thread (arg=0x7f52443ab700) at pthread_create.c:301
#5  0x00007f524f6fca8d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 6 (Thread 0x7f52439aa700 (LWP 10405)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x00007f525128bb46 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f525128c113 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f525128b969 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x00007f524fdb69d1 in start_thread (arg=0x7f52439aa700) at pthread_create.c:301
#5  0x00007f524f6fca8d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 5 (Thread 0x7f5242fa9700 (LWP 10406)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x00007f525128bb46 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f525128c113 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f525128b969 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x00007f524fdb69d1 in start_thread (arg=0x7f5242fa9700) at pthread_create.c:301
#5  0x00007f524f6fca8d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 4 (Thread 0x7f5241ba7700 (LWP 10408)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x00007f525128bb46 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f525128c113 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f525128b969 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x00007f524fdb69d1 in start_thread (arg=0x7f5241ba7700) at pthread_create.c:301
#5  0x00007f524f6fca8d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 3 (Thread 0x7f5251a5d860 (LWP 10400)):
#0  0x00007f524fdbd6fd in write () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f52512914ae in safewrite (fd=4, buf=0x1965c20, count=112) at util/util.c:130
#2  0x00007f525127f930 in virLogOutputToFd (category=<value optimized out>, priority=<value optimized out>, funcname=<value optimized out>, linenr=<value optimized out>, 
    timestamp=<value optimized out>, flags=0, str=0x1965050 "10400: debug : virEventPollRunOnce:614 : EVENT_POLL_RUN: nhandles=11 timeout=5000\n", data=0x4) at util/logging.c:846
#3  0x00007f52512802cf in virLogVMessage (category=0x7f52513d000f "trace.util/event_poll.c", priority=<value optimized out>, funcname=0x7f52513d05a0 "virEventPollRunOnce", linenr=614, 
    flags=0, fmt=<value optimized out>, vargs=0x7fffd4511000) at util/logging.c:781
#4  0x00007f525128047c in virLogMessage (category=<value optimized out>, priority=<value optimized out>, funcname=<value optimized out>, linenr=<value optimized out>, 
    flags=<value optimized out>, fmt=<value optimized out>) at util/logging.c:688
#5  0x00007f5251279591 in virEventPollRunOnce () at util/event_poll.c:612
#6  0x00007f52512787e7 in virEventRunDefaultImpl () at util/event.c:247
#7  0x00007f5251366acd in virNetServerRun (srv=0x1960180) at rpc/virnetserver.c:748
#8  0x0000000000423cc7 in main (argc=<value optimized out>, argv=<value optimized out>) at libvirtd.c:1228

Thread 2 (Thread 0x7f52457ad700 (LWP 10402)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x00007f525128bb46 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x00007f525128c113 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x00007f525128b969 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x00007f524fdb69d1 in start_thread (arg=0x7f52457ad700) at pthread_create.c:301
#5  0x00007f524f6fca8d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7f52461ae700 (LWP 10401)):
#0  0x00007f525126e316 in virBitmapIsSet (str=<value optimized out>, sep=0 '\000', bitmap=0x7f52461ad9c8, bitmapSize=<value optimized out>) at util/bitmap.c:153
#1  virBitmapParse (str=<value optimized out>, sep=0 '\000', bitmap=0x7f52461ad9c8, bitmapSize=<value optimized out>) at util/bitmap.c:335
#2  0x0000000000466d95 in qemuDomainSetNumaParameters (dom=<value optimized out>, params=<value optimized out>, nparams=1, flags=2) at qemu/qemu_driver.c:8190
#3  0x00007f525132c3fd in virDomainSetNumaParameters (domain=0x7f52300025a0, params=0x7f5230002de0, nparams=1, flags=0) at libvirt.c:3975
#4  0x000000000042ad7e in remoteDispatchDomainSetNumaParameters (server=<value optimized out>, client=<value optimized out>, msg=<value optimized out>, rerr=0x7f52461adb80, 
    args=<value optimized out>, ret=<value optimized out>) at remote_dispatch.h:5505
#5  remoteDispatchDomainSetNumaParametersHelper (server=<value optimized out>, client=<value optimized out>, msg=<value optimized out>, rerr=0x7f52461adb80, args=<value optimized out>, 
    ret=<value optimized out>) at remote_dispatch.h:5475
#6  0x00007f5251368a42 in virNetServerProgramDispatchCall (prog=0x1968a70, server=0x1960180, client=0x1965700, msg=0x195cc10) at rpc/virnetserverprogram.c:431
#7  virNetServerProgramDispatch (prog=0x1968a70, server=0x1960180, client=0x1965700, msg=0x195cc10) at rpc/virnetserverprogram.c:304
#8  0x00007f525136728e in virNetServerProcessMsg (srv=<value optimized out>, client=0x1965700, prog=<value optimized out>, msg=0x195cc10) at rpc/virnetserver.c:170
#9  0x00007f525136792c in virNetServerHandleJob (jobOpaque=<value optimized out>, opaque=0x1960180) at rpc/virnetserver.c:191
#10 0x00007f525128c07c in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:144
#11 0x00007f525128b969 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#12 0x00007f524fdb69d1 in start_thread (arg=0x7f52461ae700) at pthread_create.c:301
#13 0x00007f524f6fca8d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

--- Additional comment from Alex Jia on 2013-08-15 12:32:54 CEST ---

(In reply to Hao Liu from comment #0)
> Steps to Reproduce:
> 1. make sure a domain 'foo' is shut off.

It's not necessary.

> libvirtd backtrace:
 
> Thread 1 (Thread 0x7f52461ae700 (LWP 10401)):
> #0  0x00007f525126e316 in virBitmapIsSet (str=<value optimized out>, sep=0
> '\000', bitmap=0x7f52461ad9c8, bitmapSize=<value optimized out>) at
> util/bitmap.c:153
> #1  virBitmapParse (str=<value optimized out>, sep=0 '\000',
> bitmap=0x7f52461ad9c8, bitmapSize=<value optimized out>) at util/bitmap.c:335
> #2  0x0000000000466d95 in qemuDomainSetNumaParameters (dom=<value optimized
> out>, params=<value optimized out>, nparams=1, flags=2) at
> qemu/qemu_driver.c:8190
> #3  0x00007f525132c3fd in virDomainSetNumaParameters (domain=0x7f52300025a0,
> params=0x7f5230002de0, nparams=1, flags=0) at libvirt.c:3975
> #4  0x000000000042ad7e in remoteDispatchDomainSetNumaParameters
> (server=<value optimized out>, client=<value optimized out>, msg=<value
> optimized out>, rerr=0x7f52461adb80, 
>     args=<value optimized out>, ret=<value optimized out>) at
> remote_dispatch.h:5505
> #5  remoteDispatchDomainSetNumaParametersHelper (server=<value optimized
> out>, client=<value optimized out>, msg=<value optimized out>,
> rerr=0x7f52461adb80, args=<value optimized out>, 
>     ret=<value optimized out>) at remote_dispatch.h:5475
> #6  0x00007f5251368a42 in virNetServerProgramDispatchCall (prog=0x1968a70,
> server=0x1960180, client=0x1965700, msg=0x195cc10) at
> rpc/virnetserverprogram.c:431
> #7  virNetServerProgramDispatch (prog=0x1968a70, server=0x1960180,
> client=0x1965700, msg=0x195cc10) at rpc/virnetserverprogram.c:304
> #8  0x00007f525136728e in virNetServerProcessMsg (srv=<value optimized out>,
> client=0x1965700, prog=<value optimized out>, msg=0x195cc10) at
> rpc/virnetserver.c:170
> #9  0x00007f525136792c in virNetServerHandleJob (jobOpaque=<value optimized
> out>, opaque=0x1960180) at rpc/virnetserver.c:191
> #10 0x00007f525128c07c in virThreadPoolWorker (opaque=<value optimized out>)
> at util/threadpool.c:144
> #11 0x00007f525128b969 in virThreadHelper (data=<value optimized out>) at
> util/threads-pthread.c:161
> #12 0x00007f524fdb69d1 in start_thread (arg=0x7f52461ae700) at
> pthread_create.c:301
> #13 0x00007f524f6fca8d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

This should be enough to debug.

--- Additional comment from Peter Krempa on 2013-08-16 14:43:48 CEST ---

Fixed upstream:

commit 7efd5fd1b0225436cbbae1181ab41c2d3eca43f9
Author: Peter Krempa <pkrempa>
Date:   Fri Aug 16 12:13:27 2013 +0200

    virbitmaptest: Add test for out of bounds condition
    
    Previous patch fixed an issue where, when parsing a bitmap from the
    string, the bounds of the bitmap weren't checked. That flaw resulted into
    crashes. This test tests that case to avoid it in the future.

commit 536d38128e749fa5b149b9e168224280c3ad348c
Author: Peter Krempa <pkrempa>
Date:   Fri Aug 16 12:12:55 2013 +0200

    virbitmaptest: Fix function header formatting

commit 47b9127e883677a0d60d767030a147450e919a25
Author: Peter Krempa <pkrempa>
Date:   Fri Aug 16 12:22:32 2013 +0200

    virbitmap: Refactor virBitmapParse to avoid access beyond bounds of array
    
    The virBitmapParse function was calling virBitmapIsSet() function that
    requires the caller to check the bounds of the bitmap without checking
    them. This resulted into crashes when parsing a bitmap string that was
    exceeding the bounds used as argument.
    
    This patch refactors the function to use virBitmapSetBit without
    checking if the bit is set (this function does the checks internally)
    and then counts the bits in the bitmap afterwards (instead of keeping
    track while parsing the string).
    
    This patch also changes the "parse_error" label to a more common
    "error".
    
    The refactor should also get rid of the need to call sa_assert on the
    returned variable as the callpath should allow coverity to infer the
    possible return values.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=997367
    
    Thanks to Alex Jia for tracking down the issue. This issue is introduced
    by commit 0fc8909.

Comment 2 Wayne Sun 2013-09-02 07:25:00 UTC

pkgs:
libvirt-1.1.1-3.el7.x86_64
qemu-kvm-1.5.2-3.el7.x86_64
kernel-3.10.0-3.el7.x86_64

# numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 2 4 6
node 0 size: 8157 MB
node 0 free: 5775 MB
node 1 cpus: 8 10 12 14
node 1 size: 4096 MB
node 1 free: 2418 MB
node 2 cpus: 9 11 13 15
node 2 size: 4096 MB
node 2 free: 2307 MB
node 3 cpus: 1 3 5 7
node 3 size: 8191 MB
node 3 free: 7350 MB
node distances:
node   0   1   2   3 
  0:  10  20  20  20 
  1:  20  10  20  20 
  2:  20  20  10  20 
  3:  20  20  20  10 

steps:
# virsh list
 Id    Name                           State
----------------------------------------------------
 8     kvm-rhel6.4-x86_64-qcow2-virtio running

# virsh numatune kvm-rhel6.4-x86_64-qcow2-virtio --nodeset 1000000000
error: Unable to change numa parameters
error: internal error: Failed to parse nodeset

no crash of libvirtd.

# virsh numatune kvm-rhel6.4-x86_64-qcow2-virtio --nodeset sa
error: Unable to change numa parameters
error: internal error: Failed to parse nodeset

# virsh numatune kvm-rhel6.4-x86_64-qcow2-virtio --nodeset -1
error: Unable to change numa parameters
error: internal error: Failed to parse nodeset

# virsh numatune kvm-rhel6.4-x86_64-qcow2-virtio --nodeset 11
error: Unable to change numa parameters
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dkvm\x2drhel6.4\x2dx86_64\x2dqcow2\x2dvirtio.scope/cpuset.mems': Invalid argument

# virsh numatune kvm-rhel6.4-x86_64-qcow2-virtio --nodeset 0
error: Unable to change numa parameters
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dkvm\x2drhel6.4\x2dx86_64\x2dqcow2\x2dvirtio.scope/cpuset.mems': Device or resource busy
(kernel bug 955489)

this is fixed

Comment 3 Ludek Smid 2014-06-13 12:09:59 UTC

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.