Red Hat Bugzilla – Bug 1351473
"virsh blkiotune" causes libvirtd crash
Last modified: 2016-11-03 14:48:06 EDT
Description of problem: "virsh blkiotune" causes libvirtd crash Version-Release number of selected component (if applicable): libvirt-1.3.5-1.el7.x86_64 qemu-kvm-rhev-2.6.0-5.el7.x86_64 kernel-3.10.0-382.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. # virsh list Id Name State ---------------------------------------------------- 2 virtlab_test running 2. # virsh dumpxml virtlab_test | grep blkiotune -A10 <blkiotune> <weight>123</weight> <device> <path>/dev/sdj</path> <read_bytes_sec>1024000</read_bytes_sec> <write_bytes_sec>1024000</write_bytes_sec> </device> </blkiotune> 3. # virsh blkiotune virtlab_test error: Disconnected from qemu:///system due to I/O error error: Unable to get blkio parameters error: End of file while reading data: Input/output error gdb backtrace: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f6a2d439700 (LWP 5566)] 0x00007f6a39b6033f in _int_free () from /lib64/libc.so.6 [Thread 0x7f6a2a433700 (LWP 5572) exited] [Thread 0x7f6a2d439700 (LWP 5566) exited] [Thread 0x7f6a2cc38700 (LWP 5567) exited] [Thread 0x7f6a2c437700 (LWP 5568) exited] [Thread 0x7f6a2bc36700 (LWP 5569) exited] [Thread 0x7f6a2b435700 (LWP 5570) exited] [Thread 0x7f6a2ac34700 (LWP 5571) exited] [Thread 0x7f6a29c32700 (LWP 5573) exited] [Thread 0x7f6a29431700 (LWP 5574) exited] [Thread 0x7f6a28c30700 (LWP 5575) exited] [Thread 0x7f6a22f9a700 (LWP 5577) exited] [Thread 0x7f6a21f98700 (LWP 5579) exited] [Thread 0x7f6a21797700 (LWP 5580) exited] [Thread 0x7f6a3d4a08c0 (LWP 5565) exited] [Thread 0x7f6a2379b700 (LWP 5576) exited] Program terminated with signal SIGSEGV, Segmentation fault. The program no longer exists. Actual results: libvirtd crashed in step 3 Expected results: no crash Additional info: libvirt-1.2.17-13 succeeds with step 3 # virsh blkiotune virtlab_test weight : 123 device_weight : device_read_iops_sec: device_write_iops_sec: device_read_bytes_sec: /dev/sdj,1024000 device_write_bytes_sec: /dev/sdj,1024000
Please attach the actual backtrace. The GDB part above doesn't provide any info.
Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f7866849700 (LWP 5379)] 0x00007f787377133f in _int_free () from /lib64/libc.so.6 (gdb) bt #0 0x00007f787377133f in _int_free () from /lib64/libc.so.6 #1 0x00007f78763f79aa in virFree (ptrptr=ptrptr@entry=0x7f7848000f78) at util/viralloc.c:582 #2 0x00007f7876461023 in virTypedParamsClear (params=<optimized out>, nparams=6) at util/virtypedparam.c:1298 #3 0x00007f787646104e in virTypedParamsFree (params=0x7f7848000da0, nparams=<optimized out>) at util/virtypedparam.c:1317 #4 0x00007f7877157b16 in remoteDispatchDomainGetBlkioParameters (server=<optimized out>, msg=0x7f7878944fa0, ret=0x7f7848000930, args=<optimized out>, rerr=0x7f7866848c10, client=<optimized out>) at remote.c:2561 #5 remoteDispatchDomainGetBlkioParametersHelper (server=<optimized out>, client=<optimized out>, msg=0x7f7878944fa0, rerr=0x7f7866848c10, args=<optimized out>, ret=0x7f7848000930) at remote_dispatch.h:4896 #6 0x00007f787656eef2 in virNetServerProgramDispatchCall (msg=0x7f7878944fa0, client=0x7f78789461d0, server=0x7f787892ce40, prog=0x7f78789403c0) at rpc/virnetserverprogram.c:437 #7 virNetServerProgramDispatch (prog=0x7f78789403c0, server=server@entry=0x7f787892ce40, client=0x7f78789461d0, msg=0x7f7878944fa0) at rpc/virnetserverprogram.c:307 #8 0x00007f787717ae0d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f787892ce40) at rpc/virnetserver.c:148 #9 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f787892ce40) at rpc/virnetserver.c:169 #10 0x00007f787645d581 in virThreadPoolWorker (opaque=opaque@entry=0x7f787892cb70) at util/virthreadpool.c:167 #11 0x00007f787645c908 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #12 0x00007f7873abedc5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f78737ebe2d in clone () from /lib64/libc.so.6
Thanks I was able to locate the problem and I've fixed it upstream: commit cbe4c049d87fe1f677668fbb51ba36647f3481d0 Author: Peter Krempa <pkrempa@redhat.com> Date: Thu Jun 30 14:33:24 2016 +0200 conf: Don't free the constructed string in virDomainGetBlkioParametersAssignFromDef virTypedParameterAssign steals the string rather than copying it into the typed parameter and thus freeing it leads to a crash when attempting to serialize the results. This was introduced in commit 9f50f6e2 and later made an universal helper in 32e6339c.
Verified version : libvirt-2.0.0-6.el7.x86_64 qemu-kvm-rhev-2.6.0-22.el7.x86_64 Steps: 1. start a guest like following # virsh list Id Name State ---------------------------------------------------- 16 vm1 running # virsh dumpxml vm1 | grep blkiotune -A 15 <blkiotune> <device> <path>/dev/sdd</path> <read_iops_sec>1024000</read_iops_sec> <write_iops_sec>1024000</write_iops_sec> </device> <device> <path>/dev/sde</path> <read_iops_sec>2048000</read_iops_sec> <read_bytes_sec>2048000</read_bytes_sec> <write_bytes_sec>2048000</write_bytes_sec> </device> </blkiotune> ...... 2. check blkiotune info # virsh blkiotune vm1 weight : 1000 device_weight : device_read_iops_sec: /dev/sdd,1024000,/dev/sde,2048000 device_write_iops_sec: /dev/sdd,1024000 device_read_bytes_sec: /dev/sde,2048000 device_write_bytes_sec: /dev/sde,2048000 set and get the value again # virsh blkiotune vm1 --device-read-iops-sec /dev/sde,1024000 [root@intel-e5530-8-1 ~]# virsh blkiotune vm1 weight : 1000 device_weight : device_read_iops_sec: /dev/sdd,1024000,/dev/sde,1024000 device_write_iops_sec: /dev/sdd,1024000 device_read_bytes_sec: /dev/sde,2048000 device_write_bytes_sec: /dev/sde,2048000 libvirtd will not crash, move to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-2577.html