1351473 – "virsh blkiotune" causes libvirtd crash

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1351473 - "virsh blkiotune" causes libvirtd crash

Summary: "virsh blkiotune" causes libvirtd crash

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	7.3
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Peter Krempa
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-06-30 07:24 UTC by yisun
Modified:	2016-11-03 18:48 UTC (History)
CC List:	6 users (show)
Fixed In Version:	libvirt-2.0.0-1.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-11-03 18:48:06 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2016:2577	0	normal	SHIPPED_LIVE	Moderate: libvirt security, bug fix, and enhancement update	2016-11-03 12:07:06 UTC

Description yisun 2016-06-30 07:24:09 UTC

Description of problem:
"virsh blkiotune" causes libvirtd crash 


Version-Release number of selected component (if applicable):
libvirt-1.3.5-1.el7.x86_64
qemu-kvm-rhev-2.6.0-5.el7.x86_64
kernel-3.10.0-382.el7.x86_64

How reproducible:
100%



Steps to Reproduce:
1. # virsh list
 Id    Name                           State
----------------------------------------------------
 2     virtlab_test                   running



2. # virsh dumpxml virtlab_test | grep blkiotune -A10
  <blkiotune>
    <weight>123</weight>
    <device>
      <path>/dev/sdj</path>
      <read_bytes_sec>1024000</read_bytes_sec>
      <write_bytes_sec>1024000</write_bytes_sec>
    </device>
  </blkiotune>



3. # virsh blkiotune virtlab_test
error: Disconnected from qemu:///system due to I/O error
error: Unable to get blkio parameters
error: End of file while reading data: Input/output error


gdb backtrace:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f6a2d439700 (LWP 5566)]
0x00007f6a39b6033f in _int_free () from /lib64/libc.so.6
[Thread 0x7f6a2a433700 (LWP 5572) exited]
[Thread 0x7f6a2d439700 (LWP 5566) exited]
[Thread 0x7f6a2cc38700 (LWP 5567) exited]
[Thread 0x7f6a2c437700 (LWP 5568) exited]
[Thread 0x7f6a2bc36700 (LWP 5569) exited]
[Thread 0x7f6a2b435700 (LWP 5570) exited]
[Thread 0x7f6a2ac34700 (LWP 5571) exited]
[Thread 0x7f6a29c32700 (LWP 5573) exited]
[Thread 0x7f6a29431700 (LWP 5574) exited]
[Thread 0x7f6a28c30700 (LWP 5575) exited]
[Thread 0x7f6a22f9a700 (LWP 5577) exited]
[Thread 0x7f6a21f98700 (LWP 5579) exited]
[Thread 0x7f6a21797700 (LWP 5580) exited]
[Thread 0x7f6a3d4a08c0 (LWP 5565) exited]
[Thread 0x7f6a2379b700 (LWP 5576) exited]
Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.


Actual results:
libvirtd crashed in step 3

Expected results:
no crash 

Additional info:
libvirt-1.2.17-13 succeeds with step 3
# virsh blkiotune virtlab_test
weight         : 123
device_weight  : 
device_read_iops_sec: 
device_write_iops_sec: 
device_read_bytes_sec: /dev/sdj,1024000
device_write_bytes_sec: /dev/sdj,1024000

Comment 2 Peter Krempa 2016-06-30 07:28:45 UTC

Please attach the actual backtrace. The GDB part above doesn't provide any info.

Comment 3 yisun 2016-06-30 10:31:46 UTC

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f7866849700 (LWP 5379)]
0x00007f787377133f in _int_free () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f787377133f in _int_free () from /lib64/libc.so.6
#1  0x00007f78763f79aa in virFree (ptrptr=ptrptr@entry=0x7f7848000f78) at util/viralloc.c:582
#2  0x00007f7876461023 in virTypedParamsClear (params=<optimized out>, nparams=6) at util/virtypedparam.c:1298
#3  0x00007f787646104e in virTypedParamsFree (params=0x7f7848000da0, nparams=<optimized out>) at util/virtypedparam.c:1317
#4  0x00007f7877157b16 in remoteDispatchDomainGetBlkioParameters (server=<optimized out>, msg=0x7f7878944fa0, ret=0x7f7848000930, 
    args=<optimized out>, rerr=0x7f7866848c10, client=<optimized out>) at remote.c:2561
#5  remoteDispatchDomainGetBlkioParametersHelper (server=<optimized out>, client=<optimized out>, msg=0x7f7878944fa0, rerr=0x7f7866848c10, 
    args=<optimized out>, ret=0x7f7848000930) at remote_dispatch.h:4896
#6  0x00007f787656eef2 in virNetServerProgramDispatchCall (msg=0x7f7878944fa0, client=0x7f78789461d0, server=0x7f787892ce40, prog=0x7f78789403c0)
    at rpc/virnetserverprogram.c:437
#7  virNetServerProgramDispatch (prog=0x7f78789403c0, server=server@entry=0x7f787892ce40, client=0x7f78789461d0, msg=0x7f7878944fa0)
    at rpc/virnetserverprogram.c:307
#8  0x00007f787717ae0d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f787892ce40)
    at rpc/virnetserver.c:148
#9  virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f787892ce40) at rpc/virnetserver.c:169
#10 0x00007f787645d581 in virThreadPoolWorker (opaque=opaque@entry=0x7f787892cb70) at util/virthreadpool.c:167
#11 0x00007f787645c908 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#12 0x00007f7873abedc5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f78737ebe2d in clone () from /lib64/libc.so.6

Comment 4 Peter Krempa 2016-06-30 13:12:27 UTC

Thanks I was able to locate the problem and I've fixed it upstream:

commit cbe4c049d87fe1f677668fbb51ba36647f3481d0
Author: Peter Krempa <pkrempa>
Date:   Thu Jun 30 14:33:24 2016 +0200

    conf: Don't free the constructed string in virDomainGetBlkioParametersAssignFromDef
    
    virTypedParameterAssign steals the string rather than copying it into
    the typed parameter and thus freeing it leads to a crash when attempting
    to serialize the results.
    
    This was introduced in commit 9f50f6e2 and later made an universal
    helper in 32e6339c.

Comment 6 Pei Zhang 2016-08-25 09:32:54 UTC

Verified version :
libvirt-2.0.0-6.el7.x86_64
qemu-kvm-rhev-2.6.0-22.el7.x86_64

Steps:
1. start a guest like following 
# virsh list 
 Id    Name                           State
----------------------------------------------------
 16    vm1                            running

# virsh dumpxml vm1 | grep blkiotune  -A 15
  <blkiotune>
    <device>
      <path>/dev/sdd</path>
      <read_iops_sec>1024000</read_iops_sec>
      <write_iops_sec>1024000</write_iops_sec>
    </device>
    <device>
      <path>/dev/sde</path>
      <read_iops_sec>2048000</read_iops_sec>
      <read_bytes_sec>2048000</read_bytes_sec>
      <write_bytes_sec>2048000</write_bytes_sec>
    </device>
  </blkiotune>
......
2. check blkiotune info
# virsh blkiotune vm1 
weight         : 1000
device_weight  : 
device_read_iops_sec: /dev/sdd,1024000,/dev/sde,2048000
device_write_iops_sec: /dev/sdd,1024000
device_read_bytes_sec: /dev/sde,2048000
device_write_bytes_sec: /dev/sde,2048000

set and get the value again 

# virsh blkiotune vm1 --device-read-iops-sec /dev/sde,1024000

[root@intel-e5530-8-1 ~]# virsh blkiotune vm1 
weight         : 1000
device_weight  : 
device_read_iops_sec: /dev/sdd,1024000,/dev/sde,1024000
device_write_iops_sec: /dev/sdd,1024000
device_read_bytes_sec: /dev/sde,2048000
device_write_bytes_sec: /dev/sde,2048000

libvirtd will not crash, move to verified.

Comment 8 errata-xmlrpc 2016-11-03 18:48:06 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html

Note You need to log in before you can comment on or make changes to this bug.