Bug 1433183

Summary: libvirtd crashed when querying persistent blkdeviotune config with groupname set
Product: Red Hat Enterprise Linux 7 Reporter: Fangge Jin <fjin>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: jiyan <jiyan>
Severity: high Docs Contact:
Priority: medium    
Version: 7.4CC: dyuan, fjin, lmen, pkrempa, rbalakri, xuzhang, yafu, zpeng
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-3.2.0-6.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 17:24:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
libvirtd log
none
backtrace for comment 8 none

Description Fangge Jin 2017-03-17 02:06:17 UTC
Created attachment 1263943 [details]
libvirtd log

Description of problem:
Prepare a guest with blkdeviotune groupname, query blkdeviotune when guest is shutoff or query blkdeviotune with --config when guest is running, libvirtd will crash.

Version-Release number of selected component:
libvirt-3.1.0-2.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
0. Prepare a guest xml with blkdeviotune group_name:
# cat /tmp/guest.xml
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/image-builder.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <iotune>
        <group_name>group1</group_name>
      </iotune>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
...

1. # virsh define /tmp/guest.xml

2. # while true; do virsh blkdeviotune guest vda ; sleep 2; done
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 0
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     : group1
total_bytes_sec_max_length: 0
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0

error: Disconnected from qemu:///system due to I/O error
error: Unable to get block I/O throttle parameters
error: End of file while reading data: Input/output error

total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 0
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     : group1
total_bytes_sec_max_length: 0
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0

error: Disconnected from qemu:///system due to I/O error
error: Unable to get block I/O throttle parameters
error: End of file while reading data: Input/output error


Actual results:
libvirtd crashed when querying blkdeviotune

Expected results:
libvirtd doesn't crash


Additional info:

Comment 2 Peter Krempa 2017-03-17 07:14:46 UTC
Please also attach the backtrace of all threads if libvirtd crashes.

Comment 3 Peter Krempa 2017-03-17 07:27:19 UTC
I managed to reproduce it. The problem is that the code steals the pointer from the definition rather than copying it. The RPC layer then frees it.

Comment 4 Peter Krempa 2017-03-17 10:15:50 UTC
Fixed upstream:

commit 4b57f765027c6e157a41a7445ec23359655be272
Author: Peter Krempa <pkrempa>
Date:   Fri Mar 17 08:43:27 2017 +0100

    qemu: Don't steal pointers from 'persistentDef' in qemuDomainGetBlockIoTune
    
    While the code path that queries the monitor allocates a separate copy
    of the 'group_name' string the path querying the config would not copy
    it. The call to virTypedParameterAssign would then steal the pointer
    (without clearing it) and the RPC layer freed it. Any subsequent call
    resulted into a crash.

Comment 8 Fangge Jin 2017-05-25 02:10:58 UTC
Hi Peter

I tested with latest build libvirt-3.2.0-6.virtcov.el7.x86_64, libvirtd crash when set --group-name and other blkdeviotune parameters together by virsh command:

# virsh blkdeviotune vm1 vda --total-bytes-sec 2000 --group-name fffff
error: Disconnected from qemu:///system due to I/O error
error: Unable to change block I/O throttle
error: End of file while reading data: Input/output error

As another colleague have tried before, this problem also exists with build libvirt-3.2.0-5.virtcov.el7.x86_64

Comment 9 Peter Krempa 2017-05-25 06:27:00 UTC
please post the backtrace

Comment 10 Fangge Jin 2017-05-25 07:18:04 UTC
Created attachment 1282140 [details]
backtrace for comment 8

Comment 11 Peter Krempa 2017-05-25 11:18:53 UTC
Indeed, there was another bug in the setter code path. Would you mind filing a bug for that too?

The specific way to reproduce this is to set something other than group name along with it. The typed parameter containing the group name needs to be second or further in the list.

Comment 12 Fangge Jin 2017-05-25 11:53:57 UTC
I file a new Bug 1455510 for this problem

Comment 13 jiyan 2017-06-02 10:23:09 UTC
Version-Release number of selected component:
libvirt-3.2.0-7.el7.x86_64
qemu-kvm-rhev-2.9.0-7.el7.x86_64
Kernel-3.10.0-675.el7.x86_64

Steps to verify:
1.Prepare guest.xml with blkdeviotune group_name, prepare guest1.xml with blkdeviotune group_name and 
# cat guest.xml |grep "<disk" -A 8
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/RHEL-7.4-x86_64-latest.qcow2'/>
      <target dev='vdb' bus='virtio'/>
      <iotune>
        <group_name>group1</group_name>
      </iotune>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>

# cat guest1.xml |grep "<disk" -A 9
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/RHEL-7.4-x86_64-latest.qcow2'/>
      <target dev='vdb' bus='virtio'/>
      <iotune>
        <group_name>group1</group_name>
        <total_bytes_sec>10000000</total_bytes_sec>
      </iotune>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>

Note:reffer to the bug ,https://bugzilla.redhat.com/show_bug.cgi?id=1433180,"Don't allow setting 'group_name' alone", so the guest1.xml needs to be verified.

2.define a vm though the guest.xml and query blkdeviotune
# virsh define guest.xml 
Domain guest defined from guest.xml

# virsh start guest
error: Failed to start domain guest
error: unsupported configuration: group_name can be configured only together with settings

# virsh blkdeviotune guest vdb 
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 0
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     : group1
total_bytes_sec_max_length: 0
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0

3.define a vm though the guest1.xml and query blkdeviotune
# virsh define guest1.xml 
Domain guest1 defined from guest1.xml

# virsh blkdeviotune guest vdb 
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 0
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     : group1
total_bytes_sec_max_length: 0
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0

Comment 14 errata-xmlrpc 2017-08-01 17:24:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 15 errata-xmlrpc 2017-08-02 00:03:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846