RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1433183 - libvirtd crashed when querying persistent blkdeviotune config with groupname set
Summary: libvirtd crashed when querying persistent blkdeviotune config with groupname set
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.4
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: jiyan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-17 02:06 UTC by Fangge Jin
Modified: 2017-08-02 00:03 UTC (History)
8 users (show)

Fixed In Version: libvirt-3.2.0-6.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 17:24:15 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
libvirtd log (162.85 KB, application/x-gzip)
2017-03-17 02:06 UTC, Fangge Jin 		no flags Details
backtrace for comment 8 (10.69 KB, text/plain)
2017-05-25 07:18 UTC, Fangge Jin 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1846 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2017-08-01 18:02:50 UTC

Description Fangge Jin 2017-03-17 02:06:17 UTC 
Created attachment 1263943 [details]
libvirtd log

Description of problem:
Prepare a guest with blkdeviotune groupname, query blkdeviotune when guest is shutoff or query blkdeviotune with --config when guest is running, libvirtd will crash.

Version-Release number of selected component:
libvirt-3.1.0-2.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
0. Prepare a guest xml with blkdeviotune group_name:
# cat /tmp/guest.xml
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/image-builder.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <iotune>
        <group_name>group1</group_name>
      </iotune>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
...

1. # virsh define /tmp/guest.xml

2. # while true; do virsh blkdeviotune guest vda ; sleep 2; done
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 0
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     : group1
total_bytes_sec_max_length: 0
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0

error: Disconnected from qemu:///system due to I/O error
error: Unable to get block I/O throttle parameters
error: End of file while reading data: Input/output error

total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 0
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     : group1
total_bytes_sec_max_length: 0
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0

error: Disconnected from qemu:///system due to I/O error
error: Unable to get block I/O throttle parameters
error: End of file while reading data: Input/output error


Actual results:
libvirtd crashed when querying blkdeviotune

Expected results:
libvirtd doesn't crash


Additional info:
Comment 2 Peter Krempa 2017-03-17 07:14:46 UTC 
Please also attach the backtrace of all threads if libvirtd crashes.
Comment 3 Peter Krempa 2017-03-17 07:27:19 UTC 
I managed to reproduce it. The problem is that the code steals the pointer from the definition rather than copying it. The RPC layer then frees it.
Comment 4 Peter Krempa 2017-03-17 10:15:50 UTC 
Fixed upstream:

commit 4b57f765027c6e157a41a7445ec23359655be272
Author: Peter Krempa <pkrempa>
Date:   Fri Mar 17 08:43:27 2017 +0100

    qemu: Don't steal pointers from 'persistentDef' in qemuDomainGetBlockIoTune
    
    While the code path that queries the monitor allocates a separate copy
    of the 'group_name' string the path querying the config would not copy
    it. The call to virTypedParameterAssign would then steal the pointer
    (without clearing it) and the RPC layer freed it. Any subsequent call
    resulted into a crash.
Comment 8 Fangge Jin 2017-05-25 02:10:58 UTC 
Hi Peter

I tested with latest build libvirt-3.2.0-6.virtcov.el7.x86_64, libvirtd crash when set --group-name and other blkdeviotune parameters together by virsh command:

# virsh blkdeviotune vm1 vda --total-bytes-sec 2000 --group-name fffff
error: Disconnected from qemu:///system due to I/O error
error: Unable to change block I/O throttle
error: End of file while reading data: Input/output error

As another colleague have tried before, this problem also exists with build libvirt-3.2.0-5.virtcov.el7.x86_64
Comment 9 Peter Krempa 2017-05-25 06:27:00 UTC 
please post the backtrace
Comment 10 Fangge Jin 2017-05-25 07:18:04 UTC 
Created attachment 1282140 [details]
backtrace for comment 8
Comment 11 Peter Krempa 2017-05-25 11:18:53 UTC 
Indeed, there was another bug in the setter code path. Would you mind filing a bug for that too?

The specific way to reproduce this is to set something other than group name along with it. The typed parameter containing the group name needs to be second or further in the list.
Comment 12 Fangge Jin 2017-05-25 11:53:57 UTC 
I file a new Bug 1455510 for this problem
Comment 13 jiyan 2017-06-02 10:23:09 UTC 
Version-Release number of selected component:
libvirt-3.2.0-7.el7.x86_64
qemu-kvm-rhev-2.9.0-7.el7.x86_64
Kernel-3.10.0-675.el7.x86_64

Steps to verify:
1.Prepare guest.xml with blkdeviotune group_name, prepare guest1.xml with blkdeviotune group_name and 
# cat guest.xml |grep "<disk" -A 8
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/RHEL-7.4-x86_64-latest.qcow2'/>
      <target dev='vdb' bus='virtio'/>
      <iotune>
        <group_name>group1</group_name>
      </iotune>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>

# cat guest1.xml |grep "<disk" -A 9
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/RHEL-7.4-x86_64-latest.qcow2'/>
      <target dev='vdb' bus='virtio'/>
      <iotune>
        <group_name>group1</group_name>
        <total_bytes_sec>10000000</total_bytes_sec>
      </iotune>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>

Note:reffer to the bug ,https://bugzilla.redhat.com/show_bug.cgi?id=1433180,"Don't allow setting 'group_name' alone", so the guest1.xml needs to be verified.

2.define a vm though the guest.xml and query blkdeviotune
# virsh define guest.xml 
Domain guest defined from guest.xml

# virsh start guest
error: Failed to start domain guest
error: unsupported configuration: group_name can be configured only together with settings

# virsh blkdeviotune guest vdb 
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 0
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     : group1
total_bytes_sec_max_length: 0
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0

3.define a vm though the guest1.xml and query blkdeviotune
# virsh define guest1.xml 
Domain guest1 defined from guest1.xml

# virsh blkdeviotune guest vdb 
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 0
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     : group1
total_bytes_sec_max_length: 0
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0
Comment 14 errata-xmlrpc 2017-08-01 17:24:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 15 errata-xmlrpc 2017-08-02 00:03:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Note You need to log in before you can comment on or make changes to this bug.