Bug 1131819

Summary: Libvirtd crash while set blkdeviotune with the hotplug disk and specify the --config option
Product: Red Hat Enterprise Linux 7 Reporter: zhenfeng wang <zhwang>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.1CC: dyuan, lhuang, mzhan, pkrempa, rbalakri, vivianzhang, ydu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.2.8-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1131821 (view as bug list) Environment:
Last Closed: 2015-03-05 07:42:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1131821    
Attachments:
Description Flags
The libvirtd crash dump info none

Description zhenfeng wang 2014-08-20 06:36:55 UTC 
Description of problem:
Libvirtd crash while set blkdeviotune with the hotplug disk and specify the --config option

Version-Release number of selected component (if applicable):
libvirt-1.2.7-1.el7.x86_64
qemu-kvm-rhev-2.1.0-1.el7.x86_64
kernel-3.10.0-142.el7.x86_64


How reproducible:
100%

Steps to Reproduce:
1.Prepare a running guest
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     rhel6                          shut off

2.Create a img
#qemu-img create /var/lib/libvirt/images/test.img 1G

3.Attach the disk to the guest
#virsh attach-disk rhel6 /var/lib/libvirt/images/test.img vdb

4.Set io parameters for the vdb and specify the --config option, the libvirtd crashs

Check the libvirtd pid before the operation
# ps aux|grep libvirtd
root     26323  4.5  0.2 540600 17784 ?        Ssl  10:51   0:00 /usr/sbin/libvirtd


# virsh blkdeviotune rhel6 vdb --write-iops-sec 10 --config
error: Unable to change block I/O throttle
error: End of file while reading data: Input/output error
error: Failed to reconnect to the hypervisor

re-check the libvirtd pid, the pid changes

# ps aux|grep libvirtd
root     30602  0.2  0.2 614332 17852 ?        Ssl  11:03   0:00 /usr/sbin/libvirtd

5.The coredump info was in the attachment

Actual results:
Libvirtd crash while set blkdeviotune with the hotplug disk and specify the --config option

Expected results:
shouldn't crash
Comment 1 zhenfeng wang 2014-08-20 06:37:44 UTC 
Created attachment 928632 [details]
The libvirtd crash dump info
Comment 3 Peter Krempa 2014-08-20 09:10:05 UTC 
Fixed upstream:

commit 653137eb957a278b556c6226424aad5395a38a61
Author: Peter Krempa <pkrempa>
Date:   Wed Aug 20 10:49:49 2014 +0200

    qemu: blkiotune: Avoid accessing non-existing disk configuration
    
    When a user would try changing the persistent IO tuning settings for a
    disk that was hotplugged to a vm in a transient way, the
    qemuDomainSetBlockIoTune API would use the same index for both the
    live and config disk array. The disk was missing from the config array
    though causing a crash of libvirtd.
    
    To fix the issue, determine the indexes separately.

v1.2.7-180-g653137e
Comment 5 zhengqin 2014-09-10 06:31:48 UTC 
Verified with build libvirt-1.2.8-1.el7.x86_64, here are verify steps:

[root@rhel7-a1 ~]# qemu-img create /var/lib/libvirt/images/nfs/test.img 1G
Formatting '/var/lib/libvirt/images/nfs/test.img', fmt=raw size=1073741824 

[root@rhel7-a1 ~]# virsh start rhel6
Domain rhel6 started

[root@rhel7-a1 ~]# virsh attach-disk rhel6 /var/lib/libvirt/images/nfs/test.img vdb
Disk attached successfully

[root@rhel7-a1 ~]# ps aux | grep libvirtd
root      1269  0.0  0.9 1075152 18440 ?       Ssl  02:17   0:00 /usr/sbin/libvirtd

[root@rhel7-a1 ~]# virsh blkdeviotune rhel6 vdb --write-iops-sec 10 --config
error: Unable to change block I/O throttle
error: invalid argument: missing persistent configuration for disk 'vdb'


[root@rhel7-a1 ~]# virsh blkdeviotune rhel6 vdb --write-iops-sec 10 

[root@rhel7-a1 ~]# virsh blkdeviotune rhel6 vdb 
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 10

[root@rhel7-a1 ~]# ps aux | grep libvirtd
root      1269  0.0  0.9 1075152 18516 ?       Ssl  02:17   0:00 /usr/sbin/libvirtd

[root@rhel7-a1 ~]# systemctl status  libvirtd.service
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: active (running) since Wed 2014-09-10 02:17:51 EDT; 13min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 1269 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           ├─1269 /usr/sbin/libvirtd
           ├─2421 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default...
           └─2422 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default...
Comment 6 vivian zhang 2014-12-01 03:29:20 UTC 
I can produce this bud on build libvirt-1.2.7-1.el7.x86_64

verify it on build
libvirt-1.2.8-9.el7.x86_64
qemu-kvm-rhev-2.1.2-14.el7.x86_64
3.10.0-208.el7.x86_64

verify steps

1. prepare a normal guest in running status
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 74    rhel7new                       running

2. create a img and attach it to guest success

# qemu-img create /var/lib/libvirt/images/test1.img 1G
Formatting '/var/lib/libvirt/images/test1.img', fmt=raw size=1073741824

# virsh attach-disk rhel7new /var/lib/libvirt/images/test1.img vdb
Disk attached successfully

# virsh domblklist rhel7new
Target     Source
------------------------------------------------
hda        /var/lib/libvirt/images/rhel7new.qcow2
vdb        /var/lib/libvirt/images/test1.img

3. check libvirtd process id firstly
#ps aux |grep libvirtd
root     15315  1.2  0.2 1135052 20944 ?       Ssl  11:01   0:06 /usr/sbin/libvirtd --listen
root     15829  0.0  0.0 112644   956 pts/4    S+   11:09   0:00 grep --color=auto libvirtd

4. Set io parameters for the vdb and specify the --config option

# virsh blkdeviotune rhel7new vdb --config
error: Unable to get block I/O throttle parameters
error: invalid argument: disk 'vdb' was not found in the domain config

# virsh blkdeviotune rhel7new vdb --write-iops-sec 10 --config
error: Unable to change block I/O throttle
error: invalid argument: missing persistent configuration for disk 'vdb'

# virsh blkdeviotune rhel7new vdb --read-iops-sec 10 --config
error: Unable to change block I/O throttle
error: invalid argument: missing persistent configuration for disk 'vdb'

5. check libvirtd pid, do not crash
# ps aux |grep libvirtd
root     15315  0.7  0.2 1135052 20980 ?       Ssl  11:01   0:06 /usr/sbin/libvirtd --listen
root     16072  0.0  0.0 112644   960 pts/4    S+   11:14   0:00 grep --color=auto libvirtd

# service libvirtd status
Redirecting to /bin/systemctl status  libvirtd.service
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: active (running) since Mon 2014-12-01 11:01:47 HKT; 13min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 15315 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           ├─ 4028 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --...
           ├─ 4029 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --...
           ├─11753 /usr/sbin/glusterfs --direct-io-mode=1 --volfile-server=10.66.106....
           └─15315 /usr/sbin/libvirtd --listen

Dec 01 11:03:28 server.englab.nay.redhat.com libvirtd[15315]: 2014-12-01 03:03:28.783...
Dec 01 11:03:28 server.englab.nay.redhat.com libvirtd[15315]: 2014-12-01 03:03:28.783...
Dec 01 11:03:28 server.englab.nay.redhat.com libvirtd[15315]: 2014-12-01 03:03:28.783...
Dec 01 11:03:28 server.englab.nay.redhat.com libvirtd[15315]: 2014-12-01 03:03:28.783...
Dec 01 11:03:28 server.englab.nay.redhat.com libvirtd[15315]: 2014-12-01 03:03:28.783...
Dec 01 11:03:28 server.englab.nay.redhat.com libvirtd[15315]: 2014-12-01 03:03:28.783...
Dec 01 11:03:28 server.englab.nay.redhat.com libvirtd[15315]: 2014-12-01 03:03:28.783...
Dec 01 11:03:28 server.englab.nay.redhat.com libvirtd[15315]: 2014-12-01 03:03:28.783...
Dec 01 11:03:28 server.englab.nay.redhat.com libvirtd[15315]: 2014-12-01 03:03:28.783...
Dec 01 11:03:28 server.englab.nay.redhat.com libvirtd[15315]: 2014-12-01 03:03:28.783...
Hint: Some lines were ellipsized, use -l to show in full.

6. Set io parameters for the vdb and specify the --live/--current option

# virsh blkdeviotune rhel7new vdb --read-iops-sec 10 --current

# virsh blkdeviotune rhel7new vdb
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 10
write_iops_sec : 0

# virsh blkdeviotune rhel7new vdb --write-iops-sec 10 --live

[root@server ~]# virsh blkdeviotune rhel7new vdb
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 10

[root@server ~]# virsh blkdeviotune rhel7new vdb --read-iops-sec 10 --live

[root@server ~]# virsh blkdeviotune rhel7new vdb
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 10
write_iops_sec : 0
# virsh blkdeviotune rhel7new vdb --read-iops-sec 10 --total-bytes-sec 10 --live

# virsh blkdeviotune rhel7new vdb
total_bytes_sec: 10
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 10
write_iops_sec : 0

7. destroy guest, and start guest again, login guest work well
and vdb disk dismiss, libvirtd do not crash
# virsh domblklist rhel7new
Target     Source
------------------------------------------------
hda        /var/lib/libvirt/images/rhel7new.qcow2

# ps aux |grep libvirtd
root     15315  0.4  0.2 1135196 21084 ?       Ssl  11:01   0:06 /usr/sbin/libvirtd --listen
root     16579  0.0  0.0 112644   960 pts/4    S+   11:25   0:00 grep --color=auto libvirtd


move to verified
Comment 8 errata-xmlrpc 2015-03-05 07:42:50 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html