Bug 1666605

Summary: libvirtd crash when hotplug 'block' type 'lun' device disk without 'sgio' setting to guest
Product: Red Hat Enterprise Linux 8 Reporter: yafu <yafu>
Component: libvirtAssignee: John Ferlan <jferlan>
Status: CLOSED CURRENTRELEASE QA Contact: yisun
Severity: unspecified Docs Contact:
Priority: high    
Version: 8.0CC: dyuan, dzheng, hhan, jdenemar, lcheng, rbalakri, xuzhang, yalzhang, yisun
Target Milestone: rcKeywords: Automation, Regression
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-4.5.0-19.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1669581 (view as bug list) Environment:
Last Closed: 2019-06-14 01:01:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1669581    

Description yafu 2019-01-16 07:56:25 UTC
Description of problem:
libvirtd crash when hotplug 'block' type 'lun' device disk without 'sgio' setting to guest.

Version-Release number of selected component (if applicable):
libvirt-4.5.0-18.module+el8+2691+dc742e5d.x86_64
qemu-kvm-2.12.0-57.module+el8+2683+02b3b955.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Prepare a 'block' type 'lun' device disk xml:
#cat disk.xml
<disk device="lun" type="block">
<source dev="/dev/sdb"/>
<driver name="qemu" type="raw"/>
<target bus="scsi" dev="sdb"/>
</disk>

2.Hotplug the disk to guest:
#virsh attach-device vm1 disk.xml
error: Disconnected from qemu:///system due to end of file
error: Failed to attach device from disk.xml
error: End of file while reading data: Input/output error

3.With following xml, the attach could be successful:
<disk device="lun" type="block" ***sgio="unfiltered"***>
    <source dev="/dev/sdb"/>
    <driver name="qemu" type="raw"/>
    <target bus="scsi" dev="sdb"/>
</disk>

Expected results:
Should hotplug block' type 'lun' device disk without 'sgio' setting to guest successfully.

Actual results:
libvirtd crash when hotplug 'block' type 'lun' device disk without 'sgio' setting to guest

Additional info:
1.It's a regression, it works well with libvirt-4.5.0-16.x86_64.

2.backtrace of libvirtd:
(gdb) t a a bt

Thread 17 (Thread 0x7ffb94837700 (LWP 12651)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b7420b520) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b7420b4d0, cond=0x563b7420b4f8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x563b7420b4f8, mutex=0x563b7420b4d0) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffbde2036a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7ffb86af5700 (LWP 12653)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b7420b520) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b7420b4d0, cond=0x563b7420b4f8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x563b7420b4f8, mutex=0x563b7420b4d0) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffbde2036a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7ffb95038700 (LWP 12650)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b7420b520) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b7420b4d0, cond=0x563b7420b4f8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x563b7420b4f8, mutex=0x563b7420b4d0) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffbde2036a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7ffb872f6700 (LWP 12652)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b7420b520) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b7420b4d0, cond=0x563b7420b4f8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x563b7420b4f8, mutex=0x563b7420b4d0) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffbde2036a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7ffba7fff700 (LWP 12646)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b741a1b04) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b741a1a10, cond=0x563b741a1ad8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x563b741a1ad8, mutex=0x563b741a1a10) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffbde203654 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7ffbc7b56700 (LWP 12639)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b741a1a60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b741a1a10, cond=0x563b741a1a38) at pthread_cond_wait.c:502
--Type <RET> for more, q to quit, c to continue without paging--
#2  __pthread_cond_wait (cond=0x563b741a1a38, mutex=0x563b741a1a10) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffbde2036a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7ffba6ffd700 (LWP 12648)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b741a1b04) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b741a1a10, cond=0x563b741a1ad8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x563b741a1ad8, mutex=0x563b741a1a10) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffbde203654 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7ffb848b5700 (LWP 12693)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7ffb6810f6d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7ffb6810f670, cond=0x7ffb6810f6b0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7ffb6810f6b0, mutex=0x7ffb6810f670) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffb9c0150e4 in udevEventHandleThread () from /usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so
#5  0x00007ffbde2028fa in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7ffbdecee380 (LWP 12638)):
#0  0x00007ffbdada9771 in __GI___poll (fds=0x563b741dbd10, nfds=11, timeout=5000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007ffbde1a5cb5 in virEventPollRunOnce () from /lib64/libvirt.so.0
#2  0x00007ffbde1a4835 in virEventRunDefaultImpl () from /lib64/libvirt.so.0
#3  0x00007ffbde2d2ead in virNetDaemonRun () from /lib64/libvirt.so.0
#4  0x0000563b73dc0756 in main ()

Thread 8 (Thread 0x7ffb95839700 (LWP 12649)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b7420b520) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b7420b4d0, cond=0x563b7420b4f8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x563b7420b4f8, mutex=0x563b7420b4d0) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffbde2036a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7ffbc5351700 (LWP 12644)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b741a1b00) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b741a1a10, cond=0x563b741a1ad8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x563b741a1ad8, mutex=0x563b741a1a10) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#4  0x00007ffbde203654 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7ffbc6b54700 (LWP 12641)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b741a1a60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b741a1a10, cond=0x563b741a1a38) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x563b741a1a38, mutex=0x563b741a1a10) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffbde2036a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7ffbc7355700 (LWP 12640)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b741a1a60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b741a1a10, cond=0x563b741a1a38) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x563b741a1a38, mutex=0x563b741a1a10) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffbde2036a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7ffba77fe700 (LWP 12647)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b741a1b04) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b741a1a10, cond=0x563b741a1ad8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x563b741a1ad8, mutex=0x563b741a1a10) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffbde203654 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7ffbc4b50700 (LWP 12645)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b741a1b00) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b741a1a10, cond=0x563b741a1ad8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x563b741a1ad8, mutex=0x563b741a1a10) at pthread_cond_wait.c:655
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffbde203654 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7ffbc5b52700 (LWP 12643)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x563b741a1a64) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563b741a1a10, cond=0x563b741a1a38) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x563b741a1a38, mutex=0x563b741a1a10) at pthread_cond_wait.c:655
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x00007ffbde202b6a in virCondWait () from /lib64/libvirt.so.0
#4  0x00007ffbde2036a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7ffbc6353700 (LWP 12642)):
#0  0x00007ffb958e7d7a in qemuSetUnprivSGIO () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#1  0x00007ffb958d9d92 in qemuDomainAttachDeviceDiskLive () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#2  0x00007ffb9594fce8 in qemuDomainAttachDeviceFlags () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#3  0x00007ffbde399d71 in virDomainAttachDevice () from /lib64/libvirt.so.0
#4  0x0000563b73ded4b2 in remoteDispatchDomainAttachDeviceHelper ()
#5  0x00007ffbde2cd174 in virNetServerProgramDispatch () from /lib64/libvirt.so.0
#6  0x00007ffbde2d364c in virNetServerHandleJob () from /lib64/libvirt.so.0
#7  0x00007ffbde2035c0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#8  0x00007ffbde2028cc in virThreadHelper () from /lib64/libvirt.so.0
#9  0x00007ffbdb0842de in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007ffbdadb49f3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment 3 yisun 2019-02-02 03:27:01 UTC
Verified with:
libvirt-4.5.0-20.module+el8+2724+8292f19c.x86_64
qemu-kvm-2.12.0-60.module+el8+2725+0ab65287.x86_64

And PASSED

1. no sgio setting
[root@hp-dl380g9-02 ~]# lsscsi
[0:0:0:0]    storage HP       P440ar           2.14  -
[0:1:0:0]    disk    HP       LOGICAL VOLUME   2.14  /dev/sda
[1:0:0:0]    disk    LIO-ORG  device.emulated  4.0   /dev/sdb
[2:0:0:0]    disk    LIO-ORG  device.logical-  4.0   /dev/sdc

[root@hp-dl380g9-02 ~]# cat disk
<disk device="lun" type="block">
    <source dev="/dev/sdc"/>
    <driver name="qemu" type="raw"/>
    <target bus="scsi" dev="sdb"/>
</disk>

[root@hp-dl380g9-02 ~]# virsh attach-device avocado-vt-vm1 disk
Device attached successfully

[root@hp-dl380g9-02 ~]# cat /sys/dev/block/*/device/unpriv_sgio
0
0
0

[root@hp-dl380g9-02 ~]# virsh domblklist avocado-vt-vm1
setlocale: No such file or directory
Target     Source
------------------------------------------------
vda        /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2
sdb        /dev/sdc

[root@hp-dl380g9-02 ~]# virsh detach-device avocado-vt-vm1 disk
Device detached successfully




2. sgio=unfiltered
[root@hp-dl380g9-02 ~]# cat disk_unfiltered
<disk device="lun" type="block" sgio="unfiltered">
    <source dev="/dev/sdc"/>
    <driver name="qemu" type="raw"/>
    <target bus="scsi" dev="sdb"/>
</disk>

[root@hp-dl380g9-02 ~]# virsh attach-device avocado-vt-vm1 disk_unfiltered
Device attached successfully


[root@hp-dl380g9-02 ~]# cat /sys/dev/block/*/device/unpriv_sgio
0
0
1

[root@hp-dl380g9-02 ~]# virsh domblklist avocado-vt-vm1
Target     Source
------------------------------------------------
vda        /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2
sdb        /dev/sdc

[root@hp-dl380g9-02 ~]# virsh detach-device avocado-vt-vm1 disk_unfiltered
Device detached successfully


3. sgio=filtered
[root@hp-dl380g9-02 ~]# cat disk_filtered
<disk device="lun" type="block" sgio="filtered">
    <source dev="/dev/sdc"/>
    <driver name="qemu" type="raw"/>
    <target bus="scsi" dev="sdb"/>
</disk>

[root@hp-dl380g9-02 ~]# virsh attach-device avocado-vt-vm1 disk_filtered
Device attached successfully

[root@hp-dl380g9-02 ~]# cat /sys/dev/block/*/device/unpriv_sgio
0
0
0

[root@hp-dl380g9-02 ~]# virsh domblklist avocado-vt-vm1
Target     Source
------------------------------------------------
vda        /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2
sdb        /dev/sdc

[root@hp-dl380g9-02 ~]# virsh detach-device avocado-vt-vm1 disk_filtered
Device detached successfully