Bug 1428810 - 'Segmentation fault (core dumped)' after hot unplug one disk in a throttle group AND do guest system reset
Summary: 'Segmentation fault (core dumped)' after hot unplug one disk in a throttle gr...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Eric Blake
QA Contact: Gu Nini
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-03 12:11 UTC by Gu Nini
Modified: 2017-10-25 09:01 UTC (History)
14 users (show)

Fixed In Version: qemu-kvm-rhev-2.9.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-02 03:39:56 UTC


Attachments (Terms of Use)
GDB debug info-03032017 (2.00 KB, text/plain)
2017-03-03 12:11 UTC, Gu Nini
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:2392 normal SHIPPED_LIVE Important: qemu-kvm-rhev security, bug fix, and enhancement update 2017-08-01 20:04:36 UTC

Description Gu Nini 2017-03-03 12:11:43 UTC
Created attachment 1259527 [details]
GDB debug info-03032017

Description of problem:
Boot up a guest with 2 data disks in the same io throttling group. Hot unplug one of the disks, then do system reset for the guest. It's found 'Segmentation fault (core dumped)' to the guest.

Version-Release number of selected component (if applicable):
Host kernel: 3.10.0-561.el7.x86_64
Guest kernel: 3.10.0-514.10.2.el7.x86_64
Qemu-kvm-rhev: qemu-kvm-rhev-2.8.0-5.el7.x86_64


How reproducible:
100%

Steps to Reproduce:
1. Boot up a guest with 2 data disks in the same io throttling group:

    -drive id=drive_image2,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/hd1,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image2,drive=drive_image2 \
    -drive id=drive_image3,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/hd2,bps=512000,iops=100,group-foo \
    -device scsi-hd,id=image3,drive=drive_image3\

2. After the guest boots up, hot unplug disk image3 in qmp:

{"execute":"device_del","arguments":{"id":"image3"}}

3. Do guest system reset in qmp:

{"execute":"system_reset"} 


Actual results:
The guest 'Segmentation fault (core dumped)':

QEMU 2.8.0 monitor - type 'help' for more information
(qemu) 
(qemu) 
(qemu) 
(qemu) Segmentation fault (core dumped)


Expected results:
The guest could boot up well after the system reset

Additional info:
1. Failed to reproduce the bug on qemu2.6 with following packages, so set it as regression:
Host kernel: 3.10.0-514.6.1.el7.x86_64
Qemu-kvm-rhev: qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64
Guest kernel: 3.10.0-514.10.2.el7.x86_64
2. Please refer to the attached gdb debug info 'GDB debug info-03032017' AND core file 'core.29584' in nfs server 10.73.194.27:/vol/s2coredump for details

Comment 3 Eric Blake 2017-03-31 21:36:06 UTC
I've reproduced the problem with upstream qemu 2.9-rc2:
$ ./x86_64-softmmu/qemu-system-x86_64 -nodefaults -nographic -qmp stdio -device virtio-scsi-pci,bus=pci.0  -drive id=drive_image2,if=none,format=raw,file=file2,bps=512000,iops=100,group=foo -device scsi-hd,id=image2,drive=drive_image2 -drive id=drive_image3,if=none,format=raw,file=file3,bps=512000,iops=100,group=foo -device scsi-hd,id=image3,drive=drive_image3
{"QMP": {"version": {"qemu": {"micro": 92, "minor": 8, "major": 2}, "package": " (v2.9.0-rc2-33-gb14e856)"}, "capabilities": []}}
{'execute':'qmp_capabilities'}
{"return": {}}
{'execute':'device_del','arguments':{'id':'image3'}}
{"timestamp": {"seconds": 1490996087, "microseconds": 252990}, "event": "DEVICE_DELETED", "data": {"device": "image3", "path": "/machine/peripheral/image3"}}
{"return": {}}
{'execute':'system_reset'}
{"return": {}}
{"timestamp": {"seconds": 1490996101, "microseconds": 719720}, "event": "RESET"}
{"timestamp": {"seconds": 1490996101, "microseconds": 770165}, "event": "RESET"}
Segmentation fault (core dumped)

Now investigating to see if I can find root cause and patch...

Comment 4 Eric Blake 2017-04-06 19:10:32 UTC
Upstream patch proposed:
https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg00245.html

Comment 5 Eric Blake 2017-04-11 20:00:55 UTC
Made it in qemu 2.9.0-rc4:

commit 1606e4cf8a976513ecac70ad6642a7ec45744cf5
Author: Eric Blake <eblake@redhat.com>
Date:   Thu Apr 6 14:08:47 2017 -0500

    throttle: Remove block from group on hot-unplug
    
    When a block device that is part of a throttle group is hot-unplugged,
    we forgot to remove it from the throttle group. This leaves stale
    memory around, and causes an easily reproducible crash:
    
    $ ./x86_64-softmmu/qemu-system-x86_64 -nodefaults -nographic -qmp stdio \
    -device virtio-scsi-pci,bus=pci.0 -drive \
    id=drive_image2,if=none,format=raw,file=file2,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image2,drive=drive_image2 -drive \
    id=drive_image3,if=none,format=raw,file=file3,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image3,drive=drive_image3
    {'execute':'qmp_capabilities'}
    {'execute':'device_del','arguments':{'id':'image3'}}
    {'execute':'system_reset'}
    
    Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1428810
    
    Suggested-by: Alberto Garcia <berto@igalia.com>
    Signed-off-by: Eric Blake <eblake@redhat.com>
    Message-id: 20170406190847.29347-1-eblake@redhat.com
    Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
    Signed-off-by: Max Reitz <mreitz@redhat.com>

Comment 7 Ping Li 2017-05-24 08:08:18 UTC
Reproduced the issue with below packages:
kernel-3.10.0-588.el7.x86_64
qemu-kvm-rhev-2.8.0-6.el7

Test steps:
1. Boot up a guest with 2 data disks in the same io throttling group:
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3 \
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/pingl/diskfile/rhel74.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -drive id=drive_image2,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/pingl/diskfile/hd1.qcow2,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image2,drive=drive_image2 \
    -drive id=drive_image3,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/pingl/diskfile/hd2.qcow2,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image3,drive=drive_image3 \

2. After the guest boots up, hot unplug disk image3 in qmp:
{"execute":"device_del","arguments":{"id":"image3"}}

3. Do guest system reset in qmp:
{"execute":"system_reset"} 

4. Segmentation fault is generated by qemu-kvm


Verified the issue with below packages:
kernel-3.10.0-671.el7.x86_64
qemu-kvm-rhev-2.9.0-6.el7

Test steps:
1. Boot up a guest with 2 data disks in the same io throttling group:
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3 \
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/pingl/diskfile/rhel74.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -drive id=drive_image2,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/pingl/diskfile/hd1.qcow2,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image2,drive=drive_image2 \
    -drive id=drive_image3,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/pingl/diskfile/hd2.qcow2,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image3,drive=drive_image3 \

2. After the guest boots up, hot unplug disk image3 in qmp:
{"execute":"device_del","arguments":{"id":"image3"}}

3. Do guest system reset in qmp:
{"execute":"system_reset"} 

4. Guest boots up successfully.

Comment 9 errata-xmlrpc 2017-08-02 03:39:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392


Note You need to log in before you can comment on or make changes to this bug.