Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1428810

Summary:

'Segmentation fault (core dumped)' after hot unplug one disk in a throttle group AND do guest system reset

Product:

Red Hat Enterprise Linux 7

Reporter:

Gu Nini <ngu>

Component:

qemu-kvm-rhev

Assignee:

Eric Blake <eblake>

Status:

CLOSED ERRATA

QA Contact:

Gu Nini <ngu>

Severity:

unspecified

Docs Contact:

Priority:

unspecified

Version:

7.4

CC:

aliang, chayang, coli, eblake, hachen, juzhang, knoel, michen, mrezanin, pingl, shuang, virt-maint, xutian, xuwei

Target Milestone:

Keywords:

Regression

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

qemu-kvm-rhev-2.9.0-1.el7

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2017-08-02 03:39:56 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
GDB debug info-03032017	none

Description Gu Nini 2017-03-03 12:11:43 UTC

Created attachment 1259527 [details]
GDB debug info-03032017

Description of problem:
Boot up a guest with 2 data disks in the same io throttling group. Hot unplug one of the disks, then do system reset for the guest. It's found 'Segmentation fault (core dumped)' to the guest.

Version-Release number of selected component (if applicable):
Host kernel: 3.10.0-561.el7.x86_64
Guest kernel: 3.10.0-514.10.2.el7.x86_64
Qemu-kvm-rhev: qemu-kvm-rhev-2.8.0-5.el7.x86_64


How reproducible:
100%

Steps to Reproduce:
1. Boot up a guest with 2 data disks in the same io throttling group:

    -drive id=drive_image2,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/hd1,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image2,drive=drive_image2 \
    -drive id=drive_image3,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/hd2,bps=512000,iops=100,group-foo \
    -device scsi-hd,id=image3,drive=drive_image3\

2. After the guest boots up, hot unplug disk image3 in qmp:

{"execute":"device_del","arguments":{"id":"image3"}}

3. Do guest system reset in qmp:

{"execute":"system_reset"} 


Actual results:
The guest 'Segmentation fault (core dumped)':

QEMU 2.8.0 monitor - type 'help' for more information
(qemu) 
(qemu) 
(qemu) 
(qemu) Segmentation fault (core dumped)


Expected results:
The guest could boot up well after the system reset

Additional info:
1. Failed to reproduce the bug on qemu2.6 with following packages, so set it as regression:
Host kernel: 3.10.0-514.6.1.el7.x86_64
Qemu-kvm-rhev: qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64
Guest kernel: 3.10.0-514.10.2.el7.x86_64
2. Please refer to the attached gdb debug info 'GDB debug info-03032017' AND core file 'core.29584' in nfs server 10.73.194.27:/vol/s2coredump for details

Comment 3 Eric Blake 2017-03-31 21:36:06 UTC

I've reproduced the problem with upstream qemu 2.9-rc2:
$ ./x86_64-softmmu/qemu-system-x86_64 -nodefaults -nographic -qmp stdio -device virtio-scsi-pci,bus=pci.0  -drive id=drive_image2,if=none,format=raw,file=file2,bps=512000,iops=100,group=foo -device scsi-hd,id=image2,drive=drive_image2 -drive id=drive_image3,if=none,format=raw,file=file3,bps=512000,iops=100,group=foo -device scsi-hd,id=image3,drive=drive_image3
{"QMP": {"version": {"qemu": {"micro": 92, "minor": 8, "major": 2}, "package": " (v2.9.0-rc2-33-gb14e856)"}, "capabilities": []}}
{'execute':'qmp_capabilities'}
{"return": {}}
{'execute':'device_del','arguments':{'id':'image3'}}
{"timestamp": {"seconds": 1490996087, "microseconds": 252990}, "event": "DEVICE_DELETED", "data": {"device": "image3", "path": "/machine/peripheral/image3"}}
{"return": {}}
{'execute':'system_reset'}
{"return": {}}
{"timestamp": {"seconds": 1490996101, "microseconds": 719720}, "event": "RESET"}
{"timestamp": {"seconds": 1490996101, "microseconds": 770165}, "event": "RESET"}
Segmentation fault (core dumped)

Now investigating to see if I can find root cause and patch...

Comment 4 Eric Blake 2017-04-06 19:10:32 UTC

Upstream patch proposed:
https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg00245.html

Comment 5 Eric Blake 2017-04-11 20:00:55 UTC

Made it in qemu 2.9.0-rc4:

commit 1606e4cf8a976513ecac70ad6642a7ec45744cf5
Author: Eric Blake <eblake>
Date:   Thu Apr 6 14:08:47 2017 -0500

    throttle: Remove block from group on hot-unplug
    
    When a block device that is part of a throttle group is hot-unplugged,
    we forgot to remove it from the throttle group. This leaves stale
    memory around, and causes an easily reproducible crash:
    
    $ ./x86_64-softmmu/qemu-system-x86_64 -nodefaults -nographic -qmp stdio \
    -device virtio-scsi-pci,bus=pci.0 -drive \
    id=drive_image2,if=none,format=raw,file=file2,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image2,drive=drive_image2 -drive \
    id=drive_image3,if=none,format=raw,file=file3,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image3,drive=drive_image3
    {'execute':'qmp_capabilities'}
    {'execute':'device_del','arguments':{'id':'image3'}}
    {'execute':'system_reset'}
    
    Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1428810
    
    Suggested-by: Alberto Garcia <berto>
    Signed-off-by: Eric Blake <eblake>
    Message-id: 20170406190847.29347-1-eblake
    Reviewed-by: Stefan Hajnoczi <stefanha>
    Signed-off-by: Max Reitz <mreitz>

Comment 7 Ping Li 2017-05-24 08:08:18 UTC

Reproduced the issue with below packages:
kernel-3.10.0-588.el7.x86_64
qemu-kvm-rhev-2.8.0-6.el7

Test steps:
1. Boot up a guest with 2 data disks in the same io throttling group:
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3 \
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/pingl/diskfile/rhel74.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -drive id=drive_image2,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/pingl/diskfile/hd1.qcow2,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image2,drive=drive_image2 \
    -drive id=drive_image3,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/pingl/diskfile/hd2.qcow2,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image3,drive=drive_image3 \

2. After the guest boots up, hot unplug disk image3 in qmp:
{"execute":"device_del","arguments":{"id":"image3"}}

3. Do guest system reset in qmp:
{"execute":"system_reset"} 

4. Segmentation fault is generated by qemu-kvm


Verified the issue with below packages:
kernel-3.10.0-671.el7.x86_64
qemu-kvm-rhev-2.9.0-6.el7

Test steps:
1. Boot up a guest with 2 data disks in the same io throttling group:
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3 \
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/pingl/diskfile/rhel74.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -drive id=drive_image2,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/pingl/diskfile/hd1.qcow2,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image2,drive=drive_image2 \
    -drive id=drive_image3,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/pingl/diskfile/hd2.qcow2,bps=512000,iops=100,group=foo \
    -device scsi-hd,id=image3,drive=drive_image3 \

2. After the guest boots up, hot unplug disk image3 in qmp:
{"execute":"device_del","arguments":{"id":"image3"}}

3. Do guest system reset in qmp:
{"execute":"system_reset"} 

4. Guest boots up successfully.

Comment 9 errata-xmlrpc 2017-08-02 03:39:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392