Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 892067

Summary:	qemu-kvm sometimes core dump when unplug a using virtio data disk
Product:	Red Hat Enterprise Linux 6	Reporter:	Sibiao Luo <sluo>
Component:	qemu-kvm	Assignee:	Asias He <asias>
Status:	CLOSED WORKSFORME	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	6.4	CC:	acathrow, areis, asias, bsarathy, chayang, juzhang, michen, mkenneth, qzhang, sluo, virt-maint, xfu, xutian
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-07-17 03:05:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sibiao Luo 2013-01-05 03:39:56 UTC

Description of problem:
this bug was found when verified bug 734051. there are many core dump files generated when do unplug a using virtio data disk testing for 1000 times.

Version-Release number of selected component (if applicable):
host info:
# uname -r && rpm -q qemu-kvm-rhev
2.6.32-351.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.348.el6.x86_64
guest info:
# uname -r
2.6.32-351.el6.x86_64

How reproducible:
20/1000

Steps to Reproduce:
1.boot a vm without virito disk.
2.attach a virtio disk from monitor as disk1.
3.mount /dev/vda /mnt, and do some read/write in /mnt;
4.device_del disk1 from monitor during I/O.
5.verify if no panic occurs, kill the guest and go to step 1 directly and retest for 1000 times.

Actual results:
after test 1000 times, there are many core dump file generated. I will paste the bt logs of the core dump later.

Expected results:
there is no any core dump occur.

Additional info:

Comment 2 Sibiao Luo 2013-01-05 03:45:51 UTC

I just paste some of the bt logs of the core dump here.

Program terminated with signal 11, Segmentation fault.
#0  0x00007f979afd54fc in qdict_destroy_obj (obj=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qdict.c:470
470	            QLIST_REMOVE(entry, next);
(gdb) bt
#0  0x00007f979afd54fc in qdict_destroy_obj (obj=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qdict.c:470
#1  0x00007f979afd56cf in qobject_decref (obj=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qobject.h:99
#2  qlist_destroy_obj (obj=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qlist.c:151
#3  0x00007f979afd6739 in qobject_decref (lexer=0x7f979d995c30, token=0x7f979e499ef0, type=JSON_OPERATOR, x=37, y=36) at /usr/src/debug/qemu-kvm-0.12.1.2/qobject.h:99
#4  json_message_process_token (lexer=0x7f979d995c30, token=0x7f979e499ef0, type=JSON_OPERATOR, x=37, y=36) at /usr/src/debug/qemu-kvm-0.12.1.2/json-streamer.c:89
#5  0x00007f979afd63a0 in json_lexer_feed_char (lexer=0x7f979d995c30, ch=125 '}', flush=false) at /usr/src/debug/qemu-kvm-0.12.1.2/json-lexer.c:303
#6  0x00007f979afd64e9 in json_lexer_feed (lexer=0x7f979d995c30, buffer=0x7fff04481010 "}", size=1) at /usr/src/debug/qemu-kvm-0.12.1.2/json-lexer.c:355
#7  0x00007f979af8074e in monitor_control_read (opaque=<value optimized out>, buf=<value optimized out>, size=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:4973
#8  0x00007f979aff987a in qemu_chr_read (opaque=0x7f979d7a7700) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-char.c:180
#9  tcp_chr_read (opaque=0x7f979d7a7700) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-char.c:2211
#10 0x00007f979af7940f in main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:3975
#11 0x00007f979af9b9ba in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2244
#12 0x00007f979af7c178 in main_loop (argc=45, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4187
#13 main (argc=45, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6525
(gdb)


Program terminated with signal 11, Segmentation fault.
#0  virtio_blk_handle_request (req=0x7f62769a9510, mrb=0x7fffc0dca7a0) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:387
387	    if (req->out->type & VIRTIO_BLK_T_FLUSH) {
(gdb) bt
#0  virtio_blk_handle_request (req=0x7f62769a9510, mrb=0x7fffc0dca7a0) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:387
#1  0x00007f6273215e8b in virtio_blk_dma_restart_bh (opaque=0x7f62769a94a0) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:450
#2  0x00007f62732364a1 in qemu_bh_poll () at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:70
#3  0x00007f6273201589 in main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4017
#4  0x00007f62732239ba in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2244
#5  0x00007f6273204178 in main_loop (argc=45, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4187
#6  main (argc=45, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6525
(gdb)


Program terminated with signal 11, Segmentation fault.
#0  qemu_bh_delete (bh=0x90) at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:118
118	    bh->scheduled = 0;
(gdb) bt
#0  qemu_bh_delete (bh=0x90) at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:118
#1  0x00007f671e182e5f in virtio_blk_dma_restart_bh (opaque=0x7f6721c5cd80) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:444
#2  0x00007f671e1a34a1 in qemu_bh_poll () at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:70
#3  0x00007f671e16e589 in main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4017
#4  0x00007f671e1909ba in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2244
#5  0x00007f671e171178 in main_loop (argc=45, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4187
#6  main (argc=45, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6525
(gdb)

Best Regards.
sluo

Comment 3 Xu Tian 2013-01-08 00:52:22 UTC

(In reply to comment #2)
> I just paste some of the bt logs of the core dump here.
> 
> Program terminated with signal 11, Segmentation fault.
> #0  0x00007f979afd54fc in qdict_destroy_obj (obj=<value optimized out>) at
> /usr/src/debug/qemu-kvm-0.12.1.2/qdict.c:470
> 470	            QLIST_REMOVE(entry, next);
> (gdb) bt
> #0  0x00007f979afd54fc in qdict_destroy_obj (obj=<value optimized out>) at
> /usr/src/debug/qemu-kvm-0.12.1.2/qdict.c:470
> #1  0x00007f979afd56cf in qobject_decref (obj=<value optimized out>) at
> /usr/src/debug/qemu-kvm-0.12.1.2/qobject.h:99
> #2  qlist_destroy_obj (obj=<value optimized out>) at
> /usr/src/debug/qemu-kvm-0.12.1.2/qlist.c:151
> #3  0x00007f979afd6739 in qobject_decref (lexer=0x7f979d995c30,
> token=0x7f979e499ef0, type=JSON_OPERATOR, x=37, y=36) at
> /usr/src/debug/qemu-kvm-0.12.1.2/qobject.h:99
> #4  json_message_process_token (lexer=0x7f979d995c30, token=0x7f979e499ef0,
> type=JSON_OPERATOR, x=37, y=36) at
> /usr/src/debug/qemu-kvm-0.12.1.2/json-streamer.c:89
> #5  0x00007f979afd63a0 in json_lexer_feed_char (lexer=0x7f979d995c30, ch=125
> '}', flush=false) at /usr/src/debug/qemu-kvm-0.12.1.2/json-lexer.c:303
> #6  0x00007f979afd64e9 in json_lexer_feed (lexer=0x7f979d995c30,
> buffer=0x7fff04481010 "}", size=1) at
> /usr/src/debug/qemu-kvm-0.12.1.2/json-lexer.c:355
> #7  0x00007f979af8074e in monitor_control_read (opaque=<value optimized
> out>, buf=<value optimized out>, size=<value optimized out>)
>     at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:4973
> #8  0x00007f979aff987a in qemu_chr_read (opaque=0x7f979d7a7700) at
> /usr/src/debug/qemu-kvm-0.12.1.2/qemu-char.c:180
> #9  tcp_chr_read (opaque=0x7f979d7a7700) at
> /usr/src/debug/qemu-kvm-0.12.1.2/qemu-char.c:2211
> #10 0x00007f979af7940f in main_loop_wait (timeout=1000) at
> /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:3975
> #11 0x00007f979af9b9ba in kvm_main_loop () at
> /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2244
> #12 0x00007f979af7c178 in main_loop (argc=45, argv=<value optimized out>,
> envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4187
> #13 main (argc=45, argv=<value optimized out>, envp=<value optimized out>)
> at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6525
> (gdb)
> 
> 
> Program terminated with signal 11, Segmentation fault.
> #0  virtio_blk_handle_request (req=0x7f62769a9510, mrb=0x7fffc0dca7a0) at
> /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:387
> 387	    if (req->out->type & VIRTIO_BLK_T_FLUSH) {
> (gdb) bt
> #0  virtio_blk_handle_request (req=0x7f62769a9510, mrb=0x7fffc0dca7a0) at
> /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:387
> #1  0x00007f6273215e8b in virtio_blk_dma_restart_bh (opaque=0x7f62769a94a0)
> at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:450
> #2  0x00007f62732364a1 in qemu_bh_poll () at
> /usr/src/debug/qemu-kvm-0.12.1.2/async.c:70
> #3  0x00007f6273201589 in main_loop_wait (timeout=1000) at
> /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4017
> #4  0x00007f62732239ba in kvm_main_loop () at
> /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2244
> #5  0x00007f6273204178 in main_loop (argc=45, argv=<value optimized out>,
> envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4187
> #6  main (argc=45, argv=<value optimized out>, envp=<value optimized out>)
> at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6525
> (gdb)
> 
> 
> Program terminated with signal 11, Segmentation fault.
> #0  qemu_bh_delete (bh=0x90) at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:118
> 118	    bh->scheduled = 0;
> (gdb) bt
> #0  qemu_bh_delete (bh=0x90) at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:118
> #1  0x00007f671e182e5f in virtio_blk_dma_restart_bh (opaque=0x7f6721c5cd80)
> at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:444
> #2  0x00007f671e1a34a1 in qemu_bh_poll () at
> /usr/src/debug/qemu-kvm-0.12.1.2/async.c:70
> #3  0x00007f671e16e589 in main_loop_wait (timeout=1000) at
> /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4017
> #4  0x00007f671e1909ba in kvm_main_loop () at
> /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2244
> #5  0x00007f671e171178 in main_loop (argc=45, argv=<value optimized out>,
> envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4187
> #6  main (argc=45, argv=<value optimized out>, envp=<value optimized out>)
> at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6525
> (gdb)
> 
> Best Regards.
> sluo

It looks like bz822386;

Comment 4 RHEL Program Management 2013-01-11 06:47:10 UTC

This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 6 Asias He 2013-05-21 03:12:22 UTC

Sibiao Luo,

Can you try latest qemu build to see if we still have this problem. 

Recently, we closed another similar hotplug issue: https://bugzilla.redhat.com/show_bug.cgi?id=822386

You do not have to run it 1000 times. 100 or 50 times is enough. It is a corner case test anyway. I prefer to run it manually if possible, this way we can eliminate the affect of the custom script developed to run the test.

Comment 7 Sibiao Luo 2013-05-22 10:06:38 UTC

(In reply to Asias He from comment #6)
> Sibiao Luo,
> 
> Can you try latest qemu build to see if we still have this problem. 
> 
> Recently, we closed another similar hotplug issue:
> https://bugzilla.redhat.com/show_bug.cgi?id=822386
> 
> You do not have to run it 1000 times. 100 or 50 times is enough. It is a
> corner case test anyway. I prefer to run it manually if possible, this way
> we can eliminate the affect of the custom script developed to run the test.

Hi Asias He,

   I tried it 50 times manually testing with the same steps as comment #0, but did not met any core dump. Every time when hot-plug the disk, the process of dd will quit and prompt a message 'Read-only file system' in the guest, like:
dd: writing `/mnt/test': Read-only file system
34849987+0 records in
34849986+0 records out
174249930 bytes (174 MB) copied, 41.0052 s, 4.2 MB/s

Best Regards,
sluo

Comment 8 Asias He 2013-05-27 02:27:21 UTC

(In reply to Sibiao Luo from comment #7)
> (In reply to Asias He from comment #6)
> > Sibiao Luo,
> > 
> > Can you try latest qemu build to see if we still have this problem. 
> > 
> > Recently, we closed another similar hotplug issue:
> > https://bugzilla.redhat.com/show_bug.cgi?id=822386
> > 
> > You do not have to run it 1000 times. 100 or 50 times is enough. It is a
> > corner case test anyway. I prefer to run it manually if possible, this way
> > we can eliminate the affect of the custom script developed to run the test.
> 
> Hi Asias He,
> 
>    I tried it 50 times manually testing with the same steps as comment #0,
> but did not met any core dump. Every time when hot-plug the disk,

I guess you mean hot-unplug the disk?

> the
> process of dd will quit and prompt a message 'Read-only file system' in the
> guest, like:
> dd: writing `/mnt/test': Read-only file system
> 34849987+0 records in
> 34849986+0 records out
> 174249930 bytes (174 MB) copied, 41.0052 s, 4.2 MB/s

I think this is a acceptable behaviour in guest when you suddenly remove a disk which is in use.

Comment 9 Ademar Reis 2013-07-17 03:05:31 UTC

(In reply to Sibiao Luo from comment #7)
> (In reply to Asias He from comment #6)
> > Sibiao Luo,
> > 
> > Can you try latest qemu build to see if we still have this problem. 
> > 
> > Recently, we closed another similar hotplug issue:
> > https://bugzilla.redhat.com/show_bug.cgi?id=822386
> > 
> > You do not have to run it 1000 times. 100 or 50 times is enough. It is a
> > corner case test anyway. I prefer to run it manually if possible, this way
> > we can eliminate the affect of the custom script developed to run the test.
> 
> Hi Asias He,
> 
>    I tried it 50 times manually testing with the same steps as comment #0,
> but did not met any core dump. Every time when hot-plug the disk, the
> process of dd will quit and prompt a message 'Read-only file system' in the
> guest, like:
> dd: writing `/mnt/test': Read-only file system
> 34849987+0 records in
> 34849986+0 records out
> 174249930 bytes (174 MB) copied, 41.0052 s, 4.2 MB/s
> 

Based on that, closing as WORKSFORME. If you can reproduce it or spot it again, please reopen.