952240 – hot-plugging multi-func devices caused: qemu: hardware error: register_ioport_write: invalid opaque

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 952240 - hot-plugging multi-func devices caused: qemu: hardware error: register_ioport_write: invalid opaque

Summary: hot-plugging multi-func devices caused: qemu: hardware error: register_ioport...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	qemu-kvm
Sub Component:
Version:	6.4
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Amos Kong
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1025680
TreeView+	depends on / blocked

Reported:	2013-04-15 12:50 UTC by Amos Kong
Modified:	2015-05-25 00:07 UTC (History)
CC List:	15 users (show)
Fixed In Version:	qemu-kvm-0.12.1.2-2.374.el6
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	1025680 (view as bug list)
Environment:
Last Closed:	2013-11-21 06:48:07 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2013:1553	0	normal	SHIPPED_LIVE	Important: qemu-kvm security, bug fix, and enhancement update	2013-11-20 21:40:29 UTC

Description Amos Kong 2013-04-15 12:50:48 UTC

Description of problem:

Hot-plugging many multi-fun devices to guest, it caused qemu crash.

 qemu: hardware error: register_ioport_write: invalid opaque

If I only add less (2) slots, it works

Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.356.el6.x86_64
guest Kernel: 2.6.32-343.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1. start guest: (gdb) r -monitor unix:/tmp/m,nowait,server -vnc :2 /images/RHEL-Server-6.4-64-virtio.qcow2 -m 1000
2. execute radd.sh to hot-add disks
  
Actual results:
qemu crash

Additional info:
radd.sh
===================
for i in `seq 3 9` a b c d e f 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f;do
#for i in `seq 5 5`;do
for j in `seq 1 7` 0;do
#qemu-img create /tmp/resize$i$j.qcow2 1M -f qcow2
/bin/cp /tmp/resize0.qcow2 /tmp/resize$i$j.qcow2

echo __com.redhat_drive_add id=drv$i$j,file=/tmp/resize$i$j.qcow2
echo __com.redhat_drive_add id=drv$i$j,file=/tmp/resize$i$j.qcow2 | nc -U /tmp/m
#echo drive_add $i.$j id=drv$i$j,file=/tmp/resize$i$j.qcow2,if=none
#echo drive_add $i.$j id=drv$i$j,file=/tmp/resize$i$j.qcow2,if=none | nc -U /tmp/m

echo device_add virtio-blk-pci,id=dev$i$j,drive=drv$i$j,addr=0x$i.$j,multifunction=on
echo device_add virtio-blk-pci,id=dev$i$j,drive=drv$i$j,addr=0x$i.$j,multifunction=on | nc -U /tmp/m
done
done
===================


(gdb) r -monitor unix:/tmp/m,nowait,server -vnc :2 /images/RHEL-Server-6.4-64-virtio.qcow2 -m 1000
Starting program: /home/devel/qemu-kvm-rhel6/x86_64-softmmu/qemu-system-x86_64 -monitor unix:/tmp/m,nowait,server -vnc :2 /images/RHEL-Server-6.4-64-virtio.qcow2 -m 1000
[Thread debugging using libthread_db enabled]
warning: the debug information found in "/usr/lib/debug//usr/lib64/libspice-server.so.1.5.0.debug" does not match "/usr/lib64/libspice-server.so.1" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug/usr/lib64/libspice-server.so.1.5.0.debug" does not match "/usr/lib64/libspice-server.so.1" (CRC mismatch).

[New Thread 0x7ffff348d700 (LWP 20081)]
[New Thread 0x7ffff288b700 (LWP 20082)]
[Thread 0x7ffff348d700 (LWP 20081) exited]
[New Thread 0x7ffff348d700 (LWP 20098)]
[New Thread 0x7fff9f1be700 (LWP 20662)]
qemu: hardware error: register_ioport_write: invalid opaque
CPU #0:
RAX=0000000000000003 RBX=ffff88003dbd6400 RCX=0000000000000004 RDX=0000000000000cfc
RSI=0000000000000000 RDI=0000000000000097 RBP=ffff88003d111960 RSP=ffff88003d111930
R8 =0000000000000002 R9 =0000000000000003 R10=0000000000000058 R11=0000000000000000
R12=0000000000000246 R13=0000000000000003 R14=ffff88003ab5a090 R15=ffff88003d111d00
RIP=ffffffff8142e33a RFL=00000006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0000 0000000000000000 ffffffff 00000000
GS =0000 ffff880002200000 ffffffff 00000000
LDT=0000 0000000000000000 ffffffff 00000000
TR =0040 ffff880002214280 00002087 00008b00 DPL=0 TSS64-busy
GDT=     ffff880002204000 0000007f
IDT=     ffffffff81dde000 00000fff
CR0=8005003b CR2=00007fffa47a0980 CR3=000000003ca21000 CR4=000006f0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=ff00000000000000ff00000000000000 XMM01=003a676f6c6b6d69003a6c656e72656b
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000098010000000000000000 XMM05=40404040404040404040404040404040
XMM06=5b5b5b5b5b5b5b5b5b5b5b5b5b5b5b5b XMM07=20202020202020202020202020202020
XMM08=00000000002000000000000000000000 XMM09=ffffffffffffffff0000000000000000
XMM10=00000000000000000000000000000000 XMM11=ffffffffffffffff0000000000000000
XMM12=00000000000000000000000000000000 XMM13=00000000000000000000000000000000
XMM14=00000000000000000000000000000000 XMM15=00000000000000000000000000000000

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff288b700 (LWP 20082)]
0x00007ffff5e008a5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install SDL-1.2.14-3.el6.x86_64 celt051-0.5.1.3-0.el6.x86_64 cyrus-sasl-lib-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-md5-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-plain-2.1.23-13.el6_3.1.x86_64 db4-4.7.25-17.el6.x86_64 glib2-2.22.5-7.el6.x86_64 glibc-2.12-1.107.el6.x86_64 gnutls-2.8.5-10.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6.x86_64 libX11-1.5.0-4.el6.x86_64 libXau-1.0.6-4.el6.x86_64 libaio-0.3.107-10.el6.x86_64 libcom_err-1.41.12-14.el6.x86_64 libgcrypt-1.4.5-9.el6_2.2.x86_64 libgpg-error-1.7-4.el6.x86_64 libjpeg-turbo-1.2.1-1.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 libtasn1-2.3-3.el6_2.1.x86_64 libuuid-2.17.2-12.9.el6.x86_64 libxcb-1.8.1-1.el6.x86_64 ncurses-libs-5.7-3.20090208.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 openssl-1.0.0-27.el6.x86_64 pixman-0.26.2-4.el6.x86_64 spice-server-0.12.0-12.el6.x86_64 usbredir-0.5.1-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007ffff5e008a5 in raise () from /lib64/libc.so.6
#1  0x00007ffff5e02085 in abort () from /lib64/libc.so.6
#2  0x000000000040e688 in hw_error (fmt=<value optimized out>) at /home/devel/qemu-kvm-rhel6/vl.c:506
#3  0x0000000000477f1a in register_ioport_write (start=<value optimized out>, length=<value optimized out>, size=6, func=0xffffffffffffffff, opaque=0x7ffff288b700)
    at ioport.c:171
#4  0x0000000000423b48 in virtio_map (pci_dev=0x131db80, region_num=<value optimized out>, addr=45120, size=<value optimized out>, type=<value optimized out>)
    at /home/devel/qemu-kvm-rhel6/hw/virtio-pci.c:533
#5  0x000000000041a78b in pci_update_mappings (d=0x131db80) at /home/devel/qemu-kvm-rhel6/hw/pci.c:1067
#6  0x00000000004242c2 in virtio_write_config (pci_dev=0x131db80, address=4, val=3, len=2) at /home/devel/qemu-kvm-rhel6/hw/virtio-pci.c:559
#7  0x000000000042fe25 in kvm_handle_io (env=0xe99a50) at /home/devel/qemu-kvm-rhel6/kvm-all.c:147
#8  kvm_run (env=0xe99a50) at /home/devel/qemu-kvm-rhel6/qemu-kvm.c:1048
#9  0x000000000042fec9 in kvm_cpu_exec (env=<value optimized out>) at /home/devel/qemu-kvm-rhel6/qemu-kvm.c:1743
#10 0x0000000000430d26 in kvm_main_loop_cpu (_env=0xe99a50) at /home/devel/qemu-kvm-rhel6/qemu-kvm.c:2004
#11 ap_main_loop (_env=0xe99a50) at /home/devel/qemu-kvm-rhel6/qemu-kvm.c:2060
#12 0x00007ffff79bf851 in start_thread () from /lib64/libpthread.so.0
#13 0x00007ffff5eb690d in clone () from /lib64/libc.so.6

Comment 1 Amos Kong 2013-04-15 13:28:09 UTC

Can reproduce bug with upstream qemu & rhel6 guest. It seems ioports were repeatedly registered until no enough resource. Will debug later.


(gdb) bt
#0  0x00007ffff50888a5 in raise () from /lib64/libc.so.6
#1  0x00007ffff508a085 in abort () from /lib64/libc.so.6
#2  0x00007ffff7e4ee9d in kvm_io_ioeventfd_add (listener=0x7ffff83064c0, section=0x7ffff2ed05b0, match_data=true, data=0, e=0x7ffffc202280) at /home/devel/qemu/kvm-all.c:788
#3  0x00007ffff7e55454 in address_space_add_del_ioeventfds (as=0x7ffff8b3f040, fds_new=0x7fffe40b6980, fds_new_nb=199, fds_old=0x7fffe40a7620, fds_old_nb=198)
    at /home/devel/qemu/memory.c:616
#4  0x00007ffff7e557b7 in address_space_update_ioeventfds (as=0x7ffff8b3f040) at /home/devel/qemu/memory.c:649
#5  0x00007ffff7e56251 in address_space_update_topology (as=0x7ffff8b3f040) at /home/devel/qemu/memory.c:730
#6  0x00007ffff7e56378 in memory_region_transaction_commit () at /home/devel/qemu/memory.c:750
#7  0x00007ffff7e583b6 in memory_region_add_eventfd (mr=0x7ffffc4d6770, addr=16, size=2, match_data=true, data=0, e=0x7ffffc202280) at /home/devel/qemu/memory.c:1273
#8  0x00007ffff7d302b6 in virtio_pci_set_host_notifier_internal (proxy=0x7ffffc4d60c0, n=0, assign=true, set_handler=true) at hw/virtio-pci.c:192
#9  0x00007ffff7d303bb in virtio_pci_start_ioeventfd (proxy=0x7ffffc4d60c0) at hw/virtio-pci.c:218
#10 0x00007ffff7d30787 in virtio_ioport_write (opaque=0x7ffffc4d60c0, addr=18, val=7) at hw/virtio-pci.c:309
#11 0x00007ffff7d30c40 in virtio_pci_config_write (opaque=0x7ffffc4d60c0, addr=18, val=7, size=1) at hw/virtio-pci.c:428
#12 0x00007ffff7e53e2f in memory_region_write_accessor (opaque=0x7ffffc4d6770, addr=18, value=0x7ffff2ed0b00, size=1, shift=0, mask=255) at /home/devel/qemu/memory.c:334
#13 0x00007ffff7e53f11 in access_with_adjusted_size (addr=18, value=0x7ffff2ed0b00, size=1, access_size_min=1, access_size_max=4, access=
    0x7ffff7e53da3 <memory_region_write_accessor>, opaque=0x7ffffc4d6770) at /home/devel/qemu/memory.c:364
#14 0x00007ffff7e54399 in memory_region_iorange_write (iorange=0x7fffe40a5060, offset=18, width=1, data=7) at /home/devel/qemu/memory.c:439
#15 0x00007ffff7e4c622 in ioport_writeb_thunk (opaque=0x7fffe40a5060, addr=55378, data=7) at /home/devel/qemu/ioport.c:212
#16 0x00007ffff7e4bfd7 in ioport_write (index=0, address=55378, data=7) at /home/devel/qemu/ioport.c:83
#17 0x00007ffff7e4cbda in cpu_outb (addr=55378, val=7 '\a') at /home/devel/qemu/ioport.c:289
#18 0x00007ffff7e506a0 in kvm_handle_io (port=55378, data=0x7ffff7b42000, direction=1, size=1, count=1) at /home/devel/qemu/kvm-all.c:1424
#19 0x00007ffff7e50d33 in kvm_cpu_exec (env=0x7ffff8d28680) at /home/devel/qemu/kvm-all.c:1579
#20 0x00007ffff7dda85b in qemu_kvm_cpu_thread_fn (arg=0x7ffff8d28680) at /home/devel/qemu/cpus.c:759
#21 0x00007ffff6050851 in start_thread () from /lib64/libpthread.so.0
#22 0x00007ffff513e90d in clone () from /lib64/libc.so.6

Comment 2 Amos Kong 2013-05-22 05:01:59 UTC

For upstream issue in Comment #1: There is no enough ioeventfd in host, it caused allocate ioeventfd failed.

what's I can do:
  Q: increase ioeventfd resource in host?
  A: No, we don't support mult-func hotplug in rhel6. If we use pci-bridge,
     ioeventfd can also be exhausted. Current process is abort() without
     nice error message, I can fix it in upstream.

I have posted a patch to fix error note:
  [qemu-devel][PATCH] kvm: add detail error message when fail to add ioeventfd.



For internal issue in Comment #0, it's not caused by ioeventfds exhausted.
I can successfully hotplug 0x4~0x18 slots, fail to hotplug 0x19 slot.

Comment 3 Amos Kong 2013-05-22 06:03:32 UTC

Internal issue in comment #0, can be reproduced with qemu-kvm-0.12.1.2-2.180.el6

(multifunction option was introduced in that version).

This problem doesn't exist in latest upstream.

multiple-function hotplug isn't supported by us, so close this bug as WONTFIX.

Comment 4 Amos Kong 2013-05-22 07:58:59 UTC

Upstream fixed this problem in commit 8385b173a0ca4c2345434104e6cc2a7259adc4b9

ACPI_DBG_IO_ADDR already takes the ioport, it could not be re-allocate to hot-plugged pci device.

The following patch can fix internal issue.

diff --git a/hw/acpi.c b/hw/acpi.c
index f824b8e..ddbbda7 100644
--- a/hw/acpi.c
+++ b/hw/acpi.c
@@ -34,8 +34,6 @@
 /* i82731AB (PIIX4) compatible power management function */
 #define PM_FREQ 3579545
 
-#define ACPI_DBG_IO_ADDR  0xb044
-
 #define GPE_BASE 0xafe0
 #define PROC_BASE 0xaf00
 #define PCI_UP_BASE 0xae00
@@ -339,13 +337,6 @@ static uint32_t pm_smi_readb(void *opaque, uint32_t addr)
     return val;
 }
 
-static void acpi_dbg_writel(void *opaque, uint32_t addr, uint32_t val)
-{
-#if defined(DEBUG)
-    printf("ACPI: DBG: 0x%08x\n", val);
-#endif
-}
-
 static void smb_transaction(PIIX4PMState *s)
 {
     uint8_t prot = (s->smb_ctl >> 2) & 0x07;
@@ -702,8 +693,6 @@ static int piix4_pm_initfn(PCIDevice *dev)
     register_ioport_write(0xb2, 2, 1, pm_smi_writeb, s);
     register_ioport_read(0xb2, 2, 1, pm_smi_readb, s);
 
-    register_ioport_write(ACPI_DBG_IO_ADDR, 4, 4, acpi_dbg_writel, s);
-
     if (kvm_enabled()) {
         /* Mark SMM as already inited to prevent SMM from running.  KVM does not
          * support SMM mode. */

Comment 5 Amos Kong 2013-05-22 08:00:18 UTC

Hi Gerd,

Is my fix in last comment right? do we need to fix this in RHEL6?

Comment 6 Gerd Hoffmann 2013-05-28 05:52:53 UTC

Looks good, acpi_dbg_writel can go away.  It does nothing anyway (other than occupying an I/O port the guest doesn't know about, which leads to this bug as the guest probably tries to map an pci bar to that I/O port).

Comment 7 Amos Kong 2013-05-29 02:07:32 UTC

Re-open this bug, worth to fix this qemu crash.

Comment 8 Sibiao Luo 2013-05-30 05:03:16 UTC

Reproduce this issue on qemu-kvm-0.12.1.2-2.370.el6.x86_64, hot-plugging many multi-fun devices to guest, it caused qemu crash.

host info:
kernel-2.6.32-377.el6.x86_64
qemu-kvm-0.12.1.2-2.370.el6.x86_64
guest info:
kernel-2.6.32-377.el6.x86_64

Steps:
1.start guest.
# /usr/libexec/qemu-kvm -S -M rhel6.5.0 -cpu host -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -name sluo-test -uuid a51eb497-bfd7-47c0-8b5b-0853716e3ce5 -rtc base=localtime,clock=host,driftfix=slew -drive file=/home/RHEL-Server-6.4-64-virtio.qcow2.bk,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,serial=QEMU-DISK1 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-system-disk,id=system-disk,bootindex=1 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=08:2e:5f:0a:1d:b1,bus=pci.0,addr=0x5,bootindex=2,ioeventfd=off -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -qmp tcp:0:4444,server,nowait -k en-us -boot menu=on -vnc :1 -spice disable-ticketing,port=5931 -vga qxl -monitor unix:/tmp/monitor1,server,nowait -monitor stdio
2.execute repeat_add.sh to hot-add disks.
# cat repeat_add.sh
-------------------------------------------------------
for i in `seq 3 9` a b c d e f 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f;do
#for i in `seq 5 5`;do
for j in `seq 1 7` 0;do
qemu-img create /tmp/resize$i$j.qcow2 1M -f qcow2
sleep 2
echo __com.redhat_drive_add id=drv$i$j,file=/tmp/resize$i$j.qcow2
echo __com.redhat_drive_add id=drv$i$j,file=/tmp/resize$i$j.qcow2 | nc -U /tmp/monitor1
#echo drive_add $i.$j id=drv$i$j,file=/tmp/resize$i$j.qcow2,if=none
#echo drive_add $i.$j id=drv$i$j,file=/tmp/resize$i$j.qcow2,if=none | nc -U /tmp/monitor1
sleep 2
echo device_add virtio-blk-pci,id=dev$i$j,drive=drv$i$j,addr=0x$i.$j,multifunction=on
echo device_add virtio-blk-pci,id=dev$i$j,drive=drv$i$j,addr=0x$i.$j,multifunction=on | nc -U /tmp/monitor1
done
done
-------------------------------------------------------

Results:
after step 2, the qemu will core dump, paste the bt log here.
(qemu) qemu: hardware error: register_ioport_write: invalid opaque
CPU #0:
RAX=0000000000000003 RBX=ffff88007e773c00 RCX=0000000000000004 RDX=0000000000000cfc
RSI=0000000000000000 RDI=0000000000000097 RBP=ffff88007e711960 RSP=ffff88007e711930
R8 =0000000000000002 R9 =0000000000000003 R10=00000000000000d8 R11=0000000000000000
R12=0000000000000246 R13=0000000000000003 R14=ffff880052df4090 R15=ffff88007e711d00
RIP=ffffffff81430a7a RFL=00000006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0000 0000000000000000 ffffffff 00000000
GS =0000 ffff880002200000 ffffffff 00000000
LDT=0000 0000000000000000 ffffffff 00000000
TR =0040 ffff880002214200 00002087 00008b00 DPL=0 TSS64-busy
GDT=     ffff880002204000 0000007f
IDT=     ffffffff81ddf000 00000fff
CR0=8005003b CR2=00007f4bbd1dcac8 CR3=0000000037456000 CR4=000407f0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=000000000000000f ffff FPR1=0000000000000031 ffff
FPR2=0014000000000000 ffff FPR3=000000000000000a ffff
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 ffff
XMM00=ff000000000000000000ff0000000000 XMM01=00000000000000000000000000000000
XMM02=00007f4bbd1bb14000007f4b00696370 XMM03=00000000ff00000000ff0000000000ff
XMM04=2064657a696e676f6365726e75002f40 XMM05=5b5b5b5b5b5b5b5b5b5b5b5b5b5b5b5b
XMM06=20202020202020202020202020202020 XMM07=00000000000000000000000000000000
XMM08=ffffff0000000000ff00000000000000 XMM09=00202020202020202000202020202020
XMM10=ffffffffffffffffffffffffffffffff XMM11=00000000000000000000000000000000
XMM12=00000000000000000000000000000000 XMM13=00000000000000000000000000000000
XMM14=00000000000000000000000000000000 XMM15=00000000000000000000000000000000
CPU #1:
RAX=0000000000000000 RBX=0000000000000001 RCX=0000000000000000 RDX=0000000000000000
RSI=0000000000000001 RDI=ffffffff81de2228 RBP=ffff88007e4fbed8 RSP=ffff88007e4fbed8
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000002 R11=0000000000000000
R12=ffffffff81c07a80 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=ffffffff8103b92b RFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0000 0000000000000000 ffffffff 00000000
GS =0000 ffff880002300000 ffffffff 00000000
LDT=0000 0000000000000000 000fffff 00000000
TR =0040 ffff880002314200 00002087 00008b00 DPL=0 TSS64-busy
GDT=     ffff880002304000 0000007f
IDT=     ffffffff81ddf000 00000fff
CR0=8005003b CR2=00007f4bbb2a33d1 CR3=0000000037675000 CR4=000407e0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=ffff000000ff00000000000000000000 XMM01=ffffffffffffffffffff000000000000
XMM02=0000000000ffffff0000000000000000 XMM03=0000000000000000ffffffffff000000
XMM04=40404040404040404040404040404040 XMM05=5b5b5b5b5b5b5b5b5b5b5b5b5b5b5b5b
XMM06=20202020202020202020202020202020 XMM07=00000000000000000000000000000000
XMM08=ffffffffffffffff0000000000000000 XMM09=00000000000000000000000000000000
XMM10=ffffffffffffffff0000000000000000 XMM11=00000000000000000000000000000000
XMM12=00000000000000000000000000000000 XMM13=00000000000000000000000000000000
XMM14=00000000000000000000000000000000 XMM15=00000000000000000000000000000000
Aborted (core dumped)

(gdb) bt
#0  0x00007fbb6f8388a5 in raise () from /lib64/libc.so.6
#1  0x00007fbb6f83a085 in abort () from /lib64/libc.so.6
#2  0x00007fbb71ed73f2 in hw_error (fmt=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:508
#3  0x00007fbb71f4507d in register_ioport_write (start=<value optimized out>, length=<value optimized out>, size=6, func=
    0xffffffffffffffff, opaque=0x7fbb63fff700) at /usr/src/debug/qemu-kvm-0.12.1.2/ioport.c:171
#4  0x00007fbb71eedd1a in virtio_map (pci_dev=0x7fbb74d8db80, region_num=<value optimized out>, addr=45120, 
    size=<value optimized out>, type=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-pci.c:533
#5  0x00007fbb71ee439b in pci_update_mappings (d=0x7fbb74d8db80) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/pci.c:1067
#6  0x00007fbb71eee4b2 in virtio_write_config (pci_dev=0x7fbb74d8db80, address=4, val=3, len=2)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-pci.c:559
#7  0x00007fbb71efa695 in kvm_handle_io (env=0x7fbb74933ae0) at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:147
#8  kvm_run (env=0x7fbb74933ae0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1048
#9  0x00007fbb71efa749 in kvm_cpu_exec (env=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1743
#10 0x00007fbb71efb62d in kvm_main_loop_cpu (_env=0x7fbb74933ae0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2004
#11 ap_main_loop (_env=0x7fbb74933ae0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2060
#12 0x00007fbb7182f851 in start_thread () from /lib64/libpthread.so.0
#13 0x00007fbb6f8ee90d in clone () from /lib64/libc.so.6
(gdb) bt full
#0  0x00007fbb6f8388a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007fbb6f83a085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007fbb71ed73f2 in hw_error (fmt=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:508
        ap = {{gp_offset = 8, fp_offset = 48, overflow_arg_area = 0x7fbb63ffe9a0, reg_save_area = 0x7fbb63ffe8d0}}
        env = 0x0
#3  0x00007fbb71f4507d in register_ioport_write (start=<value optimized out>, length=<value optimized out>, size=6, func=
    0xffffffffffffffff, opaque=0x7fbb63fff700) at /usr/src/debug/qemu-kvm-0.12.1.2/ioport.c:171
        i = <value optimized out>
        bsize = <value optimized out>
#4  0x00007fbb71eedd1a in virtio_map (pci_dev=0x7fbb74d8db80, region_num=<value optimized out>, addr=45120, 
    size=<value optimized out>, type=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-pci.c:533
        proxy = 0x7fbb74d8db80
        vdev = 0x7fbb8a681d20
        config_len = 56
#5  0x00007fbb71ee439b in pci_update_mappings (d=0x7fbb74d8db80) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/pci.c:1067
        r = 0x7fbb74d8dc60
        i = <value optimized out>
        new_addr = 45120
        filtered_size = 64
#6  0x00007fbb71eee4b2 in virtio_write_config (pci_dev=0x7fbb74d8db80, address=4, val=3, len=2)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-pci.c:559
        proxy = 0x7fbb74d8db80
#7  0x00007fbb71efa695 in kvm_handle_io (env=0x7fbb74933ae0) at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:147
        i = <value optimized out>
        ptr = <value optimized out>
#8  kvm_run (env=0x7fbb74933ae0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1048
        r = <value optimized out>
        kvm = 0x7fbb747a3b80
        run = 0x7fbb71e60000
        fd = 18
#9  0x00007fbb71efa749 in kvm_cpu_exec (env=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1743
        r = <value optimized out>
#10 0x00007fbb71efb62d in kvm_main_loop_cpu (_env=0x7fbb74933ae0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2004
        run_cpu = <value optimized out>
#11 ap_main_loop (_env=0x7fbb74933ae0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2060
        env = 0x7fbb74933ae0
        signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
        data = <value optimized out>
#12 0x00007fbb7182f851 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#13 0x00007fbb6f8ee90d in clone () from /lib64/libc.so.6
No symbol table info available.
(gdb) q

Comment 13 Chao Yang 2013-06-25 06:38:46 UTC

Steps to reproduce and verify this issue:
1. launch a guest by:
/usr/libexec/qemu-kvm -M rhel6.5.0 -enable-kvm -m 2048 -smp 2,sockets=1,cores=2,threads=1 -drive file=/home/rhel6.4.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,serial=f82002eb-520c-469b-90c2-663277e90437,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -net none -nodefaults -spice port=8000,disable-ticketing -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -monitor unix:/tmp/test,server,nowait
2. after guest boots up, run a script to keep on hot plugging disks into guest:
cat hotplug_disk.sh 
#!/bin/bash

for i in `seq 4 9` a b c d e f 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f; do
    for j in `seq 1 7` 0; do
        qemu-img create /tmp/resize$i$j.qcow2 1M -f qcow2
        sleep 2
        echo __com.redhat_drive_add id=drv$i$j,file=/tmp/resize$i$j.qcow2 | nc -U /tmp/test
        sleep 2 
        echo device_add virtio-blk-pci,id=dev$i$j,drive=drv$i$j,addr=0x$i.$j,multifunction=on | nc -U /tmp/test
    done
done


Reproduction:
------------
Reproduced this issue with 2.6.32-391.el6.x86_64, qemu-kvm-0.12.1.2-2.355.el6.x86_64. Qemu-kvm crashed during hot adding disks into guest.
(gdb) r -M rhel6.4.0 -enable-kvm -m 2048 -smp 2,sockets=1,cores=2,threads=1 -drive file=/home/rhel6.4.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,serial=f82002eb-520c-469b-90c2-663277e90437,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -net none -nodefaults -spice port=8000,disable-ticketing -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -monitor unix:/tmp/test,server,nowait
Starting program: /usr/libexec/qemu-kvm -M rhel6.4.0 -enable-kvm -m 2048 -smp 2,sockets=1,cores=2,threads=1 -drive file=/home/rhel6.4.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,serial=f82002eb-520c-469b-90c2-663277e90437,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -net none -nodefaults -spice port=8000,disable-ticketing -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -monitor unix:/tmp/test,server,nowait
[Thread debugging using libthread_db enabled]
[New Thread 0x7ffff005b700 (LWP 9911)]
[New Thread 0x7fffee4bd700 (LWP 9912)]
[New Thread 0x7fffe7fff700 (LWP 9913)]
[New Thread 0x7fffe75fe700 (LWP 9914)]

[Thread 0x7ffff005b700 (LWP 9911) exited]
[New Thread 0x7ffff005b700 (LWP 9916)]
[Thread 0x7ffff005b700 (LWP 9916) exited]
[New Thread 0x7ffff005b700 (LWP 9918)]
[Thread 0x7ffff005b700 (LWP 9918) exited]
[New Thread 0x7ffff005b700 (LWP 9919)]
[Thread 0x7ffff005b700 (LWP 9919) exited]
[New Thread 0x7ffff005b700 (LWP 9920)]
[Thread 0x7ffff005b700 (LWP 9920) exited]
[New Thread 0x7ffff005b700 (LWP 9921)]
[Thread 0x7ffff005b700 (LWP 9921) exited]
[New Thread 0x7ffff005b700 (LWP 9929)]
qemu: hardware error: register_ioport_write: invalid opaque
CPU #0:
RAX=0000000000000003 RBX=ffff88007e5bc400 RCX=0000000000000004 RDX=0000000000000cfc
RSI=0000000000000000 RDI=0000000000000097 RBP=ffff88007e711960 RSP=ffff88007e711930
R8 =0000000000000002 R9 =0000000000000003 R10=00000000000000c8 R11=0000000000000000
R12=0000000000000246 R13=0000000000000003 R14=ffff88007d4e0090 R15=ffff88007e711d00
RIP=ffffffff8143095a RFL=00000006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0000 0000000000000000 ffffffff 00000000
GS =0000 ffff880002200000 ffffffff 00000000
LDT=0000 0000000000000000 ffffffff 00000000
TR =0040 ffff880002214200 00002087 00008b00 DPL=0 TSS64-busy
GDT=     ffff880002204000 0000007f
IDT=     ffffffff81dde000 00000fff
CR0=8005003b CR2=00007fc778a9ac98 CR3=0000000037eef000 CR4=000006f0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000ff0000 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=4954504f0045444f4d0050554f524700 XMM05=40404040404040404040404040404040
XMM06=5b5b5b5b5b5b5b5b5b5b5b5b5b5b5b5b XMM07=20202020202020202020202020202020
XMM08=00000000000000000000000000000000 XMM09=ffffff0000000000ff00000000000000
XMM10=00202020202020202000202020202020 XMM11=ffffffffffffffffffffffffffffffff
XMM12=00000000000000000000000000000000 XMM13=00000000000000000000000000000000
XMM14=00000000000000000000000000000000 XMM15=00000000000000000000000000000000
CPU #1:
RAX=0000000000000000 RBX=0000000000000001 RCX=0000000000000000 RDX=0000000000000000
RSI=0000000000000001 RDI=ffffffff81de1228 RBP=ffff88007e4fbed8 RSP=ffff88007e4fbed8
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=ffffffff81c07a40 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=ffffffff8103b92b RFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0000 0000000000000000 ffffffff 00000000
GS =0000 ffff880002300000 ffffffff 00000000
LDT=0000 0000000000000000 ffffffff 00000000
TR =0040 ffff880002314200 00002087 00008b00 DPL=0 TSS64-busy
GDT=     ffff880002304000 0000007f
IDT=     ffffffff81dde000 00000fff
CR0=8005003b CR2=00007fc776e323d1 CR3=000000007b51c000 CR4=000006e0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=000000000000000000ff000000000000 XMM01=2f736563697665642f0062642d696370
XMM02=00000000000000000000000000000000 XMM03=00000000ffff0000ff00000000000000
XMM04=00007f005943494c4f504f4e5f4e4f49 XMM05=40404040404040404040404040404040
XMM06=5b5b5b5b5b5b5b5b5b5b5b5b5b5b5b5b XMM07=20202020202020202020202020202020
XMM08=00000000000000000000000000000000 XMM09=ffffff0000000000ff00000000000000
XMM10=00202020202020202000202020202020 XMM11=ffffffffffffffffffffffffffffffff
XMM12=00000000000000000000000000000000 XMM13=00000000000000000000000000000000
XMM14=00000000000000000000000000000000 XMM15=00000000000000000000000000000000

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffee4bd700 (LWP 9912)]
0x00007ffff57428a5 in raise () from /lib64/libc.so.6
(gdb) 
(gdb) bt
#0  0x00007ffff57428a5 in raise () from /lib64/libc.so.6
#1  0x00007ffff5744085 in abort () from /lib64/libc.so.6
#2  0x00007ffff7de0732 in hw_error (fmt=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:505
#3  0x00007ffff7e4dcdd in register_ioport_write (start=<value optimized out>, length=<value optimized out>, size=6, func=0xffffffffffffffff, 
    opaque=0x7fffee4bd700) at /usr/src/debug/qemu-kvm-0.12.1.2/ioport.c:171
#4  0x00007ffff7df6e3a in virtio_map (pci_dev=<value optimized out>, region_num=<value optimized out>, addr=45120, size=<value optimized out>, 
    type=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-pci.c:533
#5  0x00007ffff7ded5db in pci_update_mappings (d=0x7ffff9162b80) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/pci.c:1067
#6  0x00007ffff7df75d2 in virtio_write_config (pci_dev=0x7ffff9162b80, address=4, val=3, len=2)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-pci.c:559
#7  0x00007ffff7e03825 in kvm_handle_io (env=0x7ffff8858370) at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:147
#8  kvm_run (env=0x7ffff8858370) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1048
#9  0x00007ffff7e038d9 in kvm_cpu_exec (env=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1743
#10 0x00007ffff7e047bd in kvm_main_loop_cpu (_env=0x7ffff8858370) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2004
#11 ap_main_loop (_env=0x7ffff8858370) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2060
#12 0x00007ffff7739851 in start_thread () from /lib64/libpthread.so.0
#13 0x00007ffff57f890d in clone () from /lib64/libc.so.6



Verification:
-------------
Verified with qemu-kvm-0.12.1.2-2.376.el6.x86_64, 2.6.32-391.el6.x86_64. Finished adding all disks into guest specified in script mentioned in step 2, no SIGABRT happened.
But qemu-kvm complained:
qemu-kvm: virtio_pci_set_host_notifier_internal: unable to map ioeventfd: -28
qemu-kvm: virtio_pci_start_ioeventfd: failed. Fallback to a userspace (slower).

Comment 14 Chao Yang 2013-06-25 06:41:16 UTC

Hi, Amos,
 Can you take a look at above complaint? Is it another bug?

Comment 15 Amos Kong 2013-06-25 06:47:32 UTC

(In reply to chayang from comment #14)
> Hi, Amos,
>  Can you take a look at above complaint? Is it another bug?

The complaint is normal/expected, we don't have enough ioeventfd in host kernel.

Comment 16 Chao Yang 2013-06-25 06:56:32 UTC

This bug has been fixed correctly as per Comment #13, #14 and #15.

Comment 20 errata-xmlrpc 2013-11-21 06:48:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-1553.html

Note You need to log in before you can comment on or make changes to this bug.