1447548 – Qemu core dump when do device_add.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1447548 - Qemu core dump when do device_add.

Summary: Qemu core dump when do device_add.

Keywords:
Status:	CLOSED DUPLICATE of bug 1449031
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	qemu-kvm-rhev
Sub Component:
Version:	7.4
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Markus Armbruster
QA Contact:	CongLi
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-05-03 06:54 UTC by CongLi
Modified:	2017-07-23 04:41 UTC (History)
CC List:	11 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-05-10 07:07:38 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description CongLi 2017-05-03 06:54:50 UTC

Description of problem:
There was a typo error when did device_add, and got expected error info, but qemu core dump when I tried to correct and re-run device_add.
I have not reproduced this problem now for more than 10 times.

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.9.0-2.el7.x86_64


How reproducible:
only once

Steps to Reproduce:
1. Boot up a RHEL.7.4 guest.
2. Create a block device via 'blockdev-add'.
{
    "execute": "blockdev-add",
        "arguments": {
            "node-name":"drive3",
            "driver": "qcow2",
            "file": {
                "driver":"iscsi",
                "transport":"tcp",
                "portal":"***",
                "initiator-name":"***",
                "target":"***",
                "lun":0
            }
        }
}
{"return": {}}
3. Attach a device to node in step 2.
There was a typo for drive, got the expected error info.
{
    "execute":"device_add",
        "arguments":{
            "driver":"virtio-blk-pci",
            "drive":"drive2",
            "id":"blk1"
                   }
}
{"error": {"class": "GenericError", "desc": "Property 'virtio-blk-device.drive' can't find value 'drive2'"}}
4. Then I tried it again, qemu core dump and qmp disconnected.
{
    "execute":"device_add",
        "arguments":{
            "driver":"virtio-blk-pci",
            "drive":"drive3",
            "id":"blk1"
                   }
}

Connection closed by foreign host.



Actual results:
qemu core dump

Expected results:
qemu works well

Additional info:
1. dump info:
(gdb) bt
#0  0x000055fd8b5049b1 in memory_listener_register (listener=listener@entry=0x55fd8f0ba260, as=as@entry=0x55fd8f0ba210) at /usr/src/debug/qemu-2.9.0/memory.c:2381
#1  0x000055fd8b4b53f7 in address_space_init_dispatch (as=as@entry=0x55fd8f0ba210)
    at /usr/src/debug/qemu-2.9.0/exec.c:2561
#2  0x000055fd8b504bd7 in address_space_init (as=0x55fd8f0ba210, root=0x55fd8f0ba320, name=0x55fd8f0ba0b8 "") at /usr/src/debug/qemu-2.9.0/memory.c:2425
#3  0x000055fd8b675f9f in pci_qdev_realize (errp=0x7ffe81a73560, devfn=<optimized out>, name=0x55fd8dd47ca0 "virtio-blk-pci", bus=0x55fd8e5c2000, pci_dev=0x55fd8f0ba000)
    at hw/pci/pci.c:1006
#4  0x000055fd8b675f9f in pci_qdev_realize (qdev=0x55fd8f0ba000, errp=0x7ffe81a73560)
    at hw/pci/pci.c:1994
#5  0x000055fd8b61a461 in device_set_realized (obj=<optimized out>, value=<optimized out>, errp=0x7ffe81a73698) at hw/core/qdev.c:939
#6  0x000055fd8b700b0e in property_set_bool (obj=0x55fd8f0ba000, v=<optimized out>, name=<optimized out>, opaque=0x55fd8ffcc070, errp=0x7ffe81a73698) at qom/object.c:1860
#7  0x000055fd8b7047cf in object_property_set_qobject (obj=0x55fd8f0ba000, value=<optimized out>, name=0x55fd8b82a9ab "realized", errp=0x7ffe81a73698) at qom/qom-qobject.c:27
#8  0x000055fd8b702640 in object_property_set_bool (obj=0x55fd8f0ba000, value=<optimized out>, name=0x55fd8b82a9ab "realized", errp=0x7ffe81a73698) at qom/object.c:1163
#9  0x000055fd8b5c5a43 in qdev_device_add (opts=opts@entry=0x55fd8dd4d480, errp=errp@entry=0x7ffe81a73770) at qdev-monitor.c:623
#10 0x000055fd8b5c5fd3 in qmp_device_add (qdict=<optimized out>, ret_data=<optimized out>,---Type <return> to continue, or q <return> to quit---
 errp=0x7ffe81a737b8) at qdev-monitor.c:800
#11 0x000055fd8b7b3329 in qmp_dispatch (errp=0x7ffe81a737b0, request=0x55fd90d66400, cmds=0x55fd8bdb4260 <qmp_commands>) at qapi/qmp-dispatch.c:104
#12 0x000055fd8b7b3329 in qmp_dispatch (cmds=0x55fd8bdb4260 <qmp_commands>, request=request@entry=0x55fd90d66400) at qapi/qmp-dispatch.c:131
#13 0x000055fd8b4eeb04 in handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.9.0/monitor.c:3830
#14 0x000055fd8b7b87b8 in json_message_process_token (lexer=0x55fd8dd20088, input=0x55fd8dd22d40, type=JSON_RCURLY, x=1, y=108) at qobject/json-streamer.c:105
#15 0x000055fd8b7d4a1b in json_lexer_feed_char (lexer=lexer@entry=0x55fd8dd20088, ch=125 '}', flush=flush@entry=false) at qobject/json-lexer.c:319
#16 0x000055fd8b7d4ade in json_lexer_feed (lexer=0x55fd8dd20088, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:369
#17 0x000055fd8b7b8879 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:124
#18 0x000055fd8b4ed55b in monitor_qmp_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.9.0/monitor.c:3873
#19 0x000055fd8b7704fd in tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=<optimized out>) at chardev/char-socket.c:414
#20 0x00007f6a910654c9 in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#21 0x000055fd8b7bdf8c in main_loop_wait () at util/main-loop.c:213
#22 0x000055fd8b7bdf8c in main_loop_wait (timeout=<optimized out>)
---Type <return> to continue, or q <return> to quit---
    at util/main-loop.c:261
#23 0x000055fd8b7bdf8c in main_loop_wait (nonblocking=nonblocking@entry=0)
    at util/main-loop.c:517
#24 0x000055fd8b4adfbc in main () at vl.c:1898
#25 0x000055fd8b4adfbc in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4720

2. Qemu CML:
MALLOC_PERTURB_=1  /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga cirrus  \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3 \
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel74-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -m 8192  \
    -smp 4,cores=2,threads=1,sockets=2  \
    -cpu 'Westmere',+kvm_pv_unhalt \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -device virtio-net-pci,mac=9a:2c:2d:2e:2f:30,id=idC9ewyt,vectors=4,netdev=idOdd6bW,bus=pci.0,addr=0x4  \
    -netdev tap,id=idOdd6bW,vhost=on \
    -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 \
    -device virtio-serial \
    -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 \
    -qmp tcp:localhost:4444,server,nowait \

3. host info:
processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5645  @ 2.40GHz
stepping	: 2
microcode	: 0x13
cpu MHz		: 2394.000
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 10
cpu cores	: 6
apicid		: 21
initial apicid	: 21
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb tpr_shadow vnmi flexpriority ept vpid dtherm arat
bogomips	: 4800.13
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

# free -m
              total        used        free      shared  buff/cache   available
Mem:          31968        3361       24293          24        4312       28119
Swap:         16127           0       16127

Comment 4 Markus Armbruster 2017-05-05 15:06:35 UTC

Two quick questions:

1. Can you reproduce with a "file" backend rather than "iscsi"?

2. Can you reproduce with -S?

Comment 5 CongLi 2017-05-10 02:09:02 UTC

(In reply to Markus Armbruster from comment #4)
> Two quick questions:
> 
> 1. Can you reproduce with a "file" backend rather than "iscsi"?
> 
> 2. Can you reproduce with -S?

Hi Markus,

I only met this issue once and I still can not reproduce this issue over 50 times, also with a 'file' backend or '-S'.

But there is a similar bug which has a stable reproducer.
Bug 1449031 - qemu core dump when hot-unplug/hot-plug scsi controller in turns

Could you please confirm whether it's a dup?

Thanks.

Comment 6 Markus Armbruster 2017-05-10 07:07:38 UTC

Thanks for the pointer to bug 1449031!  I figure this is indeed a
duplicate of bug 1449031, because

1. The top of the stack backtrace is identical.  Then it diverges, but
the two variants are similar: both are device realization on behalf of
QMP device_add.

2. The reproducers are both about unplugging and re-plugging a
virtio-blk device.

If we could reproduce this bug, I'd like us to verify the fix for bug
1449031 also fixes this one, but right now we're not having any luck
with reproducing it.

*** This bug has been marked as a duplicate of bug 1449031 ***

Comment 7 Ladi Prosek 2017-05-30 08:12:27 UTC

The problem, at least the one we're aware of, is in virtio-scsi and virtio-serial. If this BZ is a duplicate of bug 1449031, there must have been a hot-unplug of either of these device prior to steps 2. - 4. in comment 0.

If this crash can be reproduced with virtio-blk only, it is definitely a separate bug.

In general, as of commit c611c764, virtio devices must have their ref-counts done right so that virtio_device_instance_finalize runs and unregisters the memory listener before the underlying memory disappears. Any ref-counting issue will likely lead to a crash similar to this one.

Note You need to log in before you can comment on or make changes to this bug.