RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1186765 - libvirtd crashes after chardev hotplug crashes qemu
Summary: libvirtd crashes after chardev hotplug crashes qemu
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Ján Tomko
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 1161024
Blocks: 1195155
TreeView+ depends on / blocked
 
Reported: 2015-01-28 14:02 UTC by Ján Tomko
Modified: 2015-11-19 06:08 UTC (History)
13 users (show)

Fixed In Version: libvirt-1.2.13-1.el7
Doc Type: Bug Fix
Doc Text:
When attaching a character device, libvirt did not correctly check the status of the domain after the libvirtd daemon exited the monitor operation. As a consequence, freed data from the domain definition could be accessed even when the domain had terminated unexpectedly, which caused libvirtd to crash as well. Now, libvirtd verifies that the domain is online before accessing its definition, which prevents libvirtd from crashing in the described situation.
Clone Of: 1161024
: 1195155 (view as bug list)
Environment:
Last Closed: 2015-11-19 06:08:54 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2202 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2015-11-19 08:17:58 UTC

Description Ján Tomko 2015-01-28 14:02:44 UTC 
+++ This bug was initially created as a clone of Bug #1161024 +++
--- Additional comment from Luyao Huang on 2015-01-27 11:36:19 CET ---

Hi Jan,

when i try to verify this bug, but i still found libvirtd still crashed in these case(different reason):

1.# virsh list --all
 Id    Name                           State
----------------------------------------------------
 2     r7                             running

2.# cat lxcconsole.xml
  <console type='pty'>
      <target type='virtio' port='1'/>
    </console>

3. use gdb attach libvirtd set breakpoint at qemuDomainAttachChrDevice

4. # virsh attach-device r7 lxcconsole.xml

5.open another terminal kill qemu

# kill -11 26914


6.libvirtd failed to exit monitor then goto cleanup and return -1, but libvirtd will crash in qemuDomainAttachDeviceFlags:

1492	    if (qemuDomainChrInsert(vmdef, chr) < 0)
(gdb) 
1496	    qemuDomainObjEnterMonitor(driver, vm);
(gdb) 
1497	    if (qemuMonitorAttachCharDev(priv->mon, charAlias, &chr->source) < 0) {
(gdb) n
1509	        if (qemuDomainObjExitMonitor(driver, vm) < 0) {
(gdb) n
1530	    VIR_FREE(charAlias);
(gdb) 
1531	    VIR_FREE(devstr);
(gdb) 
1533	}
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00007fe09158856c in free () from /lib64/libc.so.6
(gdb) bt
#0  0x00007fe09158856c in free () from /lib64/libc.so.6
#1  0x00007fe09466effa in virFree (ptrptr=ptrptr@entry=0x7fe075d3a068) at util/viralloc.c:582
#2  0x00007fe0946e2739 in virDomainDeviceInfoClear (info=0x7fe075d3a068) at conf/domain_conf.c:2709
#3  0x00007fe0946e28fb in virDomainChrDefFree (def=0x7fe075d3a020) at conf/domain_conf.c:1651
#4  0x00007fe0946f0de9 in virDomainDeviceDefFree (def=def@entry=0x7fe0740040c0) at conf/domain_conf.c:1942
#5  0x00007fe07daa98ab in qemuDomainAttachDeviceFlags (dom=<optimized out>, xml=<optimized out>, flags=<optimized out>) at qemu/qemu_driver.c:7646
#6  0x00007fe094765dd6 in virDomainAttachDevice (domain=domain@entry=0x7fe075d36300, xml=0x7fe075aa3fe0 "  <console type='pty'>\n      <target type='virtio'/>\n    </console>\n") at libvirt.c:10385
#7  0x00007fe09520aaa0 in remoteDispatchDomainAttachDevice (server=<optimized out>, msg=<optimized out>, args=0x7fe074da12a0, rerr=0x7fe085af6c80, client=<optimized out>) at remote_dispatch.h:2485
#8  remoteDispatchDomainAttachDeviceHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7fe085af6c80, args=0x7fe074da12a0, ret=<optimized out>) at remote_dispatch.h:2463
#9  0x00007fe0947c4382 in virNetServerProgramDispatchCall (msg=0x7fe095aa1ae0, client=0x7fe095bbbab0, server=0x7fe095a91f10, prog=0x7fe095a9f000) at rpc/virnetserverprogram.c:437
#10 virNetServerProgramDispatch (prog=0x7fe095a9f000, server=server@entry=0x7fe095a91f10, client=0x7fe095bbbab0, msg=0x7fe095aa1ae0) at rpc/virnetserverprogram.c:307
#11 0x00007fe0952183fd in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7fe095a91f10) at rpc/virnetserver.c:172
#12 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7fe095a91f10) at rpc/virnetserver.c:193
#13 0x00007fe0946c7fa5 in virThreadPoolWorker (opaque=opaque@entry=0x7fe095a71b90) at util/virthreadpool.c:145
#14 0x00007fe0946c793e in virThreadHelper (data=<optimized out>) at util/virthread.c:197
#15 0x00007fe091ce7df5 in start_thread (arg=0x7fe085af7700) at pthread_create.c:308
#16 0x00007fe0915fe1ad in clone () from /lib64/libc.so.6
(gdb) 

So would you please help me to check out if it is the same issue? or need i open a new bug?

--- Additional comment from Jan Tomko on 2015-01-27 15:21:38 CET ---

This is the same issue, for attaching chardevs, the series didn't remove usage of freed data completely. Upstream patch posted:
https://www.redhat.com/archives/libvir-list/2015-January/msg00973.html

--- Additional comment from Luyao Huang on 2015-01-27 15:24:19 CET ---

(In reply to Jan Tomko from comment #18)
> This is the same issue, for attaching chardevs, the series didn't remove
> usage of freed data completely. Upstream patch posted:
> https://www.redhat.com/archives/libvir-list/2015-January/msg00973.html

Thanks a lot for your reply!

--- Additional comment from Jan Tomko on 2015-01-28 12:21:32 CET ---

v2 of the patch:
https://www.redhat.com/archives/libvir-list/2015-January/msg00993.html

Pushed upstream as:
commit daf51be5f1b0f7b41c0813d43d6b66edfbe4f6d9
    Split qemuDomainChrInsert into two parts
commit 21e0e8866e341da74e296ca3cf2d97812e847a66
    hotplug: only add a chardev to vmdef after monitor call
git describe: v1.2.12-29-g21e0e88
Comment 4 Hu Jianwei 2015-05-20 03:40:59 UTC 
I met the unclear error message during testing, maybe it was caused by below bug's patch. The correct message should be like this:
error: operation failed: domain is no longer running

Bug 1196934 - libvirt sometimes output useless error when qemu failed to start/migrate/restore
qemu: do not overwrite the error in qemuDomainObjExitMonitor


Steps for reproduced and verified:
I can reproduce it in old libvirt
[root@localhost ~]# rpm -q libvirt
libvirt-1.2.8-16.el7.x86_64

[root@localhost ~]# virsh attach-device r71 console.xml
error: Failed to attach device from console.xml
error: Cannot recv data: Connection reset by peer
error: Failed to reconnect to the hypervisor

...
1496	    qemuDomainObjEnterMonitor(driver, vm);
(gdb) s
qemuDomainObjEnterMonitor (driver=driver@entry=0x7eff2c1ac640, obj=obj@entry=0x7eff2c234930) at qemu/qemu_domain.c:1605
1605	    ignore_value(qemuDomainObjEnterMonitorInternal(driver, obj,
(gdb) c
Continuing.

Program received signal SIGABRT, Aborted.
0x00007eff48e325d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);

Check it on latest fixed version, no crash happened.
[root@localhost ~]# rpm -q libvirt qemu-kvm-rhev
libvirt-1.2.15-2.el7.x86_64
qemu-kvm-rhev-2.3.0-1.el7.x86_64

[root@localhost ~]# ps aux | grep libvirtd| grep -v grep
root      4003  0.0  0.2 1116280 22240 ?       Ssl  11:04   0:00 /usr/sbin/libvirtd

[root@localhost network-scripts]# gdb -p `pidof libvirtd`
(gdb) info b
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00007fb0cc1535e0 in qemuDomainAttachChrDevice at qemu/qemu_hotplug.c:1538
[root@localhost ~]# virsh start r71
Domain r71 started

[root@localhost ~]# virsh attach-device r71 console.xml
error: Failed to attach device from console.xml
error: internal error: End of file from monitor     <==== here is a new issue.

During executing attach-device action, libvirtd will hit below breakpoint.
Breakpoint 1, qemuDomainAttachChrDevice (driver=driver@entry=0x7fb0c412db20, vm=vm@entry=0x7fb0c421e2b0, chr=0x7fb0c4245290) at qemu/qemu_hotplug.c:1538
1538	{
(gdb) n
1540	    qemuDomainObjPrivatePtr priv = vm->privateData;
(gdb) 
1541	    virDomainDefPtr vmdef = vm->def;
(gdb) 
1538	{
(gdb) 
1547	    if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_DEVICE)) {
(gdb) 
1542	    char *devstr = NULL;
(gdb) 
1538	{
(gdb) 
1547	    if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_DEVICE)) {
(gdb) 
1543	    char *charAlias = NULL;
(gdb) 
1547	    if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_DEVICE)) {
(gdb) 
1553	    if (qemuAssignDeviceChrAlias(vmdef, chr, -1) < 0)
(gdb) 
1556	    if (chr->deviceType == VIR_DOMAIN_CHR_DEVICE_TYPE_CONSOLE &&
(gdb) 
1560	    if (virDomainVirtioSerialAddrAutoAssign(NULL,
(gdb) 
1567	    if (qemuBuildChrDeviceStr(&devstr, vm->def, chr, priv->qemuCaps) < 0)
(gdb) 
1570	    if (virAsprintf(&charAlias, "char%s", chr->info.alias) < 0)
(gdb) 
1573	    if (qemuDomainChrPreInsert(vmdef, chr) < 0)
(gdb) 
1576	    qemuDomainObjEnterMonitor(driver, vm);
(gdb) s
qemuDomainObjEnterMonitor (driver=driver@entry=0x7fb0c412db20, obj=obj@entry=0x7fb0c421e2b0) at qemu/qemu_domain.c:1637
1637	    ignore_value(qemuDomainObjEnterMonitorInternal(driver, obj,
(gdb) n
qemuDomainObjEnterMonitorInternal (driver=driver@entry=0x7fb0c412db20, obj=obj@entry=0x7fb0c421e2b0, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE) at qemu/qemu_domain.c:1579
1579	{
(gdb) c
Continuing.

[root@localhost ~]# ps aux | grep libvirtd| grep -v grep
root      4003  0.0  0.2 1116280 22520 ?       Ssl  11:04   0:00 /usr/sbin/libvirtd
Comment 5 Hu Jianwei 2015-06-30 09:35:11 UTC 
Is there any suggestion for comment 4? 

Thanks.
Comment 6 Ján Tomko 2015-06-30 14:34:37 UTC 
"End of file from monitor" is more specific than "domain is no longer running", because it says why libvirt no longer thinks the domain is running.

Even though the less specific error message is more human-readable, using it instead of the more specific one would make debugging harder.
Comment 7 Hu Jianwei 2015-07-01 02:57:19 UTC 
Great, I agree with you.

According to comment 4 and comment 6, moved to Verified.
Comment 9 errata-xmlrpc 2015-11-19 06:08:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html
Note You need to log in before you can comment on or make changes to this bug.