RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1181408 - Libvirtd crash while hotplug the guest agent without target type for many times
Summary: Libvirtd crash while hotplug the guest agent without target type for many times
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.1
Hardware: x86_64
OS: Unspecified
unspecified
high
Target Milestone: rc
: ---
Assignee: Ján Tomko
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-01-13 06:03 UTC by zhenfeng wang
Modified: 2015-03-05 07:49 UTC (History)
8 users (show)

Fixed In Version: libvirt-1.2.8-13.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-03-05 07:49:07 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
The libvirtd crash coredump info (11.69 KB, text/plain)
2015-01-13 06:05 UTC, zhenfeng wang 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:0323 0 normal SHIPPED_LIVE Low: libvirt security, bug fix, and enhancement update 2015-03-05 12:10:54 UTC

Description zhenfeng wang 2015-01-13 06:03:30 UTC 
Description of problem:
Libvirtd crash while hotplug the guest agent without target type for many times
Version-Release number of selected component (if applicable):
host:
libvirt-1.2.8-12.el7.x86_64
kernel-3.10.0-222.el7.x86_64
qemu-kvm-rhev-2.1.2-18.el7.x86_64

guest
qemu-guest-agent-2.1.0-4.el7.x86_64

How reproducible:
100%

Steps to Reproduce:

1.Prepare a guest with the following xml
#virsh dumpxml rhel7
--
<channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/rhel7f.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>

2.Start the guest, then install the qemu-guest-agent service inside the guest
#virsh start rhel7.0
inside guest# yum install qemu-guest-agent
inside guest# systemctl start qemu-guest-agent

3.Prepare a guest agent xml without target type configured in the xml
#cat agent.xml
<channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/rhel7f.org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='7'/>
    </channel>

4.Hotplug the guest agent for serveral times, the libvirtd will crash

# virsh attach-device rhel7f agent.xml
error: Failed to attach device from agent.xml
error: invalid argument: device not present in domain configuration

# virsh attach-device rhel7f agent.xml
error: Failed to attach device from agent.xml
error: invalid argument: device not present in domain configuration

# virsh attach-device rhel7f agent.xml
^[[Aerror: Failed to attach device from agent.xml
error: End of file while reading data: Input/output error
error: Failed to reconnect to the hypervisor


5.check the coredump info
# gdb -c coredump
Core was generated by `/usr/sbin/libvirtd'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f630df8dd3c in __strncmp_sse42 () from /lib64/libc.so.6

--
Thread 1 (Thread 0x7f6301e8f700 (LWP 23692)):
#0  0x00007f630df8dd3c in __strncmp_sse42 () from /lib64/libc.so.6
#1  0x00007f62f9e13d0c in qemuDomainDeviceAliasIndex (prefix=prefix@entry=0x7f62f9eb45e8 "channel", info=<optimized out>) at qemu/qemu_command.c:591
#2  0x00007f62f9e16897 in qemuGetNextChrDevIndex (chr=0x7f62f4252560, prefix=0x7f62f9eb45e8 "channel", def=0x2) at qemu/qemu_command.c:924
#3  qemuAssignDeviceChrAlias (def=def@entry=0x7f62e4009460, chr=chr@entry=0x7f62f4252560, idx=idx@entry=-1) at qemu/qemu_command.c:967
#4  0x00007f62f9e3ba1f in qemuDomainAttachChrDevice (driver=driver@entry=0x7f62f40dbe70, vm=vm@entry=0x7f62f41e5bb0, chr=0x7f62f4252560) at qemu/qemu_hotplug.c:1468
#5  0x00007f62f9e9fe5e in qemuDomainAttachDeviceLive (dom=0x7f62f424d160, dev=0x7f62f424fb90, vm=0x7f62f41e5bb0) at qemu/qemu_driver.c:7011
#6  qemuDomainAttachDeviceFlags (dom=0x7f62f424d160, xml=<optimized out>, flags=<optimized out>) at qemu/qemu_driver.c:7563
#7  0x00007f6310e37a86 in virDomainAttachDevice (domain=domain@entry=0x7f62f424d160,
    xml=0x7f62f423e4e0 "    <channel type='unix'>\n      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/rhel7.0.org.qemu.guest_agent.0'/>\n      <address type='virtio-serial' controller='0' bus='0' port='7'/>\n "...) at libvirt.c:10385
#8  0x00007f63118c9a90 in remoteDispatchDomainAttachDevice (server=<optimized out>, msg=<optimized out>, args=0x7f62f4495090, rerr=0x7f6301e8ec80, client=<optimized out>) at remote_dispatch.h:2485
#9  remoteDispatchDomainAttachDeviceHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7f6301e8ec80, args=0x7f62f4495090, ret=<optimized out>) at remote_dispatch.h:2463
#10 0x00007f6310e96032 in virNetServerProgramDispatchCall (msg=0x7f6312f8f860, client=0x7f6312f8f9d0, server=0x7f6312f806b0, prog=0x7f6312f8c730) at rpc/virnetserverprogram.c:437
#11 virNetServerProgramDispatch (prog=0x7f6312f8c730, server=server@entry=0x7f6312f806b0, client=0x7f6312f8f9d0, msg=0x7f6312f8f860) at rpc/virnetserverprogram.c:307
---Type <return> to continue, or q <return> to quit---
#12 0x00007f63118d73ed in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f6312f806b0) at rpc/virnetserver.c:172
#13 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f6312f806b0) at rpc/virnetserver.c:193
#14 0x00007f6310d99cb5 in virThreadPoolWorker (opaque=opaque@entry=0x7f6312f74db0) at util/virthreadpool.c:145
#15 0x00007f6310d9964e in virThreadHelper (data=<optimized out>) at util/virthread.c:197
#16 0x00007f630e62adf5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f630df511ad in clone () from /lib64/libc.so.6
(gdb)

Actual results:
libvirtd crash

Expected results:
libvirtd shouldn't crash
Comment 1 zhenfeng wang 2015-01-13 06:05:22 UTC 
Created attachment 979480 [details]
The libvirtd crash coredump info
Comment 2 zhenfeng wang 2015-01-13 06:19:52 UTC 
can Also hit this issue by hotplug the spicevmc without target type for many times
# cat spice.xml 
    <channel type='spicevmc'>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>

# virsh attach-device rhel7f spice.xml 
error: Failed to attach device from spice.xml
error: invalid argument: device not present in domain configuration

[root@zhwangrhel71 ~]# virsh attach-device rhel7f spice.xml 
error: Failed to attach device from spice.xml
error: End of file while reading data: Input/output error
error: Failed to reconnect to the hypervisor

Thread 1 (Thread 0x7faef0238700 (LWP 2454)):
#0  0x00007faefcb37d3c in __strncmp_sse42 () from /lib64/libc.so.6
#1  0x00007faee89bdd0c in qemuDomainDeviceAliasIndex (prefix=prefix@entry=0x7faee8a5e5e8 "channel", info=<optimized out>) at qemu/qemu_command.c:591
#2  0x00007faee89c0897 in qemuGetNextChrDevIndex (chr=0x7faed4000c60, prefix=0x7faee8a5e5e8 "channel", def=0x1) at qemu/qemu_command.c:924
#3  qemuAssignDeviceChrAlias (def=def@entry=0x7faee01f9930, chr=chr@entry=0x7faed4000c60, idx=idx@entry=-1) at qemu/qemu_command.c:967
#4  0x00007faee89e5a1f in qemuDomainAttachChrDevice (driver=driver@entry=0x7faee00efa30, vm=vm@entry=0x7faee020bcf0, chr=0x7faed4000c60) at qemu/qemu_hotplug.c:1468
#5  0x00007faee8a49e5e in qemuDomainAttachDeviceLive (dom=0x7faed4000a90, dev=0x7faed4000b90, vm=0x7faee020bcf0) at qemu/qemu_driver.c:7011
#6  qemuDomainAttachDeviceFlags (dom=0x7faed4000a90, xml=<optimized out>, flags=<optimized out>) at qemu/qemu_driver.c:7563
#7  0x00007faeff9e1a86 in virDomainAttachDevice (domain=domain@entry=0x7faed4000a90, 
    xml=0x7faed40011a0 "    <channel type='spicevmc'>\n      <alias name='channel0'/>\n      <address type='virtio-serial' controller='0' bus='0' port='2'/>\n    </channel>\n") at libvirt.c:10385
#8  0x00007faf00473a90 in remoteDispatchDomainAttachDevice (server=<optimized out>, msg=<optimized out>, args=0x7faed4000bb0, rerr=0x7faef0237c80, client=<optimized out>) at remote_dispatch.h:2485
#9  remoteDispatchDomainAttachDeviceHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7faef0237c80, args=0x7faed4000bb0, ret=<optimized out>) at remote_dispatch.h:2463
#10 0x00007faeffa40032 in virNetServerProgramDispatchCall (msg=0x7faf01689090, client=0x7faf01689650, server=0x7faf0167a7e0, prog=0x7faf01685b00) at rpc/virnetserverprogram.c:437
#11 virNetServerProgramDispatch (prog=0x7faf01685b00, server=server@entry=0x7faf0167a7e0, client=0x7faf01689650, msg=0x7faf01689090) at rpc/virnetserverprogram.c:307
#12 0x00007faf004813ed in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7faf0167a7e0) at rpc/virnetserver.c:172
#13 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7faf0167a7e0) at rpc/virnetserver.c:193
#14 0x00007faeff943cb5 in virThreadPoolWorker (opaque=opaque@entry=0x7faf0166ec50) at util/virthreadpool.c:145
#15 0x00007faeff94364e in virThreadHelper (data=<optimized out>) at util/virthread.c:197
#16 0x00007faefd1d4df5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007faefcafb1ad in clone () from /lib64/libc.so.6
(gdb)
Comment 4 Ján Tomko 2015-01-13 09:35:50 UTC 
Fixed upstream by:
commit fba7173f7236c705344aa84bf9715074abdc6ea7
Author:     Luyao Huang <lhuang>
AuthorDate: 2015-01-13 16:41:05 +0800
Commit:     Ján Tomko <jtomko>
CommitDate: 2015-01-13 09:56:56 +0100

    conf: fix crash when hotplug a channel chr device with no target
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1181408
    
    When we try to hotplug a channel chr device with no target, we
    will get success (which should fail) in virDomainChrDefParseXML,
    because we use goto cleanup this place and return an incomplete
    definition (with no target). In qemuDomainAttachChrDevice,
    we add it to the domain definition, but fail to remove it from
    there when chardev-add fails, because virDomainChrRemove
    matches chardevices according to the target name.
    The device definition is then freed in qemuDomainAttachDeviceFlags,
    leaving a stale pointer in the domain definition.
    
    Signed-off-by: Luyao Huang <lhuang>
    Signed-off-by: Ján Tomko <jtomko>

git describe: v1.2.11-144-gfba7173
Comment 7 zhenfeng wang 2015-01-20 07:57:04 UTC 
Verify this bug with libvirt-1.2.8-13.el7
steps
1.Prepare a shutoff guest
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     rhel7f                        shut off

2.Prepare a xml like following
#cat agent.xml
<channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/rhel7f.org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='7'/>
    </channel>

3.Attach the guest agent to the guest, will attach fail to attach the
agent to guest and get the expect error
# virsh attach-device rhel7f agent.xml --current
error: Failed to attach device from agent.xml
error: XML error: target type must be specified for channel device

# virsh attach-device rhel7f agent.xml --config
error: Failed to attach device from agent.xml
error: XML error: target type must be specified for channel device

4.Start the guest, after the guest start comepletly, re-attach the guest agent
to the guest, will also fail to attach the agent to guest and get the expect error, also the libvirtd didn't crash
# virsh attach-device rhel7f agent.xml 
error: Failed to attach device from agent.xml
error: XML error: target type must be specified for channel device

# virsh attach-device rhel7f agent.xml --config
error: Failed to attach device from agent.xml
error: XML error: target type must be specified for channel device

# virsh attach-device rhel7f agent.xml --current
error: Failed to attach device from agent.xml
error: XML error: target type must be specified for channel device

5.Prepare a normal guest agent xml
#cat agent.xml
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/rhel7f.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>

6.The guest agent could be attached successfully, install the guest agent service inside the guest, the guest agent service works as expectly after restart libvirtd service, there was an exsiting bug 1168530 trace this known issue

# virsh attach-device rhel7f agent.xml 
Device attached successfully

7.Detach the guest agent with the invalid guest agent xml, could get the expect error
#cat agent.xml
<channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/rhel7f.org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='7'/>
    </channel>

# virsh detach-device rhel7f agent.xml 
error: Failed to detach device from agent.xml
error: XML error: target type must be specified for channel device

According to the upper steps ,mark this bug verifed
Comment 9 errata-xmlrpc 2015-03-05 07:49:07 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html
Note You need to log in before you can comment on or make changes to this bug.