Bug 1071181
Summary: | Libvirtd crashed on a light loop of starting domain with CTRL_IP_LEARNING=dhcp | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Hu Jianwei <jiahu> | ||||
Component: | libvirt | Assignee: | Laine Stump <laine> | ||||
Status: | CLOSED CURRENTRELEASE | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | medium | ||||||
Version: | 7.0 | CC: | acathrow, bugproxy, dallan, dyuan, hannsj_uhl, honzhang, jdenemar, jmiao, lagarcia, laine, mzhan, stefanb | ||||
Target Milestone: | rc | Keywords: | Patch | ||||
Target Release: | --- | ||||||
Hardware: | x86_64 | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | libvirt-1.1.1-29.el7 | Doc Type: | Bug Fix | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2014-06-13 10:44:30 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 731059, 744225 | ||||||
Attachments: |
|
Do you see an error like the following in the libvirt log? virNWFilterSnoopDHCPOpen:1116 : internal error: setup of pcap handle failed I think this is the root cause for the VM not even starting the first time. This error does not occur with libvirt-1.1.1 on F18 (libpcap-1.3.0-2). RHEL7 has libpcap-1.5.3-3. With libpcap-1.5.3 on FC18 I also need to supply at least a 128kb buffer for libpcap's pcap_set_buffer_szie() otherwise there will be an error related to mmaping the rx buffer ring when pcap_activate() is called. I will submit a patch for this tomorrow (3/3). fyi .. update in LTC bug 73493 - RH731059- [7.0 FEAT] libvirt: Support DHCP Snooping and Dynamic ARP Inspection (DCN) " Patches to fix this issue have been pushed to upstream repo. commit 49b59a151f60b0a178b023b727bac30f80bd6000 Author: Stefan Berger <stefanb.ibm.com> Date: Mon Mar 3 15:13:50 2014 -0500 nwfilter: Increase buffer size for libpcap Libpcap 1.5 requires a larger buffer than previous pcap versions. Adjust the size of the buffer to 128kb. This patch should address symptoms in BZ 1071181 and BZ 731059 Signed-off-by: Stefan Berger <stefanb.ibm.com> commit 64df4c75189b42799f82a8d8816c7c55598d2b6e Author: Stefan Berger <stefanb.ibm.com> Date: Mon Mar 3 15:13:47 2014 -0500 nwfilter: Display the pcap errror message Display the pcap error message in the log. Signed-off-by: Stefan Berger <stefanb.ibm.com> " ... Do the patches above specifically fix the crash reported here? Or do they just improve functionality of DHCP Snooping (Bug 731059)? Do we need to move this BZ to 7.0 and set blocker? (In reply to comment #7) > Do the patches above specifically fix the crash reported here? Or do they > just improve functionality of DHCP Snooping (Bug 731059)? > > Do we need to move this BZ to 7.0 and set blocker? . Comment from Stefan Berger 2014-03-18 10:24:44 EDT With these patches applied the crash did not occur anymore. This commit fixes the actual reason for the crash of libvirtd found in the error path. The above mentioned patches (comment 3) masked the crash by fixing the problem that caused the error. commit 963dcf905c5ee0358d6b0b74b124ff340cbbbd2b Author: Stefan Berger <stefanb.ibm.com> Date: Wed Mar 19 13:38:44 2014 -0400 nwfilter: Fix double free of pointer https://bugzilla.redhat.com/show_bug.cgi?id=1071181 Commit 49b59a15 fixed one problem but masks another one related to pointer freeing. Avoid putting of the virNWFilterSnoopReq once the thread has been started. It belongs to the thread and the thread will call virNWFilterSnoopReqPut() o Signed-off-by: Stefan Berger <stefanb.ibm.com> In latest libvirt-1.1.1-29.el7, the crash problem is gone: # rpm -q libvirt libvirt-1.1.1-29.el7.x86_64 # virsh dumpxml r6 | grep -A10 '<interface' <interface type='network'> <mac address='52:54:00:21:0d:9b'/> <source network='default'/> <target dev='vnet0'/> <model type='virtio'/> <filterref filter='clean-traffic'> <parameter name='CTRL_IP_LEARNING' value='dhcp'/> </filterref> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> # i=1;while true; do echo -----------------$i---------------; virsh start r6;i=$((i + 1)); done -----------------1--------------- Domain r6 started -----------------2--------------- error: Domain is already active -----------------3--------------- error: Domain is already active -----------------4--------------- error: Domain is already active . . . -----------------725--------------- error: Domain is already active -----------------726--------------- error: Domain is already active -----------------727--------------- error: Domain is already active No crash happened. So I change the status to VERIFIED. Change the status to VERIFIED VERIFIED also here now that libvirt-1.1.1-29.el7.x86_64 is available. This request was resolved in Red Hat Enterprise Linux 7.0. Contact your manager or support representative in case you have further questions about the request. ------- Comment From hannsj_uhl.com 2019-04-08 10:20 EDT------- |
Created attachment 868890 [details] libvirtd_debug.log Description of problem: libvirtd crashed on a light loop of starting domain with CTRL_IP_LEARNING=dhcp Version-Release number of selected component (if applicable): libvirt-1.1.1-25.el7.x86_64 qemu-kvm-rhev-1.5.3-50.el7.x86_64 kernel-3.10.0-97.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. Set "dhcp" value to CTRL_IP_LEARNING [root@intel-e5530-8-2 ~]# virsh dumpxml r7 | grep interface -A8 <interface type='network'> <mac address='52:54:00:7f:44:cb'/> <source network='default'/> <model type='rtl8139'/> <filterref filter='clean-traffic'> <parameter name='CTRL_IP_LEARNING' value='dhcp'/> </filterref> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> ... 2. Do a light loop starting the domain. [root@intel-e5530-8-2 ~]# i=1;while true; do echo -----------------$i---------------; virsh start r7;i=$((i + 1)); done -----------------1--------------- error: Failed to start domain r7 error: An error occurred, but the cause is unknown ... -----------------27--------------- error: Failed to start domain r7 error: An error occurred, but the cause is unknown -----------------28--------------- error: Failed to start domain r7 error: End of file while reading data: Input/output error error: One or more references were leaked after disconnect from the hypervisor error: Failed to reconnect to the hypervisor -----------------29--------------- error: failed to connect to the hypervisor error: no valid connection error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused Actual results: As shown above, domain failed to start and caused libvirtd crash. [root@intel-e5530-8-2 ccpp-2014-02-28-14:06:44-4926]# service libvirtd status Redirecting to /bin/systemctl status libvirtd.service libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled) Active: failed (Result: core-dump) since Fri 2014-02-28 14:06:45 CST; 3min 34s ago Process: 4926 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=dumped, signal=ABRT) Main PID: 4926 (code=dumped, signal=ABRT) Feb 28 14:06:44 intel-e5530-8-2.englab.nay.redhat.com libvirtd[4926]: 7f67c9707000-7f67c9708000 rw-p 00000000 00:00 0 Feb 28 14:06:44 intel-e5530-8-2.englab.nay.redhat.com libvirtd[4926]: 7f67c9708000-7f67c975f000 r-xp 00000000 08:01 68017482 /usr/sbin/libvirtd Feb 28 14:06:44 intel-e5530-8-2.englab.nay.redhat.com libvirtd[4926]: 7f67c995f000-7f67c9961000 r--p 00057000 08:01 68017482 /usr/sbin/libvirtd Feb 28 14:06:44 intel-e5530-8-2.englab.nay.redhat.com libvirtd[4926]: 7f67c9961000-7f67c9965000 rw-p 00059000 08:01 68017482 /usr/sbin/libvirtd Feb 28 14:06:44 intel-e5530-8-2.englab.nay.redhat.com libvirtd[4926]: 7f67cadca000-7f67cae9e000 rw-p 00000000 00:00 0 [heap] Feb 28 14:06:44 intel-e5530-8-2.englab.nay.redhat.com libvirtd[4926]: 7fff082cc000-7fff082ed000 rw-p 00000000 00:00 0 [stack] Feb 28 14:06:44 intel-e5530-8-2.englab.nay.redhat.com libvirtd[4926]: 7fff0830b000-7fff0830d000 r-xp 00000000 00:00 0 [vdso] Feb 28 14:06:44 intel-e5530-8-2.englab.nay.redhat.com libvirtd[4926]: ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Feb 28 14:06:45 intel-e5530-8-2.englab.nay.redhat.com systemd[1]: libvirtd.service: main process exited, code=dumped, status=6/ABRT Feb 28 14:06:45 intel-e5530-8-2.englab.nay.redhat.com systemd[1]: Unit libvirtd.service entered failed state. Expected results: Below two detailed problems, should be handled. 1. For "error: An error occurred, but the cause is unknown", domain with CTRL_IP_LEARNING=dhcp should be started successfully. 2. For libvirtd crash, I think libvitd should keep living after meeting above negative behavior. Additional info: (gdb) bt #0 0x00007f67c5dea989 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007f67c5dec098 in __GI_abort () at abort.c:90 #2 0x00007f67c5e2b177 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f67c5f33b48 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196 #3 0x00007f67c5e30f87 in malloc_printerr (action=<optimized out>, str=0x7f67c5f31238 "corrupted double-linked list", ptr=<optimized out>) at malloc.c:4972 #4 0x00007f67c5e325ba in _int_free (av=0x7f6790000020, p=0x7f6790002010, have_lock=0) at malloc.c:3960 #5 0x00007f67c8c62f8a in virFree (ptrptr=0x7f6790009720) at util/viralloc.c:566 #6 0x00007f67c8c8176b in virHashFree (table=0x7f6790009720) at util/virhash.c:264 #7 0x00007f67c8cf1a77 in virNWFilterHashTableFree (table=table@entry=0x7f6790009700) at conf/nwfilter_params.c:684 #8 0x00007f67b39feb5b in virNWFilterInstantiate (forceWithPendingReq=false, driver=0x633a34343a66373a, macaddr=0x7f67ac222fe4, teardownOld=true, foundNewFilter=<optimized out>, useNewFilter=INSTANTIATE_ALWAYS, vars=0x7f67900104a0, linkdev=0x0, ifindex=747, ifname=0x7f6790007a90 "vnet0", filter=0x7f67ac016cd0, nettype=VIR_DOMAIN_NET_TYPE_NETWORK, techdriver=<optimized out>, vmuuid=<optimized out>) at nwfilter/nwfilter_gentech_driver.c:780 #9 __virNWFilterInstantiateFilter (driver=driver@entry=0x7f67ac01e9f0, vmuuid=vmuuid@entry=0x7f67ac27e1c8 "!\236'\261\366\071L\277\200\241\236\324[6W\362p\350'\254g\177", teardownOld=teardownOld@entry=true, ifname=0x7f6790007a90 "vnet0", ifindex=747, linkdev=linkdev@entry=0x0, nettype=VIR_DOMAIN_NET_TYPE_NETWORK, macaddr=macaddr@entry=0x7f67ac222fe4, filtername=0x7f67ac27e720 "clean-traffic", filterparams=0x7f67ac27e740, useNewFilter=useNewFilter@entry=INSTANTIATE_ALWAYS, forceWithPendingReq=forceWithPendingReq@entry=false, foundNewFilter=foundNewFilter@entry=0x7f67b83bfac7) at nwfilter/nwfilter_gentech_driver.c:894 #10 0x00007f67b39ff131 in _virNWFilterInstantiateFilter (driver=0x7f67ac01e9f0, vmuuid=0x7f67ac27e1c8 "!\236'\261\366\071L\277\200\241\236\324[6W\362p\350'\254g\177", net=0x7f67ac222fe0, teardownOld=teardownOld@entry=true, useNewFilter=useNewFilter@entry=INSTANTIATE_ALWAYS, foundNewFilter=foundNewFilter@entry=0x7f67b83bfac7) at nwfilter/nwfilter_gentech_driver.c:949 #11 0x00007f67b39ff29b in virNWFilterInstantiateFilter (driver=<optimized out>, vmuuid=<optimized out>, net=<optimized out>) at nwfilter/nwfilter_gentech_driver.c:1020 #12 0x00007f67b2389c04 in qemuNetworkIfaceConnect (def=def@entry=0x7f67ac27e1c0, conn=conn@entry=0x7f678800a930, driver=driver@entry=0x7f67ac1b20f0, net=net@entry=0x7f67ac222fe0, qemuCaps=qemuCaps@entry=0x7f67900034b0, tapfd=0x7f6790008a70, tapfdSize=tapfdSize@entry=0x7f67b83bfcb0) at qemu/qemu_command.c:400 #13 0x00007f67b23970de in qemuBuildInterfaceCommandLine (vmop=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, bootindex=0, vlan=-1, qemuCaps=0x7f67900034b0, net=0x7f67ac222fe0, def=0x7f67ac27e1c0, conn=0x7f678800a930, driver=0x7f67ac1b20f0, cmd=0x7f6790010390) at qemu/qemu_command.c:7386 #14 qemuBuildCommandLine (conn=conn@entry=0x7f678800a930, driver=driver@entry=0x7f67ac1b20f0, def=<optimized out>, monitor_chr=<optimized out>, monitor_json=<optimized out>, qemuCaps=0x7f67900034b0, migrateFrom=migrateFrom@entry=0x0, migrateFd=migrateFd@entry=-1, snapshot=snapshot@entry=0x0, vmop=vmop@entry=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, callbacks=0x7f67b2655b50 <buildCommandLineCallbacks>) at qemu/qemu_command.c:8473 #15 0x00007f67b23c0464 in qemuProcessStart (conn=conn@entry=0x7f678800a930, driver=driver@entry=0x7f67ac1b20f0, vm=vm@entry=0x7f67ac224830, migrateFrom=migrateFrom@entry=0x0, stdin_fd=stdin_fd@entry=-1, stdin_path=stdin_path@entry=0x0, snapshot=snapshot@entry=0x0, vmop=vmop@entry=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, flags=flags@entry=1) at qemu/qemu_process.c:3828 #16 0x00007f67b240c1b3 in qemuDomainObjStart (conn=0x7f678800a930, driver=driver@entry=0x7f67ac1b20f0, vm=vm@entry=0x7f67ac224830, flags=flags@entry=0) at qemu/qemu_driver.c:6136 #17 0x00007f67b240c762 in qemuDomainCreateWithFlags (dom=0x7f6790004260, flags=0) at qemu/qemu_driver.c:6190 #18 0x00007f67c8d332a7 in virDomainCreate (domain=domain@entry=0x7f6790004260) at libvirt.c:9511 #19 0x00007f67c973d7b7 in remoteDispatchDomainCreate (server=<optimized out>, msg=<optimized out>, args=<optimized out>, rerr=0x7f67b83c0c80, client=0x7f67cae6de50) at remote_dispatch.h:2888 #20 remoteDispatchDomainCreateHelper (server=<optimized out>, client=0x7f67cae6de50, msg=<optimized out>, rerr=0x7f67b83c0c80, args=<optimized out>, ret=<optimized out>) at remote_dispatch.h:2866 #21 0x00007f67c8d8b09a in virNetServerProgramDispatchCall (msg=0x7f67cae6dd80, client=0x7f67cae6de50, server=0x7f67cae5ed00, prog=0x7f67cae6ab40) at rpc/virnetserverprogram.c:435 ---Type <return> to continue, or q <return> to quit--- #22 virNetServerProgramDispatch (prog=0x7f67cae6ab40, server=server@entry=0x7f67cae5ed00, client=0x7f67cae6de50, msg=0x7f67cae6dd80) at rpc/virnetserverprogram.c:305 #23 0x00007f67c8d85c08 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f67cae5ed00) at rpc/virnetserver.c:166 #24 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f67cae5ed00) at rpc/virnetserver.c:187 #25 0x00007f67c8caa4d5 in virThreadPoolWorker (opaque=opaque@entry=0x7f67cadf1860) at util/virthreadpool.c:144 #26 0x00007f67c8ca9e6e in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:194 #27 0x00007f67c6584df3 in start_thread (arg=0x7f67b83c1700) at pthread_create.c:308 #28 0x00007f67c5eab39d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 (gdb)