Bug 815270

Summary: [Regression]Libvirtd will die if start a guest with macvtap nic.
Product: Red Hat Enterprise Linux 6 Reporter: Geyang Kong <gkong>
Component: libvirtAssignee: Laine Stump <laine>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.3CC: acathrow, ajia, dallan, dyasny, dyuan, jwu, mjenner, mzhan, rwu, syeghiay, whuang, ydu, yupzhang, zpeng
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-0.9.10-16.el6 Doc Type: Bug Fix
Doc Text:
(This was a bug in new functionality, so it was never in a released version of RHEL. No tech note is needed)
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-20 06:57:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
libvirtd's log file none

Description Geyang Kong 2012-04-23 09:11:59 UTC
Created attachment 579455 [details]
libvirtd's log file

Description of problem:
  Libvirtd will die if start a guest with macvtap nic.

Version-Release number of selected component (if applicable):
libvirt-0.9.10-13.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Launch virt-manager
2. Start create a new virtual machine wizard.
3. Following the wizard to the step 4, then check the "Customize configuration before install"
4. Click "Finish", and change its NIC to Host device eth0:macvtap
5. Click "Begin Installation".

Actual results:
1. After step 5, libvirtd service stoped. Following is the error.
-------------------------------------------------------------------------------
Unable to complete install: 'Cannot recv data: Connection reset by peer'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 44, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 1910, in do_install
    guest.start_install(False, meter=meter)
  File "/usr/lib/python2.6/site-packages/virtinst/Guest.py", line 1223, in start_install
    noboot)
  File "/usr/lib/python2.6/site-packages/virtinst/Guest.py", line 1291, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2400, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: Cannot recv data: Connection reset by peer
-------------------------------------------------------------------------------

Expected results:
1. Guest can be installed successfully.

Additional info:
1. This issue cannot be reproduced by libvirt-0.9.10-12.el6.x86_64
2. I can only open the log file from terminal, if I open it by gedit, its content will be displayed by garbage characters.

Comment 2 Alex Jia 2012-04-23 10:13:20 UTC
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f6c8d0d7700 (LWP 20218)]
0x00000037f84686fa in virNetDevMacVLanVPortProfileRegisterCallback (ifname=0x7f6c8d0d5f00 "macvtap1", macaddress=0x7f6c7400b304 "RT", linkdev=0x7f6c7400b620 "eth0", 
    vmuuid=0x7f6c7400aeb8 "͐\206M\365_3\234\061\201֯\004_\375\200 \022", virtPortProfile=0x0, vmOp=VIR_NETDEV_VPORT_PROFILE_OP_CREATE) at /usr/include/bits/string3.h:52
52	  return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest));
(gdb) bt
#0  0x00000037f84686fa in virNetDevMacVLanVPortProfileRegisterCallback (ifname=0x7f6c8d0d5f00 "macvtap1", macaddress=0x7f6c7400b304 "RT", linkdev=0x7f6c7400b620 "eth0", 
    vmuuid=0x7f6c7400aeb8 "͐\206M\365_3\234\061\201֯\004_\375\200 \022", virtPortProfile=0x0, vmOp=VIR_NETDEV_VPORT_PROFILE_OP_CREATE) at /usr/include/bits/string3.h:52
#1  0x00000037f846a1e0 in virNetDevMacVLanCreateWithVPortProfile (tgifname=<value optimized out>, macaddress=0x7f6c7400b304 "RT", linkdev=0x7f6c7400b620 "eth0", mode=<value optimized out>, withTap=true, 
    vnet_hdr=0, vmuuid=0x7f6c7400aeb8 "͐\206M\365_3\234\061\201֯\004_\375\200 \022", virtPortProfile=0x0, res_ifname=0x7f6c8d0d5fa8, vmOp=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, 
    stateDir=0x7f6c80020110 "/var/run/libvirt/qemu", bandwidth=0x0) at util/virnetdevmacvlan.c:969
#2  0x000000000046ddd2 in qemuPhysIfaceConnect (def=0x7f6c7400aeb0, driver=<value optimized out>, net=0x7f6c7400b300, qemuCaps=<value optimized out>, vmop=VIR_NETDEV_VPORT_PROFILE_OP_CREATE)
    at qemu/qemu_command.c:152
#3  0x00000000004795ea in qemuBuildCommandLine (conn=0x7f6c7c000ae0, driver=0x7f6c800a2800, def=0x7f6c7400aeb0, monitor_chr=0x7f6c8d0d62b8, monitor_json=24, qemuCaps=0x7f6c74001480, migrateFrom=0x0, 
    migrateFd=-1, snapshot=0x0, vmop=VIR_NETDEV_VPORT_PROFILE_OP_CREATE) at qemu/qemu_command.c:4892
#4  0x000000000048fddb in qemuProcessStart (conn=0x7f6c7c000ae0, driver=0x7f6c800a2800, vm=0x7f6c74008800, migrateFrom=0x0, cold_boot=<value optimized out>, start_paused=false, autodestroy=false, 
    stdin_fd=-1, stdin_path=0x0, snapshot=0x0, vmop=VIR_NETDEV_VPORT_PROFILE_OP_CREATE) at qemu/qemu_process.c:3427
#5  0x000000000045f794 in qemudDomainCreate (conn=0x7f6c7c000ae0, xml=<value optimized out>, flags=<value optimized out>) at qemu/qemu_driver.c:1393
#6  0x00000037f84cac8f in virDomainCreateXML (conn=0x7f6c7c000ae0, 
    xmlDesc=0x7f6c74001570 "<domain type='kvm'>\n  <name>try</name>\n  <uuid>cd90864d-f55f-339c-3181-d6af045ffd80</uuid>\n  <memory>1048576</memory>\n  <currentMemory>1048576</currentMemory>\n  <vcpu>1</vcpu>\n  <os>\n    <type arch='x"..., flags=0) at libvirt.c:1976
#7  0x0000000000439691 in remoteDispatchDomainCreateXML (server=<value optimized out>, client=0xe72040, msg=<value optimized out>, rerr=0x7f6c8d0d6bc0, args=0x7f6c74003480, ret=0x7f6c74003400)
    at remote_dispatch.h:958
#8  remoteDispatchDomainCreateXMLHelper (server=<value optimized out>, client=0xe72040, msg=<value optimized out>, rerr=0x7f6c8d0d6bc0, args=0x7f6c74003480, ret=0x7f6c74003400) at remote_dispatch.h:938
#9  0x00000037f85152c5 in virNetServerProgramDispatchCall (prog=0xe72ec0, server=0xe67b30, client=0xe72040, msg=0xeb3b30) at rpc/virnetserverprogram.c:416
#10 virNetServerProgramDispatch (prog=0xe72ec0, server=0xe67b30, client=0xe72040, msg=0xeb3b30) at rpc/virnetserverprogram.c:289
#11 0x00000037f85177b1 in virNetServerHandleJob (jobOpaque=<value optimized out>, opaque=0xe67b30) at rpc/virnetserver.c:164
#12 0x00000037f8458d8c in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:144
#13 0x00000037f84586a9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#14 0x00000038baa077f1 in start_thread (arg=0x7f6c8d0d7700) at pthread_create.c:301
#15 0x00000033f68e570d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Comment 3 Alex Jia 2012-04-25 09:43:56 UTC
Hi Laine,
If I use "Host device eth0: macvtap" virtual network interface in virt-manager, I must set virtual port profile data, right? if so, we need to judge whether these data aren't empty/NULL, for example:

@@ -769,7 +769,7 @@ virNetDevMacVLanVPortProfileRegisterCallback(const char *ifname,
             goto memory_error;
         if ((calld->cr_ifname = strdup(ifname)) == NULL)
             goto memory_error;
-        if (VIR_ALLOC(calld->virtPortProfile) < 0)
+        if (!calld->virtPortProfile || VIR_ALLOC(calld->virtPortProfile) < 0)
             goto memory_error;
         memcpy(calld->virtPortProfile, virtPortProfile, sizeof(*virtPortProfile));
         if (VIR_ALLOC_N(calld->macaddress, VIR_MAC_BUFLEN) < 0)


Although libvirtd is alive, it's uncompleted correct, if calld->virtPortProfile is NULL then goto memory_error.

Comment 4 Laine Stump 2012-04-25 14:27:43 UTC
(In reply to comment #3)
> Hi Laine,
> If I use "Host device eth0: macvtap" virtual network interface in virt-manager,
> I must set virtual port profile data, right? 

No, not necessarily. virtual port profile data is only needed for 802.1Qbg or 802.1Qbh types of interfaces. It isn't needed for straight 'bridge' mode, for example, and it is completely acceptable (and very common) for an interface to be one of the direct types, and have no virtPortProfile (in which case it will be NULL).

> if so, we need to judge whether
> these data aren't empty/NULL, for example:
> 
> @@ -769,7 +769,7 @@ virNetDevMacVLanVPortProfileRegisterCallback(const char
> *ifname,
>              goto memory_error;
>          if ((calld->cr_ifname = strdup(ifname)) == NULL)
>              goto memory_error;
> -        if (VIR_ALLOC(calld->virtPortProfile) < 0)
> +        if (!calld->virtPortProfile || VIR_ALLOC(calld->virtPortProfile) < 0)
>              goto memory_error;

This change doesn't make sense. calld was just allocated, and VIR_ALLOC always nulls out the memory it allocates, so calld->virtPortProfile will always be NULL.

Likewise, Michal posted a patch yesterday to add in a check for non-NULL virtPortProfile (the one passed as an arg to the function), but that is also pointless here, because it was already checked for non-NULL in the main if() at the top of the function:

   if (virtPortProfile && virNetlinkEventServiceIsRunning()) {

Comment 5 Laine Stump 2012-04-26 07:04:41 UTC
The real source of this problem was that the prototype for virNetDevMacVLanVPortProfileRegisterCallback specified ATTRIBUTE_NONNULL() for virtPortProfile, which resulted in gcc optimizing out the (already existing) check for non-NULL virtPortProfile. There is a (very old) bug filed against gcc on this exact topic:

     http://gcc.gnu.org/bugzilla/show_bug.cgi?id=17308

A fix has been pushed upstream:

commit f78024b9f594605053184ee41a5cba88ce3a2c8b
Author: Laine Stump <laine>
Date:   Wed Apr 25 15:49:44 2012 -0400

    util: fix crash when starting macvtap interfaces
    
    This patch resolves https://bugzilla.redhat.com/show_bug.cgi?id=815270
    
    The function virNetDevMacVLanVPortProfileRegisterCallback() takes an
    arg "virtPortProfile", and was checking it for non-NULL before using
    it. However, the prototype for
    virNetDevMacVLanPortProfileRegisterCallback had marked that arg with
    ATTRIBUTE_NONNULL(). Contrary to what one may think,
    ATTRIBUTE_NONNULL() does not provide any guarantee that an arg marked
    as such really is always non-null; the only effect to the code
    generated by gcc, is that gcc *assumes* it is non-NULL; this results
    in, for example, the check for a non-NULL value being optimized out.
    
    (Unfortunately, this code removal only occurs when optimization is
    enabled, and I am in the habit of doing local builds with optimization
    off to ease debugging, so the bug didn't show up in my earlier local
    testing).
    
    In general, virPortProfile might always be NULL, so it shouldn't be
    marked as ATTRIBUTE_NONNULL. One other function prototype made this
    same error, so this patch fixes it as well.

Comment 10 yanbing du 2012-05-03 10:34:31 UTC
Following the bug description steps, verify the bug with libvirt-0.9.10-16.el6.x86_64.

Comment 11 Laine Stump 2012-05-08 18:09:15 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
(This was a bug in new functionality, so it was never in a released version of RHEL. No tech note is needed)

Comment 13 errata-xmlrpc 2012-06-20 06:57:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0748.html