RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1599973 - libvirtd crashed during stop if there is a guest set filterref in guest vNIC
Summary: libvirtd crashed during stop if there is a guest set filterref in guest vNIC
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.6
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: John Ferlan
QA Contact: Luyao Huang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-11 04:47 UTC by Luyao Huang
Modified: 2018-10-30 09:58 UTC (History)
4 users (show)

Fixed In Version: libvirt-4.5.0-5.el7
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2018-10-30 09:57:31 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:3113 0 None None None 2018-10-30 09:58:30 UTC

Description Luyao Huang 2018-07-11 04:47:42 UTC 
Description of problem:
libvirtd crashed during stop if there is a guest set filterref in guest vNIC

Version-Release number of selected component (if applicable):
libvirt-4.5.0-2.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. prepare a guest which set filterref in guest vNIC:

    <interface type='network'>
      <mac address='52:54:00:2c:76:c5'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vif200'/>
      <model type='e1000e'/>
      <filterref filter='clean-traffic'>
        <parameter name='CTRL_IP_LEARNING' value='dhcp'/>
      </filterref>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>

2. start guest

3. wait guest get ip and check nwfilter.leases

# cat /run/libvirt/network/nwfilter.leases 
1531218124 8a39e5ec-1e91-4480-80f3-c106ce1eeeec-52:54:00:2c:76:c5 192.168.122.238 192.168.122.1
1531218126 8a39e5ec-1e91-4480-80f3-c106ce1eeeec-52:54:00:2c:76:c5 192.168.122.238 192.168.122.1
1531218127 8a39e5ec-1e91-4480-80f3-c106ce1eeeec-52:54:00:2c:76:c5 192.168.122.238 192.168.122.1
1531218129 8a39e5ec-1e91-4480-80f3-c106ce1eeeec-52:54:00:2c:76:c5 192.168.122.238 192.168.122.1
1531218136 8a39e5ec-1e91-4480-80f3-c106ce1eeeec-52:54:00:2c:76:c5 192.168.122.238 192.168.122.1
1531218142 8a39e5ec-1e91-4480-80f3-c106ce1eeeec-52:54:00:2c:76:c5 192.168.122.238 192.168.122.1
1531218145 8a39e5ec-1e91-4480-80f3-c106ce1eeeec-52:54:00:2c:76:c5 192.168.122.238 192.168.122.1

4. restart libvirtd

5. gdb attach libvirtd and use another terminal to restart/stop libvirtd

Actual results:

libvirtd received signal SIGSEGV in step 5

Expected results:

There is no crash in step 5

Additional info:

backtrace:

Program received signal SIGSEGV, Segmentation fault.
0x00007f6c85931cbc in virNWFilterSnoopRemAllReqIter (payload=0x7f6c7c102cb0, name=<optimized out>, data=<optimized out>) at nwfilter/nwfilter_dhcpsnoop.c:2040
2040        if (req->binding->portdevname) {
(gdb) bt
#0  0x00007f6c85931cbc in virNWFilterSnoopRemAllReqIter (payload=0x7f6c7c102cb0, name=<optimized out>, data=<optimized out>) at nwfilter/nwfilter_dhcpsnoop.c:2040
#1  0x00007f6cb18bd214 in virHashRemoveSet (table=0x7f6c7c0f15d0, iter=iter@entry=0x7f6c85931ca0 <virNWFilterSnoopRemAllReqIter>, data=data@entry=0x0) at util/virhash.c:624
#2  0x00007f6c85931c85 in virNWFilterSnoopEndThreads () at nwfilter/nwfilter_dhcpsnoop.c:2068
#3  0x00007f6c859345f9 in virNWFilterDHCPSnoopShutdown () at nwfilter/nwfilter_dhcpsnoop.c:2183
#4  0x00007f6c8592eaa2 in nwfilterStateCleanup () at nwfilter/nwfilter_driver.c:340
#5  0x00007f6cb1a8c04b in virStateCleanup () at libvirt.c:699
#6  0x0000558d90e919bc in main (argc=<optimized out>, argv=<optimized out>) at remote/remote_daemon.c:1478
(gdb) p req->binding
$1 = (virNWFilterBindingDefPtr) 0x0


And another crash like this:

Program received signal SIGSEGV, Segmentation fault.
0x00007fffcb528aaf in virNWFilterSnoopReqLeaseDel (req=0x7fffc00ff180, ipaddr=0x7fffc00fff30, update_leasefile=update_leasefile@entry=false, instantiate=instantiate@entry=false)
    at nwfilter/nwfilter_dhcpsnoop.c:870
870     ipAddrLeft = virNWFilterIPAddrMapDelIPAddr(req->binding->portdevname, ipstr);
(gdb) bt
#0  0x00007fffcb528aaf in virNWFilterSnoopReqLeaseDel (req=0x7fffc00ff180, ipaddr=0x7fffc00fff30, update_leasefile=update_leasefile@entry=false, instantiate=instantiate@entry=false)
    at nwfilter/nwfilter_dhcpsnoop.c:870
#1  0x00007fffcb528c89 in virNWFilterSnoopReqFree (req=0x7fffc00ff180) at nwfilter/nwfilter_dhcpsnoop.c:636
#2  virNWFilterSnoopReqRelease (req0=0x7fffc00ff180, name=<optimized out>) at nwfilter/nwfilter_dhcpsnoop.c:685
#3  0x00007ffff74b3025 in virHashRemoveEntry (table=0x7fffc00f0e60, name=name@entry=0x7fffc00ff19c) at util/virhash.c:539
#4  0x00007fffcb528d72 in virNWFilterSnoopReqPut (req=0x7fffc00ff180) at nwfilter/nwfilter_dhcpsnoop.c:733
#5  0x00007fffcb528372 in virNWFilterSnoopLeaseFileLoad () at nwfilter/nwfilter_dhcpsnoop.c:2000
#6  0x00007fffcb52a461 in virNWFilterDHCPSnoopInit () at nwfilter/nwfilter_dhcpsnoop.c:2096
#7  0x00007fffcb524c0b in nwfilterStateInitialize (privileged=<optimized out>, callback=<optimized out>, opaque=<optimized out>) at nwfilter/nwfilter_driver.c:213
#8  0x00007ffff7681eef in virStateInitialize (privileged=true, callback=callback@entry=0x5555555775c0 <daemonInhibitCallback>, opaque=opaque@entry=0x5555557f4440) at libvirt.c:662
#9  0x000055555557761b in daemonRunStateInit (opaque=0x5555557f4440) at remote/remote_daemon.c:803
#10 0x00007ffff7505512 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#11 0x00007ffff48f2dd5 in start_thread () from /lib64/libpthread.so.0
#12 0x00007ffff461bead in clone () from /lib64/libc.so.6
Comment 1 John Ferlan 2018-07-23 15:31:32 UTC 
A patch to resolve is posted upstream:

https://www.redhat.com/archives/libvir-list/2018-July/msg01523.html
Comment 2 John Ferlan 2018-07-26 13:38:10 UTC 
The patch is now pushed:

commit 5229494b01acf97dbc6f3028e9718667e9e1426a
Author: John Ferlan <jferlan>
Date:   Fri Jul 20 17:44:08 2018 -0400

    nwfilter: Resolve SEGV for NWFilter Snoop processing
    
...
    
    Commit id fca9afa08 changed the @req->ifname to use
    @req->binding->portdevname fillingin the @req->binding
    in a similar way that @req->ifname would have been
    filled in during virNWFilterDHCPSnoopReq processing.
    
    However, in doing so it did not take into account some
    code paths where the @req->binding should be checked
    instead of @req->binding->portdevname. These checks
    led to SEGVs in some cases during libvirtd reload
    processing in virNWFilterSnoopRemAllReqIter (for
    stop during nwfilterStateCleanup processing) and
    virNWFilterSnoopReqLeaseDel (for start during
    nwfilterStateInitialize processing).
    
    In particular, when reading the nwfilter.leases file
    a new @req is created, but the @req->binding is not
    filled in. That's left to virNWFilterDHCPSnoopReq
    processing which checks if the @req already exists
    in the @virNWFilterSnoopState.snoopReqs hash table
    after adding a virNWFilterSnoopState.ifnameToKey
    entry for the @req->binding->portdevname by a
    @ref->ikey value.
    
    NB: virNWFilterSnoopIPLeaseInstallRule and
        virNWFilterDHCPSnoopThread do not need the
        req->binding check since they can only be called
        after the filter->binding is created/assigned.
    
    Signed-off-by: John Ferlan <jferlan>
    ACKed-by: Michal Privoznik <mprivozn>

$ git describe 5229494b01acf97dbc6f3028e9718667e9e1426a
v4.5.0-297-g5229494b01
$
Comment 5 Luyao Huang 2018-08-16 03:26:04 UTC 
Verify this bug on libvirt-4.5.0-6.el7.x86_64:

1. prepare a running guest with vNIC xml like this:

    <interface type='network'>
      <mac address='52:54:00:2c:76:c5'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='rtl8139'/>
      <filterref filter='clean-traffic'>
        <parameter name='CTRL_IP_LEARNING' value='dhcp'/>
      </filterref>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>


2. check nwfilter.leases file:

# cat /run/libvirt/network/nwfilter.leases 
1534387047 4d73262f-35a7-437e-96ad-e5246aac79ee-52:54:00:2c:76:c5 192.168.122.238 192.168.122.1
1534387048 4d73262f-35a7-437e-96ad-e5246aac79ee-52:54:00:2c:76:c5 192.168.122.238 192.168.122.1
1534388701 4d73262f-35a7-437e-96ad-e5246aac79ee-52:54:00:2c:76:c5 192.168.122.238 192.168.122.1
1534389984 4d73262f-35a7-437e-96ad-e5246aac79ee-52:54:00:2c:76:c5 192.168.122.238 192.168.122.1
1534391426 4d73262f-35a7-437e-96ad-e5246aac79ee-52:54:00:2c:76:c5 192.168.122.238 192.168.122.1
1534392838 4d73262f-35a7-437e-96ad-e5246aac79ee-52:54:00:2c:76:c5 192.168.122.238 192.168.122.1


3. use gdb attach running libvirtd and open another terminal to stop libvirtd, cannot find crash report

4. start libvirtd with gdb and check there is no crash report
Comment 7 errata-xmlrpc 2018-10-30 09:57:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113
Note You need to log in before you can comment on or make changes to this bug.