RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1529338 - VM network is blocked when using "CTRL_IP_LEARNING" and "DHCPSERVER" filters
Summary: VM network is blocked when using "CTRL_IP_LEARNING" and "DHCPSERVER" filters
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.5
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: yalzhang@redhat.com
URL:
Whiteboard:
: 1529339 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-27 17:11 UTC by Gal Ben Haim
Modified: 2023-09-14 04:14 UTC (History)
24 users (show)

Fixed In Version: libvirt-3.9.0-14.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-10 11:04:21 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
var log messages with the filters (530.22 KB, text/plain)
2017-12-27 17:11 UTC, Gal Ben Haim 		no flags Details
var log messages without the filter (608.11 KB, text/plain)
2017-12-27 17:15 UTC, Gal Ben Haim 		no flags Details
dom xml with filters (11.66 KB, text/plain)
2017-12-27 17:16 UTC, Gal Ben Haim 		no flags Details
dom xml without filter (11.55 KB, text/plain)
2017-12-27 17:18 UTC, Gal Ben Haim 		no flags Details
requested logs (7.57 KB, application/x-gzip)
2018-02-06 16:37 UTC, Gal Ben Haim 		no flags Details
capture while running dhcp handshake (1.22 KB, application/x-gzip)
2018-02-13 16:06 UTC, Gal Ben Haim 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:0704 0 None None None 2018-04-10 11:05:38 UTC

Description Gal Ben Haim 2017-12-27 17:11:11 UTC 
Created attachment 1372839 [details]
var log messages with the filters

Description of problem:

VM network is blocked when using "CTRL_IP_LEARNING" and "DHCPSERVER" filters.
After removing the filters, running ifup/ifdown in the VM, the network wasn't blocked anymore, and the VM could be reached from the host.

Version-Release number of selected component (if applicable):

libvirt version - 3.9.0-6.el7.x86_64
Host version - Red Hat Enterprise Linux Server release 7.5 Beta (Maipo)
VM version - CirrOS 0.3.5


Steps to Reproduce:

Create a VM with network filters "CTRL_IP_LEARNING" and "DHCPSERVER" and try to reach it using the network (for example ping).

Actual results:

The VM isn't reachable through network

Expected results:

The VM is reachable through network

Additional info:
Comment 1 Gal Ben Haim 2017-12-27 17:14:20 UTC 
*** Bug 1529339 has been marked as a duplicate of this bug. ***
Comment 2 Gal Ben Haim 2017-12-27 17:15:01 UTC 
Created attachment 1372840 [details]
var log messages without the filter
Comment 3 Gal Ben Haim 2017-12-27 17:16:56 UTC 
Created attachment 1372841 [details]
dom xml with filters
Comment 4 Gal Ben Haim 2017-12-27 17:18:06 UTC 
Created attachment 1372842 [details]
dom xml without filter
Comment 5 Yaniv Kaul 2017-12-27 17:59:19 UTC 
I assume this works well in 7.4?
This should be opened on RHEL?
Comment 6 Gal Ben Haim 2017-12-28 09:00:03 UTC 
Yes, it works in RHEL 7.4, but the libvirt version is different (3.2.0-14.el7_4.5.x86_64). 

I think that this is the correct place for the bug.
Comment 10 Gal Ben Haim 2018-01-15 14:32:40 UTC 
Any update?
Comment 11 Laine Stump 2018-01-17 16:57:26 UTC 
Can provide more details about what you mean by "the VM network is blocked". You say that it doesn't respond to incoming pings from the host, but can the guest do an outgoing ping? Does the guest's interface get an IP address from the DHCP server?

Also can you connect gdb to the libvirtd process while the guest is running and issue the command "thread apply all bt" and attach the result? (the libvirt-debuginfo package must be installed beforehand for the results to be of any use. you'll need to hit enter several times to be sure the backtrace of all threads is output).

Additionally, can you attach the output of "iptables -S" while the guest is running both with and without the filter? (the same output on a host running the older libvirt that doesn't have this problem would also be useful).
Comment 12 Gal Ben Haim 2018-01-18 12:49:12 UTC 
The guest can't do an outgoing ping.
The guest's interface got an IP address from the DHCP server.

Let me reproduce and then I'll provide the additional info that you have asked.
Comment 13 Laine Stump 2018-01-22 14:58:36 UTC 
I can't reproduce this on my RHEL7 system (with libvirt-3.9.0-8), so it's something beyond just having the clean-traffic filter in place that's causing your problem.

Running through all of this on my system reminded me that there is more than just the output of "iptables -S" that has useful info. Please gather the output of these 4 commands on both a working and non-working system:


 iptables -S
 iptables -t nat -S
 ebtables -L
 ebtables -t nat -L

Those can all be placed in a single attachment for each test run. In addition, if you can add this line to libvirtd.conf, then attach everything that's sent to the libvirtd log while a guest is being started (and for the first couple minutes after it's started):


  log_filters="1:util.firewall 1:nwfilter"
Comment 14 spower 2018-02-05 13:23:28 UTC 
Gal, do you have an update on the reproduction of this issue, it is critical for RHV testing with RHEL 7.5
Comment 16 Laine Stump 2018-02-05 14:27:21 UTC 
A note from the libvirt side on what I'm doing while waiting for the info: due to this bug, as well as a pending patch series upstream, I've dusted off the libvirt-tck tests, which have a fairly extensive test suite for libvirt's nwfilter, including concurrency tests that attempt to break the locking (which is the first suspect any time there is a nwfilter-related bug).

Unfortunately this test can't be easily installed on RHEL because many of its package dependencies aren't available even in EPEL. For that reason I'm running the test on Fedora; the tests fail when run on Fedora 27, but the same tests with the same version of libvirt succeed on Fedora 26. At least, though, I have a proper "control" to start from - libvirt-3.2.1 on F26 succeeds - and am working forward to see if it breaks with a newer libvirt.
Comment 19 Gal Ben Haim 2018-02-06 16:37:48 UTC 
Created attachment 1392201 [details]
requested logs
Comment 20 Laine Stump 2018-02-12 02:16:22 UTC 
The output of ebtables -t nat -L shows that there are two guests running on the host with active nwfilters. The one using vnet0 has a simple

  <filterref filter='clean-traffic'/>

with no CTRL_IP_LEARNING, and the one using vnet1 has CTRL_IP_LEARNING in its clean-traffic rule, but has not yet learned its IP address. You can tell this by looking at the chain called "libvirt-I-vnet1" in the ebtables -t nat -L output - it has one rule that allows packets to port 67 (dhcp) and just below that is "-j DROP", so the only traffic allowed is DHCP.

As soon as the dhcp response packet has been seen, what *should* happen is that different rules are loaded so that other traffic is also allowed.

In the morning I will figure out the best set of libvirtd.conf additions to collect maximum data. In the meantime, can you try downgrading the libvirt packages on the host to RHE7.4's libvirt (3.2.0-whatever) and see if it continues to fail or if it now succeeds. There are two packages involved with the DHCP snooping necessary to learn the guest's IP addres - libvirt and libpcap. I wan't to make sure that libvirt is definitely the source of the problem.
Comment 21 Dan Kenigsberg 2018-02-13 15:18:55 UTC 
(In reply to Laine Stump from comment #20)
> In the meantime, can you try downgrading the libvirt
> packages on the host to RHE7.4's libvirt (3.2.0-whatever) and see if it
> continues to fail or if it now succeeds. There are two packages involved
> with the DHCP snooping necessary to learn the guest's IP addres - libvirt
> and libpcap. I wan't to make sure that libvirt is definitely the source of
> the problem.

Unfortunately, I don't have any easy way to do so. Our CI moved to rhel-7.5-beta, and we explicitly require the newer libvirt for other bugfixes.
Comment 22 Gal Ben Haim 2018-02-13 16:06:43 UTC 
Created attachment 1395460 [details]
capture while running dhcp handshake
Comment 23 Gal Ben Haim 2018-02-13 16:12:20 UTC 
I tried to retrigger the DHCP handshake from the VM, while capturing the packets from its vnic. I don't see any change to ebtables rules.
You can see the details in "capture while running dhcp handshake".
Comment 24 Gal Ben Haim 2018-02-13 16:49:22 UTC 
(In reply to Laine Stump from comment #20)
> The output of ebtables -t nat -L shows that there are two guests running on
> the host with active nwfilters. The one using vnet0 has a simple
> 
>   <filterref filter='clean-traffic'/>
> 
> with no CTRL_IP_LEARNING, and the one using vnet1 has CTRL_IP_LEARNING in
> its clean-traffic rule, but has not yet learned its IP address. You can tell
> this by looking at the chain called "libvirt-I-vnet1" in the ebtables -t nat
> -L output - it has one rule that allows packets to port 67 (dhcp) and just
> below that is "-j DROP", so the only traffic allowed is DHCP.
> 
> As soon as the dhcp response packet has been seen, what *should* happen is
> that different rules are loaded so that other traffic is also allowed.
> 
> In the morning I will figure out the best set of libvirtd.conf additions to
> collect maximum data. In the meantime, can you try downgrading the libvirt
> packages on the host to RHE7.4's libvirt (3.2.0-whatever) and see if it
> continues to fail or if it now succeeds. There are two packages involved
> with the DHCP snooping necessary to learn the guest's IP addres - libvirt
> and libpcap. I wan't to make sure that libvirt is definitely the source of
> the problem.

I downgraded libvirt, but now libvirtd doesn't start:

Feb 13 11:34:01 lago-rhv-suite-4-2-host-0 libvirtd: /usr/sbin/libvirtd: error while loading shared libraries: libvirt-admin.so.0: cannot open shared object file: No such file or directory
Comment 25 Meni Yakove 2018-02-14 14:48:08 UTC 
I have reproduced this one RHEL 7.4 and 7.5.

ebtables -t nat -L
Bridge table: nat

Bridge chain: PREROUTING, entries: 2, policy: ACCEPT
-j PREROUTING_direct
-i vnet0 -j libvirt-I-vnet0

Bridge chain: OUTPUT, entries: 1, policy: ACCEPT
-j OUTPUT_direct

Bridge chain: POSTROUTING, entries: 2, policy: ACCEPT
-j POSTROUTING_direct
-o vnet0 -j libvirt-O-vnet0

Bridge chain: PREROUTING_direct, entries: 0, policy: RETURN

Bridge chain: POSTROUTING_direct, entries: 0, policy: RETURN

Bridge chain: OUTPUT_direct, entries: 0, policy: RETURN

Bridge chain: libvirt-I-vnet0, entries: 2, policy: ACCEPT
-p IPv4 -s 0:1a:4a:16:20:16 --ip-proto udp --ip-sport 68 --ip-dport 67 -j ACCEPT 
-j DROP 

Bridge chain: libvirt-O-vnet0, entries: 3, policy: ACCEPT
-p IPv4 -d 0:1a:4a:16:20:16 --ip-src 10.35.70.162 --ip-proto udp --ip-sport 67 --ip-dport 68 -j ACCEPT 
-p IPv4 -d Broadcast --ip-src 10.35.70.162 --ip-proto udp --ip-sport 67 --ip-dport 68 -j ACCEPT 
-j DROP


tcpdump -i vnet0 -n port 67 and port 68 -v
tcpdump: listening on vnet0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:45:14.682235 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:1a:4a:16:20:16, length 300, xid 0x6c900b11, Flags [none]
	  Client-Ethernet-Address 00:1a:4a:16:20:16
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Requested-IP Option 50, length 4: 10.35.128.113
	    Parameter-Request Option 55, length 13: 
	      Subnet-Mask, BR, Time-Zone, Classless-Static-Route
	      Domain-Name, Domain-Name-Server, Hostname, YD
	      YS, NTP, MTU, Option 119
	      Default-Gateway


tcpdump -i ovirtmgmt -n port 67 and port 68 -v
tcpdump: listening on ovirtmgmt, link-type EN10MB (Ethernet), capture size 262144 bytes
16:45:34.141332 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:1a:4a:16:20:16, length 300, xid 0xdd07ee25, Flags [none]
	  Client-Ethernet-Address 00:1a:4a:16:20:16
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Requested-IP Option 50, length 4: 10.35.128.113
	    Parameter-Request Option 55, length 13: 
	      Subnet-Mask, BR, Time-Zone, Classless-Static-Route
	      Domain-Name, Domain-Name-Server, Hostname, YD
	      YS, NTP, MTU, Option 119
	      Default-Gateway
16:45:34.170040 IP (tos 0x0, ttl 64, id 45491, offset 0, flags [none], proto UDP (17), length 392)
    10.35.128.254.bootps > 10.35.128.113.bootpc: BOOTP/DHCP, Reply, length 364, hops 2, xid 0xdd07ee25, Flags [none]
	  Your-IP 10.35.128.113
	  Server-IP 10.35.70.162
	  Client-Ethernet-Address 00:1a:4a:16:20:16
	  file "pxelinux.0"[|bootp]


On the bridge I see the DHCP replay but no in the vnet, and no new rule was created to allow traffic from the VM.


Laine, I have env for reproduce if you need.
Comment 26 Laine Stump 2018-02-14 19:26:26 UTC 
> I downgraded libvirt, but now libvirtd doesn't start:
> 
> Feb 13 11:34:01 lago-rhv-suite-4-2-host-0 libvirtd: /usr/sbin/libvirtd: error
> while loading shared libraries: libvirt-admin.so.0: cannot open shared object
> file: No such file or directory

Yeah, I encountered that too. For some reason "yum downgrade" doesn't update the symlinks from version-less .so filenames to the .so files specific to a particular libvirt build, so the old symlinks are left pointing to a .so that no longer exists. Fortunately if you run "yum reinstall ...." with the same arguments that you had just used for yum downgrade, the symlinks are properly updated.

BTW, I have now reproduced the problem on my RHEL7 system. I've found that the problem is not present with a libvirt-3.8.0 build from brew, but is present with 3.9.0.

One thing that was confusing the results of my tests until I figured it out - if you start a guest once and the "post-DHCP" rules are successfully loaded (meaning that a DHCP packet was seen by the necessary code), then any time after that until the next time the host system is restarted, the post-DHCP rules will be loaded *immediately*, without the need for a DHCP packet to be seen on the network. One result of this is that even a "broken" libvirtd will end up loading the correct rules, thus masking the failure. Only after rebooting the host will it properly wait for the DHCP packet to be seen before loading the rules. I tried restarting libvirtd, restarting the virtual network, and also clearing out the ARP cache, and none of these changed the behavior - the only way I can find so far is to reboot the host.

I now have two avenues to pursue - bisecting between 3.8.0 and 3.9.0, and attaching gdb to put breakpoints in the DHCPSnoop code.
Comment 27 Laine Stump 2018-02-17 21:14:52 UTC 
After a very long investigation with many red herrings and side trips to fix testing rig, I've found the (completely unexpected) source of this problem. The description in the patch I posted upstream will give more detailed info, but in short the problem was that the struct virMacAddr was changed to include an extra member ("generated"), not taking into account the fact that virMacAddr was used as part of the struct virNWFilterSnoopEthHdr, which was used to overlay a snooped ethernet packet and extract its info - the change in size of virMacAddr led to misalignment, and thus incorrect info was retrieved.

  https://www.redhat.com/archives/libvir-list/2018-February/msg00829.html

Since the proposed patch changes the fix for Bug 1343919, that bug should be retested after this patch is applied, to assure there is no regression in the behavior described in Bug 1343919.
Comment 28 Dan Kenigsberg 2018-02-18 18:53:34 UTC 
Laine, that's interesting. To ease verification from our side, would you produce a koji/brew build of libvirt-3.9 with your patch on top?
Comment 29 Laine Stump 2018-02-19 18:17:30 UTC 
A fix has been pushed to upstream libvirt master, so it will be included in libvirt-4.1.0:

commit e62cb4a9b78c7f4499a206635fb4f06e6ac627e5
Author: Laine Stump <laine>
Date:   Fri Feb 16 12:26:17 2018 -0500

    conf: move 'generated' member from virMacAddr to virDomainNetDef
Comment 31 Dan Kenigsberg 2018-02-20 09:13:20 UTC 
Given the assertion of a libvirt-side regression and the reproducer, I am re-requesting a resolution in 7.5.0.

I'll ask our QA to help in the verification.
Comment 33 Jiri Denemark 2018-02-20 12:48:56 UTC 
Fixed upstream by v4.0.0-262-ge62cb4a9b7.
Comment 34 Mor 2018-02-20 13:47:33 UTC 
Hi, I tried to check if the bew(In reply to Dan Kenigsberg from comment #31)
> Given the assertion of a libvirt-side regression and the reproducer, I am
> re-requesting a resolution in 7.5.0.
> 
> I'll ask our QA to help in the verification.

Hi Dan, Laine,

I tried to verify the issue with the fix, but I still don't get IP on the VM.

On host:
# rpm -qa |grep libvirt
libvirt-daemon-driver-network-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-config-network-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-storage-3.9.0-13.bz1529338.el7.x86_64
libvirt-devel-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-nodedev-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-kvm-3.9.0-13.bz1529338.el7.x86_64
libvirt-debuginfo-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-interface-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-qemu-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-storage-rbd-3.9.0-13.bz1529338.el7.x86_64
libvirt-3.9.0-13.bz1529338.el7.x86_64
libvirt-lock-sanlock-3.9.0-13.bz1529338.el7.x86_64
libvirt-docs-3.9.0-13.bz1529338.el7.x86_64
libvirt-libs-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-nwfilter-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-lxc-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-storage-disk-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-storage-mpath-3.9.0-13.bz1529338.el7.x86_64
libvirt-client-3.9.0-13.bz1529338.el7.x86_64
libvirt-nss-3.9.0-13.bz1529338.el7.x86_64
libvirt-login-shell-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-secret-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-storage-logical-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-lxc-3.9.0-13.bz1529338.el7.x86_64
libvirt-python-3.9.0-1.el7.x86_64
libvirt-daemon-driver-storage-core-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-config-nwfilter-3.9.0-13.bz1529338.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.9.0-13.bz1529338.el7.x86_64
libvirt-admin-3.9.0-13.bz1529338.el7.x86_64

# ebtables -t nat -L
Bridge table: nat

Bridge chain: PREROUTING, entries: 2, policy: ACCEPT
-j PREROUTING_direct
-i vnet0 -j libvirt-I-vnet0

Bridge chain: OUTPUT, entries: 1, policy: ACCEPT
-j OUTPUT_direct

Bridge chain: POSTROUTING, entries: 2, policy: ACCEPT
-j POSTROUTING_direct
-o vnet0 -j libvirt-O-vnet0

Bridge chain: PREROUTING_direct, entries: 0, policy: RETURN

Bridge chain: POSTROUTING_direct, entries: 0, policy: RETURN

Bridge chain: OUTPUT_direct, entries: 0, policy: RETURN

Bridge chain: libvirt-I-vnet0, entries: 2, policy: ACCEPT
-p IPv4 -s 0:1a:4a:16:20:31 --ip-proto udp --ip-sport 68 --ip-dport 67 -j ACCEPT 
-j DROP 

Bridge chain: libvirt-O-vnet0, entries: 3, policy: ACCEPT
-p IPv4 -d 0:1a:4a:16:20:31 --ip-src 10.46.16.106 --ip-proto udp --ip-sport 67 --ip-dport 68 -j ACCEPT 
-p IPv4 -d Broadcast --ip-src 10.46.16.106 --ip-proto udp --ip-sport 67 --ip-dport 68 -j ACCEPT 
-j DROP

mor_test_bug is a bridge created specially for testing this (both Request and Replay are captured):

# tcpdump -i mor_test_bug -n port 67 and port 68 -v
tcpdump: listening on mor_test_bug, link-type EN10MB (Ethernet), capture size 262144 bytes
15:31:09.238453 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:1a:4a:16:20:31, length 300, xid 0x2b817a15, Flags [none]
	  Client-Ethernet-Address 00:1a:4a:16:20:31
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Requested-IP Option 50, length 4: 10.35.128.242
	    Parameter-Request Option 55, length 13: 
	      Subnet-Mask, BR, Time-Zone, Classless-Static-Route
	      Domain-Name, Domain-Name-Server, Hostname, YD
	      YS, NTP, MTU, Option 119
	      Default-Gateway
15:31:09.253102 IP (tos 0x0, ttl 64, id 23004, offset 0, flags [none], proto UDP (17), length 392)
    10.35.128.254.bootps > 10.35.128.131.bootpc: BOOTP/DHCP, Reply, length 364, hops 2, xid 0x2b817a15, Flags [none]
	  Your-IP 10.35.128.131
	  Server-IP 10.46.16.106
	  Client-Ethernet-Address 00:1a:4a:16:20:31
	  file "pxelinux.0"[|bootp]
15:31:16.984653 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:1a:4a:16:20:31, length 300, xid 0x2b817a15, secs 7, Flags [none]
	  Client-Ethernet-Address 00:1a:4a:16:20:31
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Requested-IP Option 50, length 4: 10.35.128.242
	    Parameter-Request Option 55, length 13: 
	      Subnet-Mask, BR, Time-Zone, Classless-Static-Route
	      Domain-Name, Domain-Name-Server, Hostname, YD
	      YS, NTP, MTU, Option 119
	      Default-Gateway
15:31:17.013082 IP (tos 0x0, ttl 64, id 23831, offset 0, flags [none], proto UDP (17), length 392)
    10.35.128.254.bootps > 10.35.128.131.bootpc: BOOTP/DHCP, Reply, length 364, hops 2, xid 0x2b817a15, secs 7, Flags [none]
	  Your-IP 10.35.128.131
	  Server-IP 10.46.16.106
	  Client-Ethernet-Address 00:1a:4a:16:20:31
	  file "pxelinux.0"[|bootp]
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

On vnet0 only request is captured:

# tcpdump -i vnet0 -n port 67 and port 68 -v
tcpdump: listening on vnet0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:31:37.427198 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:1a:4a:16:20:31, length 300, xid 0xf753c81f, Flags [none]
	  Client-Ethernet-Address 00:1a:4a:16:20:31
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Requested-IP Option 50, length 4: 10.35.128.242
	    Parameter-Request Option 55, length 13: 
	      Subnet-Mask, BR, Time-Zone, Classless-Static-Route
	      Domain-Name, Domain-Name-Server, Hostname, YD
	      YS, NTP, MTU, Option 119
	      Default-Gateway
15:31:43.405517 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:1a:4a:16:20:31, length 300, xid 0xf753c81f, secs 6, Flags [none]
	  Client-Ethernet-Address 00:1a:4a:16:20:31
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Requested-IP Option 50, length 4: 10.35.128.242
	    Parameter-Request Option 55, length 13: 
	      Subnet-Mask, BR, Time-Zone, Classless-Static-Route
	      Domain-Name, Domain-Name-Server, Hostname, YD
	      YS, NTP, MTU, Option 119
	      Default-Gateway
15:32:01.342833 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:1a:4a:16:20:31, length 300, xid 0xf753c81f, secs 24, Flags [none]
	  Client-Ethernet-Address 00:1a:4a:16:20:31
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Requested-IP Option 50, length 4: 10.35.128.242
	    Parameter-Request Option 55, length 13: 
	      Subnet-Mask, BR, Time-Zone, Classless-Static-Route
	      Domain-Name, Domain-Name-Server, Hostname, YD
	      YS, NTP, MTU, Option 119
	      Default-Gateway
15:32:19.928297 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:1a:4a:16:20:31, length 300, xid 0xf753c81f, secs 42, Flags [none]
	  Client-Ethernet-Address 00:1a:4a:16:20:31
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Requested-IP Option 50, length 4: 10.35.128.242
	    Parameter-Request Option 55, length 13: 
	      Subnet-Mask, BR, Time-Zone, Classless-Static-Route
	      Domain-Name, Domain-Name-Server, Hostname, YD
	      YS, NTP, MTU, Option 119
	      Default-Gateway
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel


# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N FORWARD_IN_ZONES
-N FORWARD_IN_ZONES_SOURCE
-N FORWARD_OUT_ZONES
-N FORWARD_OUT_ZONES_SOURCE
-N FORWARD_direct
-N FWDI_public
-N FWDI_public_allow
-N FWDI_public_deny
-N FWDI_public_log
-N FWDO_public
-N FWDO_public_allow
-N FWDO_public_deny
-N FWDO_public_log
-N INPUT_ZONES
-N INPUT_ZONES_SOURCE
-N INPUT_direct
-N IN_public
-N IN_public_allow
-N IN_public_deny
-N IN_public_log
-N OUTPUT_direct
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 9090 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 16514 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 161 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p udp -m udp --dport 161 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 54321 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 5900:6923 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 49152:49216 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 54322 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 2223 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p udp -m udp --dport 6081 -m conntrack --ctstate NEW -j ACCEPT
Comment 35 Laine Stump 2018-02-20 15:40:15 UTC 
 The symptom of the bug wasn't that the guest would not get an IP address - the guest would get an IP address, but no traffic other than DHCP would be allowed to pass. Also, the original report said that the failure only occurred with RHEL7..5, not RHEL7.4.

The test systems in Comment 25 and Comment 34 are not reproducing the originally reported bug, and so are not testing the fix; they are displaying a *different* failure (which happens for them whether they are using RHEL7.4 or RHEL7.5), and one that I suspect is not related to anything libvirt is doing with iptables/ebtables - I think the host firewall (i.e. not configured by libvirt) is rejecting the DHCP responses arriving on vnet0. You can see if this is the case by running the following command in a shell on the host while the guest is starting:

while true; do iptables -v -S -Z |\
 grep -v -e "-c 0 0 " | grep -v ^Zero; sleep 5; done


This will print out all iptables rules that have matched a packet within the last 5 seconds - look for one that shows DROP or REJECT. Once you've seen the rule, look in the iptables -S output to see where it is and why it may have gotten there.

Oddly, I see slightly different behavior when I load up the exact set of iptables rules you have - I see the dhcp request packets on both interfaces, but don't see the response on either.

At any rate, you need to first set your host firewall to allow dhcp requests and responses in both directions on both your bridge and the tap, and *then* you can check whether or not this bug has been fixed
Comment 36 Laine Stump 2018-02-20 15:42:36 UTC 
Actually, this command will clear a lot more of the useless cruft from the iptables output: 


while true; do clear; iptables -v -S -Z |\
   grep -v -e "-c 0 0 " | grep -v ^Zero | grep -v ^-N; sleep 5; done
Comment 37 Mor 2018-02-21 08:25:18 UTC 
Hi Laine,

The host on my environment is running RHEL 7.5 Beta but with firewalld service turned on (iptables service is disabled).

If the filter settings (CTRL_IP_LEARNING and DHCPSERVER) are set on the vNIC, the VM is unable to get IP - I only see the DHCP requests on the vnet. If I stop firewalld on the host, the VM able to get IP. Is it good enough for verifying the original bug with the fix?
Comment 38 Mor 2018-02-21 08:27:47 UTC 
One more thing - if I remove the filter settings with firewalld on, the VM is able to get IP.
Comment 40 Laine Stump 2018-02-21 14:01:01 UTC 
Mor: The important thing for verifying the fix is the following:

1) setup the host *without the patched scratch build* so that when CTRL_IP_LEARNING and DHCPSERVER are *not* set in the guest config, the guest is able to get an IP address and ping the host (or some other external IP address).

2) add CTRL_IP_LEARNING and DHCPSERVER to the guest config, halt and restart the guest, and verify that it can get an IP address, but can't ping the host.

3) install the patched scratch build, halt and restart the guest, and verify that it can get an IP address and afterwards can ping the host / external machine.

Dan: for decisions about 0-day, I think we need to ask jsuchane. It may or may not be possible.
Comment 47 yalzhang@redhat.com 2018-02-28 05:07:44 UTC 
Reproduce on libvirt-3.9.0-13.el7.x86_64 by comment 40, and test on upstream libvirt-4.1.0-1.el7.x86_64, it works well.

1. Make sure firewalld is active. 
Test interfaces connected to default network and host existing bridge. 
# rpm -q libvirt
libvirt-3.9.0-13.el7.x86_64
# virsh dumpxml rhel | grep /interface -B8
    <interface type='network'>
      <mac address='52:54:00:00:56:27'/>
      <source network='default'/>
      <model type='virtio'/>
      <filterref filter='clean-traffic'>
        <parameter name='CTRL_IP_LEARNING' value='dhcp'/>
      </filterref>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

 # virsh dumpxml rhelq35 | grep /interface -B8
    <interface type='bridge'>
      <mac address='52:54:00:68:eb:c3'/>
      <source bridge='br0'/>
      <model type='rtl8139'/>
      <filterref filter='clean-traffic'>
        <parameter name='CTRL_IP_LEARNING' value='dhcp'/>
      </filterref>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/>
    </interface>

2. Start the 2 guests and log in the 2 guests, and check they both can obtain ip address, but can not ping outside

3. Upgrade to upstream libvirt-4.1.0-1.el7.x86_64
# yum update /tmp/libvirt/*
# systemctl restart libvirtd

4. log in the 2 guests and restart network service, the guests can ping outside successfully
Comment 52 Laine Stump 2018-03-06 22:09:01 UTC 
A more thorough test of libvirt's nwfilter on RHEL7 (including a test that exercises the exact problem described in this BZ - "220-no-ip-spoofing" will do the trick) is available by adding the repo file at:

  http://people.redhat.com/lstump/libvirt-tck-rhel7/libvirt-tck-rhel7.repo

into /etc/yum.repos.d, installing libvirt-tck:

  yum install perl-Sys-Virt-TCK

(this will likely install *a lot* of dependencies) then running the following command:

  libvirt-tck --force --testdir /usr/share/libvirt-tck/tests/nwfilter

Add --verbose to see more details during the test, or if there are failures. Also, unless you have domains/networks/nwfilters/storage pools/secrets with names that start with "tck", you can ignore the warning that the test will erase all your libvirt config.
Comment 55 yalzhang@redhat.com 2018-03-08 16:09:46 UTC 
Test on libvirt-3.9.0-14.el7.x86_64, the issue is fixed.

1. start vm set as below

# virsh dumpxml rhel | grep /interface -B12
    <interface type='network'>
      <mac address='52:54:00:1b:a5:b9'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <filterref filter='clean-traffic'>
        <parameter name='CTRL_IP_LEARNING' value='dhcp'/>
        <parameter name='DHCPSERVER' value='192.168.122.1'/>
      </filterref>
      <link state='up'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:c0:74:fa'/>
      <source bridge='testbr'/>
      <target dev='vnet1'/>
      <model type='rtl8139'/>
      <filterref filter='clean-traffic'>
        <parameter name='CTRL_IP_LEARNING' value='dhcp'/>
        <parameter name='DHCPSERVER' value='10.73.x.x'/>
      </filterref>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>

2. log in vm and check the 2 interfaces both can get ip addresses and ping outside.

Also run test in comment 52, all tests pass, and we plan to add these tests into our CI job.
Comment 58 errata-xmlrpc 2018-04-10 11:04:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0704
Comment 59 Red Hat Bugzilla 2023-09-14 04:14:54 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days
Note You need to log in before you can comment on or make changes to this bug.