Bug 1832723 - Libvirt VMs running on Bridged Network don't get IPv4 DHCP, only IPv6
Summary: Libvirt VMs running on Bridged Network don't get IPv4 DHCP, only IPv6
Alias: None
Product: Fedora
Classification: Fedora
Component: podman
Version: 32
Hardware: x86_64
OS: Unspecified
Target Milestone: ---
Assignee: Lokesh Mandvekar
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2020-05-07 07:54 UTC by Andreas Stolzenberger
Modified: 2020-09-10 13:09 UTC (History)
32 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-09-10 13:09:40 UTC
Type: Bug

Attachments (Terms of Use)
Screenshot Virt-Viewer, Start VM with firewall Default (88.57 KB, image/png)
2020-05-07 08:07 UTC, Andreas Stolzenberger
no flags Details
Virt-Viewer. VM start with firewalld turned off on F32 host (96.60 KB, image/png)
2020-05-07 08:10 UTC, Andreas Stolzenberger
no flags Details
File to save as /etc/sysctl.d/81-bridge-nf-off.conf as a temporary workaround until we have a final solution (286 bytes, text/plain)
2020-06-07 23:12 UTC, Edgar Hoch
no flags Details

Description Andreas Stolzenberger 2020-05-07 07:54:17 UTC
Description of problem:

When I run a VM on a bridged Network (Bridge0, connected to enp39s0), the VM cannot get a DHCP4 address only IPv6.

Version-Release number of selected component (if applicable):
Fedora 32 (it worked on Fedora 31)

How reproducible:
Create Bridged Network, add new VM on Bridged Network, try to boot via PXE

Steps to Reproduce:
1. Create new VM on Bridged Network
2. Start VM. No PXE Boot. 
3. Boot from CD, enable Network: VM only get's an IPv6 address

Actual results:
No PXE Boot, no IPV4 address through DHCP

Expected results:
PXE Boot Menu and valid IPv4 address

Additional info:
Setup worked flawless on Fedora 31, But showed up after Update to Fedora 32
No user rules on the firewall

Comment 1 Andreas Stolzenberger 2020-05-07 08:07:52 UTC
Created attachment 1686083 [details]
Screenshot Virt-Viewer, Start VM with firewall Default

Launching a CentOS VM on a Bridged Network with the Default FW-Settings on the Fedora 32 Host. VM only gets an IPv6 adress.

Comment 2 Andreas Stolzenberger 2020-05-07 08:10:45 UTC
Created attachment 1686084 [details]
Virt-Viewer.  VM start with firewalld turned off on F32 host

After turning off FirewallD on Fedora 32 Host, the bridged VM gets an IPv4 and IPv6 adress as expected. -> On Fedora 31 the VM worked this way with FirewallD switched on.

Comment 3 Nathanael Noblet 2020-05-08 20:51:16 UTC
I have similar issue. I have multiple VMs that have worked for years in this setup that now I have to disable firewalld for them to get IPv4 addresses.

Comment 4 Laine Stump 2020-05-08 22:30:12 UTC
Nathan and Andreas - if you manually set and IPv4 address, does IPv4 work? (i.e. is it just DHCP that's being blocked? Or is it all IPv4 traffic?

Eric - has there been a change to firewalld rules in F32 vs F31 you're aware of that would break IPv4 for a tap device connected to a bridge device that's attached to the host's ethernet?

Comment 5 Laine Stump 2020-05-09 04:20:14 UTC
Wait. firewalld switched from using the iptables backend in F31 to using the nftables backend in F32. However, RHEL8 firewalld is also using the nftables backend, and traffic goes across ethernet-attached bridges with no problems there. Eric?

I'm re-assigning this to firewalld, since it's occuring with VMs attached to a bridge that is directly attached to an ethernet (not to a libvirt-created bridge). libvirt has never done any sort of management of the firewall for bridges like this, which are created by the host network config (usually NetworkManager) - traffic between VMs attached to these bridges and the physical network has always been untouched by the host system firewall.

Comment 6 Laine Stump 2020-05-09 04:42:21 UTC
Oh piffle.

It's because something is again causing the br_netfilter module to load, and so all traffic traversing bridges is sent through the firewall.

This happens no matter what the firewalld backend is set to. It isn't a firewalld problem after all. We just need to figure out who is causing br_netfilter to load. There was a problem with this a year or two ago, where attempting to *remove* a particular type of iptables rule would cause iptables to autoload br_netfilter (or something like that). I'll try a few things out in the morning. But in the meantime I'm moving the bug back to libvirt (it's not our bug, but it *really* isn't firewalld's bug :-)

In the meantime, if you run "rmmod br_netfilter", your guests will once again be able to pass IPv4 traffic (until whatever is autoloading it does that again, anyway).

Comment 7 Juan Orti 2020-05-09 20:54:36 UTC
Same here. After the upgrade to F32, my bridged VMs are not getting IP with DHCPv4. I'm sniffing at the external switch and the broadcast packets never reach out of the host.


Comment 8 Laine Stump 2020-05-10 03:56:26 UTC

On a freshly installed F32 Workstation:

  # cat /etc/modules-load.d/podman.conf
  br_netfilter        <====== this means that br_netfilter will be modloaded at every boot.

  dnf provides /etc/modules-load.d/podman.conf
  podman-2:1.9.1-1.fc32.x86_64 : Manage Pods, Containers and Container Images
  Repo        : @System
  Matched from:
  Filename    : /etc/modules-load.d/podman.conf

On an F31 machine:

  (/etc/modules-load.d/podman.conf doesn't exist)

  # rpm -qa | grep podman

  # dnf provides /etc/modules-load.d/podman.conf
  Error: no matches found

So between podman-1.9.0 and 1.9.1 podman has apparently added default configuration that forces br_netfilter to be loaded unconditionally at boot time, and since the default for net.bridge.bridge-nf-call-iptables (which takes effect whenever br_netfilter loaded) is 1, this means that all traffic traversing the bridge will be sent through iptables/nftables (and most likely fail if there are any sort of rules there at all).

IMO, podman shouldn't be assuming they can change such a basic behavior of networking as the default - if they need this filtering (which apparently is because they want to take advantage of port redirection DNAT rules? - see Bug 1703261) then they should at least make it configurable, and OFF by default.

I'm moving this to podman to see what they have to say about it.

Comment 9 Nathanael Noblet 2020-05-11 03:43:55 UTC
Apologies I just saw this now. Although it seems the source has been found, after ipv4 address is assigned by temporarily disabling firewalld normal IPv4 traffic flows as expected. If the br_netfilter module is the culprit and podman needs it what is my work around? I did install podman on this machine to start playing with it but there are no containers running when the problem occurs.

Comment 10 Andreas Stolzenberger 2020-05-11 08:24:58 UTC
Unloading br_netfilter solves the problem. Disabling firewalld while leaving br_netfilter loaded does *not* work. IPv4 DHCP gets through then, but TFTP is blocked so PXE-booting VMs can't load pxelinux.0.

Comment 11 Nathanael Noblet 2020-05-11 15:43:36 UTC
Ah yeah sorry Andreas, in my case I'm not using PXE/TFTP. Just DHCP. In my case unloading it and booting a VM allows DHCP through with firewalld enabled. With br_netfilter loaded I can get DHCP through by disabling firewalld for DHCP and then re-enabling it for normal traffic. Looks like different use cases.

Comment 12 Matthew Heon 2020-05-11 15:46:08 UTC
Sorry about this one. I've talked with the rest of the Podman team, and we're going to revert this. We were unaware there would be an impact on Libvirt - wouldn't have landed this if we realized.

Comment 13 Tom Sweeney 2020-05-11 21:44:38 UTC
This is most likely related to https://bugzilla.redhat.com/show_bug.cgi?id=1832723.  The same upstream fix should address this BZ and that one, but with different packaging needs.

Comment 14 Tom Sweeney 2020-05-12 16:02:10 UTC
My last comment pointed to this BZ rather than the correct one:  https://bugzilla.redhat.com/show_bug.cgi?id=1703261

Comment 15 Laine Stump 2020-05-15 00:00:29 UTC
When you say "revert", you mean that you'll revert it upstream and also backport the revert to Fedora 32 and Rawhide, correct? As long as this remains in the fedora packages, we will continue to get bug reports that bridged networking no longer works.


Comment 16 Matthew Heon 2020-05-15 13:11:53 UTC
Podman v1.9.2, which should be in updates-testing now for F31/F32, removes the `br_netfilter` requirement.

Comment 17 Tom Sweeney 2020-05-15 18:10:55 UTC
Assigning to Jindrich for any packaging needs.

Comment 19 Edgar Hoch 2020-06-07 22:28:34 UTC
podman-1.9.3-1.fc32.x86_64 - current version for Fedora 32 - still contains /etc/modules-load.d/podman.conf which loads br_netfilter.

I confirm that unloading br_netfilter solves the problem with my virtual machines using a bridge connected to an ethernet device.

Another solution would be to disable the use *tables when br_netfilter is loaded. It seams that the br_netfilter adds the following kernel parameters, they don't exist when br_netfilter is not loaded.

# sysctl -a|grep bridge
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-pass-vlan-input-dev = 0

Setting net.bridge.bridge-nf-call-iptables=0 also solves the problem for IPv4 for me. net.bridge.bridge-nf-call-ip6tables=0 will do the same for IPv6. I'm not sure if it neccessary to set net.bridge.bridge-nf-call-arptables=0.

For being permanent, these assignments can be put in a file in /etc/sysctl.d/*.conf (as a temporary workaround for admins).

I have tried to set some rules for firewalld or nft when br_netfilter was loaded and the bridge kernel parameter on default value, but I was not successful. I think it would be more flexible to have firewall rules that allow or deny forwarding over a bridge in general and selectively for bridges and slave devices (e.g. virtual machines). So some vms and containers can filer there network access, and others may block or allow all.

It would also be fine to have options / parameters for libvirt / virsh / virt-manager to set network filter rules on network devices assigned to virtual machines that are automatically applied when the vm starts (because it created a dynamic network device vnet*). I found nwfilter* command in virsh, but no documentation and examples how I can assign rules to a vm. virt-manager doesn't show any configuration options about nwfilter.

Comment 20 Edgar Hoch 2020-06-07 23:12:07 UTC
Created attachment 1695942 [details]
File to save as /etc/sysctl.d/81-bridge-nf-off.conf as a temporary workaround until we have a final solution

Comment 21 Daniel Walsh 2020-06-08 19:52:25 UTC
Fixed in  podman-2.0.0-rc4.1.fc32

Comment 22 Edgar Hoch 2020-06-11 10:29:13 UTC
(In reply to Daniel Walsh from comment #21)
> Fixed in  podman-2.0.0-rc4.1.fc32

Thanks for the update. But is podman 2.0.0 rc4 ready for production?
Bodhi: "This update's test gating status has been changed to 'failed'." Status is still Pending->Testing.
Brent Baude has build podman 2.0.0 rc5 on koji in the meantime.

If podman 2.* isn't ready for general use now, I suggest build an updated version of podman 1.9.* which is currently shipped with Fedora 32.

Comment 23 Daniel Walsh 2020-06-16 21:44:26 UTC
We are working to get podman 2.0 out the door. It will be the next release.

Comment 24 Lokesh Mandvekar 2020-09-10 13:09:40 UTC
Closing, please reopen if issue isn't fixed.

Note You need to log in before you can comment on or make changes to this bug.