Bug 2177350

Tested with both `FirewallBackend=nftables` (initially while reporting)
and `FirewallBackend=iptables` (later).  None of those worked.

The work-around which works for us for now:

1. uninstall firewalld - install iptables-service
2. install /etc/sysconfig/ip*tables rules related to host
3. enable && start iptables.service
4. restart libvirtd to re-install iptables rules from nwfilter(s)
   (and anytime iptables.service is restarted in the future)

Eric, does RHEL8 firewalld enable bridge-nf-call-iptables or something?

(In reply to Phil Sutter from comment #3)
> Eric, does RHEL8 firewalld enable bridge-nf-call-iptables or something?

No. It does not.

Reported kernel version already contains the fix mentioned in bug 1650382. So a
simple "-D ... -m physdev" doesn't cause loading of br_netfilter module. Yet
adding a rule with '-m physdev' does.

If starting firewalld causes the problem, I guess it adds a rule using physdev.
It's strange though that using nftables backend doesn't avoid the issue.

So where does this BZ belong? It's not libvirt. Should I reassign to firewalld? iptables? kernel/netfilter? I'm not trying to just get rid of it, but if it stays assigned to libvirt then it risks falling through the cracks, because there's nothing we (libvirt) can do for it.

(For Pavel's benefit: Bug 1650382 was the bug I was trying to remember when I responded to the email on tech-list. I finally found it this morning after a conversation with Phil on IRC. It was a *very* similar situation. Also, just to be clear - libvirt does not "pull in the br_netfilter module" as you've said in the description (well, if Bug 1650382 has returned then libvirt could be triggering the bug, but that's about it)

In the meantime, Pavel - maybe you could try this to gather more info:

1) stop firewalld
2) rmmod br_netfilter (and verify that it is unloaded)
3) start firewalld

If br_netfilter is once again loaded, then get a dump of all iptables rules that can be checked to see if there is any rule that would be triggering the load of br_netfilter).

This may or may not give us useful information.

BTW, I don't know if it's still the case, but at one time podman was talking about (or maybe did it) forcing br_netfilter to be loaded on all systems that have podman installed. See my objection to that idea here: Bug 1703261 Comment 34.

Is podman installed on this machine?

Also resurfaced later - Bug 1832723 Comment 8

Pavel, how can I reproduce this? I have a RHEL8.8 machine with firewalld
running. Starting a libvirt VM with some nwfilter defined does not trigger the
problem. What specific nwfilter do you use?

We can let anyone more experience in (SSH), and experiment.  Just ping me
on IRC.

(In reply to Phil Sutter from comment #5)
> If starting firewalld causes the problem, I guess it adds a rule using
> physdev.

I'm not sure the exact conditions, but sometimes firewalld loads the module.
When firewalld is disabled, libvirtd loads it anyway on its own.

> It's strange though that using nftables backend doesn't avoid the issue.

I tested most of the time with this default.  I did all the testing once more
today (2h) and never switched this default.

(In reply to Laine Stump from comment #7)
> In the meantime, Pavel - maybe you could try this to gather more info:
> 
> 1) stop firewalld
> 2) rmmod br_netfilter (and verify that it is unloaded)
> 3) start firewalld

This doesn't always happen, happened just once again to me (see the IRC flow)
but once everything settles down - stopping and starting firewalld doesn't
load the module.

(In reply to Laine Stump from comment #8)
> BTW, I don't know if it's still the case, but at one time podman was talking
> about (or maybe did it) forcing br_netfilter to be loaded on all systems
> that have podman installed. See my objection to that idea here: Bug 1703261
> Comment 34.
> 
> Is podman installed on this machine?

There is Podman installed, but nobody touched that for weeks.  The (latest)
version of Podman we have doesn't install any modprobe.d config files that
would cause loading br_netfilter anyway.

(In reply to Phil Sutter from comment #10)
> Pavel, how can I reproduce this? I have a RHEL8.8 machine with firewalld
> running. Starting a libvirt VM with some nwfilter defined does not
> trigger the problem. What specific nwfilter do you use?

- I'm on RHEL 8.7, does it make any difference?
- We do have 'br0' device which is the "master" of eno1
- We connect the VMs through
  <network connections='13'>
    <name>host-bridge</name>
    <uuid>72e0ce12-9f1f-4bca-a316-1797531116bc</uuid>
    <forward mode='bridge'/>
    <bridge name='br0'/>
  </network>
  But the same happens when we start a VM on br0 directly (without network
  abstraction).
- podman is installed but unused

But I'm not the best person to judge I assume, you'd see much more if you
tried to SSH-in.

Laine, in libvirt upstream sources I see nwfilter_ebiptables_driver.c making
use of physdev match. For unclear reasons I can't provoke libvirt in RHEL8 to
actually emit such a rule. Can you tell why this does not happen? To me it
seems I have to use some nwfilter which uses IPv4, but the no-ip-spoofing one
at least doesn't do the trick.

Is the ebiptables driver used at all if firewalld is active? Maybe I'm trying
to catch red herrings.

If it is possible to make libvirt add a rule using '-m physdev', this will
cause loading of br_netfilter.ko.

Laine,

In Pavel's setup I see iptables rules using physdev, e.g.:

| -A libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0

they are clearly added by libvirt. So the latter is (indirectly) responsible
for loading br_netfilter.ko. Is this expected if firewalld is running?

Is firewalld supported with libvirt or must it be disabled?

> Is firewalld supported with libvirt or must it be disabled?

The virtual network driver in libvirt plays reasonably well with firewalld (we place virtual networks into their own zone), and we've always madde it a priority to maintain that compatibility, since a lot of people use libvirt's virtual networks on systems that have firewalld enabled by default.

But the nwfilter driver was written prior to firewalld even existing, and the level of libvirt-team support on it has been just to keep it generating the same rules as it did in the beginning (with very few exceptions), and none of the higher-level management systems (OpenStack, RHV, Kubevirt) use it (except maybe the clean-traffic filter in some cases), so it hasn't had much importance or visibility.

Oh, and BTW, I obviously take back Comment 6. The fact that I didn't have this info in my brain shows just what the level of use/interest in the nwfilter driver is; we may have encountered this same problem before, but if we did it's been so long that it has completely run off the end of my FIFO.

Does that mean that there are no plans to make this work in the future?
I'd at least recommend properly documenting this.

Turning off firewalld and switching to iptables then (with br_netfilter
loaded) was the only correct thing we could do to connect the guests to LAN
and have _both_ host and guests firewalled.  At least without an additional
layer of networking or other logic (iow. plain libvirt).  Note our guests are
controlled by our users, so we can't just rely on the guest-level firewalls
that could be easily turned off.

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.