RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1942805 - cannot restart default network and firewalld: iptables: No chain/target/match by that name.
Summary: cannot restart default network and firewalld: iptables: No chain/target/match...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: libvirt
Version: 8.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: yalzhang@redhat.com
URL:
Whiteboard:
: 1943817 (view as bug list)
Depends On: 1813830
Blocks: 1958301
TreeView+ depends on / blocked
 
Reported: 2021-03-25 03:12 UTC by Laine Stump
Modified: 2021-11-09 21:47 UTC (History)
22 users (show)

Fixed In Version: libvirt-6.0.0-36.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1813830
: 1958301 (view as bug list)
Environment:
Last Closed: 2021-11-09 18:00:11 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:4191 0 None None None 2021-11-09 18:00:53 UTC

Description Laine Stump 2021-03-25 03:12:23 UTC 
(This was just reported by a RHEL (CentOS?) user in IRC, which led me to realize that while the RHEL-AV libvirt package had this bug fixed by virtue of a rebase to libvirt-6.6.0 in RHEL-AV-8.3.0, the standard RHEL8 libvirt package actually had this bug *introduced* in RHEL-8.3.0 by rebasing from libvirt-4.5.0 (when we didn't yet have private iptables chains) to libvirt-6.0.0.

Since libvirt in standard RHEL8 will remain based on 6.0.0 for quite some time still, we need to backport the patches described here. I think there are also one or two other similar fixes related to switching to private chains that need to be found and backported at the same time.

+++ This bug was initially created as a clone of Bug #1813830 +++

Description of problem: In a stock Fedora 31 or 32 one often cannot restart libvirt's "default" network (virbr0). Right after booting, in a standard install, the default network is active and registered as a zone in firewalld:


# virsh net-list --all; firewall-cmd --get-active-zones
 Name      State    Autostart   Persistent
--------------------------------------------
 default   active   yes         yes

libvirt
  interfaces: virbr0
public
  interfaces: eth0

While firewalld stays running, I can `virsh net-destroy default` and `virsh net-start default`, and everything works.

I can also `systemctl stop firewalld` and `systemctl start firewalld`, and everything is still in  order.

But I can't do both together:

# virsh net-destroy default; systemctl stop firewalld

# virsh net-list --all; firewall-cmd --get-active-zones
 Name      State      Autostart   Persistent
----------------------------------------------
 default   inactive   yes         yes

FirewallD is not running

# systemctl start firewalld; virsh net-start default

error: Failed to start network default
error: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --insert LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: No chain/target/match by that name.


This isn't a race condition -- you can wait between starting firewalld and the default network, or retry the latter, always the same result.


There are a lot of errors in the journal during `virsh net-start default`:

NetworkManager[640]: <info>  [1584349179.8789] manager: (virbr0): new Bridge device (/org/freedesktop/NetworkManager/Devices/10)
kernel: virbr0: port 1(virbr0-nic) entered blocking state
kernel: virbr0: port 1(virbr0-nic) entered disabled state
kernel: device virbr0-nic entered promiscuous mode
kernel: kauditd_printk_skb: 8 callbacks suppressed
kernel: audit: type=1700 audit(1584349179.887:530): dev=virbr0-nic prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
audit: ANOM_PROMISCUOUS dev=virbr0-nic prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
systemd-udevd[7994]: Using default interface naming scheme 'v243'.
systemd-udevd[7995]: Using default interface naming scheme 'v243'.
systemd-udevd[7994]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
systemd-udevd[7995]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
NetworkManager[640]: <info>  [1584349179.9267] manager: (virbr0-nic): new Tun device (/org/freedesktop/NetworkManager/Devices/11)
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --insert LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: No chain/target/match by that name.
libvirtd[733]: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --insert LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: No chain/target/match by that name.
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 --destination 224.0.0.0/24 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 --destination 255.255.255.255/32 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 -p tcp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 -p udp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: Bad rule (does a matching rule exist in that chain?).
NetworkManager[640]: <info>  [1584349179.9985] device (virbr0-nic): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
NetworkManager[640]: <info>  [1584349180.0011] device (virbr0-nic): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external')
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 ! --destination 192.168.122.0/24 --jump MASQUERADE' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWI --destination 192.168.122.0/24 --out-interface virbr0 --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWO --source 192.168.122.0/24 --in-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWX --in-interface virbr0 --out-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWI --out-interface virbr0 --jump REJECT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWO --in-interface virbr0 --jump REJECT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_INP --in-interface virbr0 --protocol udp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_OUT --out-interface virbr0 --protocol udp --destination-port 68 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_INP --in-interface virbr0 --protocol udp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
kernel: device virbr0-nic left promiscuous mode
kernel: virbr0: port 1(virbr0-nic) entered disabled state
kernel: audit: type=1700 audit(1584349180.080:531): dev=virbr0-nic prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
audit: ANOM_PROMISCUOUS dev=virbr0-nic prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
NetworkManager[640]: <info>  [1584349180.0930] device (virbr0-nic): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed')
NetworkManager[640]: <info>  [1584349180.0951] device (virbr0-nic): released from master device virbr0


Version-Release number of selected component (if applicable):

libvirt-daemon-5.6.0-5.fc31.x86_64
firewalld-0.7.3-1.fc31.noarch
iptables-1.8.3-7.fc31.x86_64

Also confirmed on current F32:

libvirt-daemon-6.1.0-1.fc32.x86_64
firewalld-0.8.1-1.fc32.noarch
iptables-nft-1.8.4-7.fc32.x86_64
nftables-0.9.3-2.fc32.x86_64


How reproducible: Always

--- Additional comment from Martin Pitt on 2020-03-16 09:10:30 UTC ---

I think I found a workaround: When restarting libvirtd.service *after* starting firewalld, it works:

# systemctl start firewalld; systemctl try-restart libvirtd; virsh net-start default

(restarting it before starting firewalld it doesn't work). This is good enough for our integration tests.


--- Additional comment from Laine Stump on 2020-05-08 01:53:44 UTC ---

This is due to an optimization in libvirt's network driver. Because 1) attempting to create a private iptables chain when it already exists results in an error, and 2) repeatedly looking to see if a chain exists for every network that is started is very inefficient and makes for a slow startup time, the network driver only calls the function that creates the private chains once per libvirtd run. This was done based on the assumption that firewalld would keep all of the iptables chains created by libvirt intact, which is in fact not the case.

Instead we need to recreate the private chains any time we are notified (via dbus) that firewalld has restarted.

--- Additional comment from Laine Stump on 2020-05-08 02:53:52 UTC ---

I just posted these two patches upstream:

https://www.redhat.com/archives/libvir-list/2020-May/msg00344.html

--- Additional comment from Laine Stump on 2020-05-19 17:41:06 UTC ---

The following two patches will be in upstream libvirt-6.4.0. We should backport them to both the F31 and F32 builds of libvirt.

commit de110f110fb917a31b9f33ad8e4b3c1d3284766a
Author: Laine Stump <laine>
Date:   Thu May 7 22:32:59 2020 -0400

    network: make it safe to call networkSetupPrivateChains() multiple times

commit f5418b427e7d2f26803880309478de9103680826
Author: Laine Stump <laine>
Date:   Thu May 7 21:54:39 2020 -0400

    network: force re-creation of iptables private chains on firewalld restart
Comment 1 Laine Stump 2021-03-28 00:26:02 UTC 
*** Bug 1943817 has been marked as a duplicate of this bug. ***
Comment 7 yalzhang@redhat.com 2021-05-17 04:02:11 UTC 
Reproduce this bug on libvirt-6.0.0-35.module+el8.5.0+10709+b3edb581.x86_64

1. Ensure libvirtd is started when the firewalld is running;
# systemctl restart firewalld; systemctl restart libvirtd; virsh net-start default

2. destroy the network and stop firewalld;
# virsh net-destroy default; systemctl stop firewalld
Network default destroyed

# virsh net-list --all; firewall-cmd --get-active-zones
 Name      State      Autostart   Persistent
----------------------------------------------
 default   inactive   yes         yes

FirewallD is not running


2. Start the firewalld, then try to start the network;
# systemctl start firewalld; virsh net-start default
error: Failed to start network default
error: internal error: Failed to apply firewall rules /usr/sbin/iptables -w --table filter --insert LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT: iptables: No chain/target/match by that name.

And test on libvirt-7.3.0-1.module+el8.5.0+11004+f4810536.x86_64 which should include the patch, the result is pass.
Comment 12 yalzhang@redhat.com 2021-06-04 11:41:21 UTC 
> I can also `systemctl stop firewalld` and `systemctl start firewalld`, and everything is still in  order.

When I change the firewalld status since libvirtd started, no matter from inactive to active or vice versa, or just restart, it will clear all the rules and chains. And the NAT for the default network will not work, not "still in order".

I remember there was a related bugzilla comment a long time ago, but I can not find it now. I remember its general idea was “do not change firewalld status when libvirtd is running” or “restart libvirtd once firewalld status changes, including restart”. Maybe my memory or understanding is wrong. And I never test such scenarios nor consider it as a bug. But seems the behavior is different now. What do you think about it? Please help to confirm. And please help to check scenario 2, network fail to start after firewalld restart.

Test on libvirt-6.0.0-36.module+el8.5.0+11222+c889b3f3.x86_64 with scenarios as below:
Scenario 1: Same steps as the reproduce procedure "do both together":
1) ensure the network is active:
# virsh net-list --all
 Name      State    Autostart   Persistent
--------------------------------------------
 default   active   yes         yes

2) destroy network and stop firewalld:
# virsh net-destroy default; systemctl stop firewalld
Network default destroyed

# virsh net-list --all; firewall-cmd --get-active-zones
 Name      State      Autostart   Persistent
----------------------------------------------
 default   inactive   yes         yes

FirewallD is not running

3) start firewalld and try to start the network, it can start successfully, which is expected.
# systemctl start firewalld; virsh net-start default
Network default started


Scenario 2: start network after restart firewalld, network can not start successfully. It happens about 80% of all the times I have tried.

# cat restart_test.sh
#!/bin/bash
# this is to confirm the env is clean
systemctl restart libvirtd
virsh net-start default
virsh net-list --all
systemctl restart firewalld; systemctl restart libvirtd; 
echo "*************before firewalld restart*****************"
iptables -L
systemctl restart firewalld
echo "*************after firewalld restart*************************"
iptables -L
virsh net-destroy default;
virsh net-start default;

# ./restart_test.sh
Network default started

 Name      State    Autostart   Persistent
--------------------------------------------
 default   active   yes         yes

*************before firewalld restart*****************
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_INP  all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_FWX  all  --  anywhere             anywhere            
LIBVIRT_FWI  all  --  anywhere             anywhere            
LIBVIRT_FWO  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_OUT  all  --  anywhere             anywhere            

Chain LIBVIRT_INP (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootps

Chain LIBVIRT_OUT (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootpc
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootpc

Chain LIBVIRT_FWO (1 references)
target     prot opt source               destination         
ACCEPT     all  --  192.168.124.0/24     anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain LIBVIRT_FWI (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             192.168.124.0/24     ctstate RELATED,ESTABLISHED
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
*************after firewalld restart*************************
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
Network default destroyed

error: Failed to start network default
error: internal error: Failed to apply firewall rules /usr/sbin/iptables -w --table filter --insert LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT: iptables: No chain/target/match by that name.
Comment 13 Laine Stump 2021-06-10 05:18:50 UTC 
The thing that you're remembering is that it's not supported to either stop firewalld (and have it remain stopped) or start firewalld (and have it remain started) while a single libvirtd is running - if you do this you need to restart libvirtd.

The test that you're running *should* be working properly though, because you only stop firewalld and then immediately start it (which is effectively the same as a restart, which is fine). My guess is that possibly you are running iptables -L and attempting to start the default network the final time so soon after firewalld has been restarted that there hasn't yet been time for libvirtd to receive and process the dbus message telling it that firewalld has been restarted. Try putting a sleep 5 or something into the script just before the final "iptables -L".

Assuming my theory is correct, I don't think there is any way for libvirt to guarantee a successful start of a network in this case - it doesn't know that firewalld has restarted until it gets the dbus message, but that message won't arrive until some time after all of libvirt's chains/rules have already been removed by the firewalld restart; if libvirt tries to start a new network during that time, it won't know that it's chains are temporarily missing, and so will be correct in complaining about it. As long as the rules are "eventually" added back, and a delayed attempt to start a new network is successful without any other intervention, then I think we've done the best that we can do.
Comment 14 yalzhang@redhat.com 2021-06-15 01:56:24 UTC 
Hi laine, after sleep 10, it still fail, please help to have a look:

# rpm -q libvirt firewalld
libvirt-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
firewalld-0.8.2-6.el8.noarch

# cat test.sh
#!/bin/bash
# this is to confirm the env is clean
systemctl restart libvirtd
virsh net-start default
virsh net-list --all
systemctl restart firewalld; sleep 5; systemctl restart libvirtd; 
echo "*************before firewalld restart*********************"
sleep 5
iptables -L
systemctl restart firewalld
echo "*************after firewalld restart*************************"
echo "sleep 10 here:"
sleep 10
iptables -L
virsh net-destroy default;
virsh net-start default;

# ./test.sh
Network default started

 Name          State    Autostart   Persistent
------------------------------------------------
 default       active   yes         yes
 host-bridge   active   no          yes
 hostdev-net   active   no          yes

*************before firewalld restart*********************
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_INP  all  --  anywhere             anywhere

...(the Chains, rules are all expected)...

*************after firewalld restart*************************
sleep 10 here:
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
Network default destroyed

error: Failed to start network default
error: internal error: Failed to apply firewall rules /usr/sbin/iptables -w --table filter --insert LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT: iptables: No chain/target/match by that name.
Comment 15 yalzhang@redhat.com 2021-06-16 14:19:41 UTC 
Sorry, I used the old version in comment 14. With new version libvirt-6.0.0-36.module+el8.5.0+11222+c889b3f3.x86_64, when I set sleep time, the result is as expected. Set the bug to be verified.
Comment 17 errata-xmlrpc 2021-11-09 18:00:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4191
Note You need to log in before you can comment on or make changes to this bug.