Bug 1813830

Summary:	cannot restart default network and firewalld: iptables: No chain/target/match by that name.
Product:	[Fedora] Fedora	Reporter:	Martin Pitt <mpitt>
Component:	libvirt	Assignee:	Laine Stump <laine>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	31	CC:	agedosier, berrange, clalancette, itamar, jfehlig, jforbes, laine, libvirt-maint, rjones, veillard, virt-maint
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-5.6.0-7.fc31	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1942805 (view as bug list)		Environment:
Last Closed:	2020-06-16 01:18:33 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1942805

Description Martin Pitt 2020-03-16 09:01:55 UTC

Description of problem: In a stock Fedora 31 or 32 one often cannot restart libvirt's "default" network (virbr0). Right after booting, in a standard install, the default network is active and registered as a zone in firewalld:


# virsh net-list --all; firewall-cmd --get-active-zones
 Name      State    Autostart   Persistent
--------------------------------------------
 default   active   yes         yes

libvirt
  interfaces: virbr0
public
  interfaces: eth0

While firewalld stays running, I can `virsh net-destroy default` and `virsh net-start default`, and everything works.

I can also `systemctl stop firewalld` and `systemctl start firewalld`, and everything is still in  order.

But I can't do both together:

# virsh net-destroy default; systemctl stop firewalld

# virsh net-list --all; firewall-cmd --get-active-zones
 Name      State      Autostart   Persistent
----------------------------------------------
 default   inactive   yes         yes

FirewallD is not running

# systemctl start firewalld; virsh net-start default

error: Failed to start network default
error: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --insert LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: No chain/target/match by that name.


This isn't a race condition -- you can wait between starting firewalld and the default network, or retry the latter, always the same result.


There are a lot of errors in the journal during `virsh net-start default`:

NetworkManager[640]: <info>  [1584349179.8789] manager: (virbr0): new Bridge device (/org/freedesktop/NetworkManager/Devices/10)
kernel: virbr0: port 1(virbr0-nic) entered blocking state
kernel: virbr0: port 1(virbr0-nic) entered disabled state
kernel: device virbr0-nic entered promiscuous mode
kernel: kauditd_printk_skb: 8 callbacks suppressed
kernel: audit: type=1700 audit(1584349179.887:530): dev=virbr0-nic prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
audit: ANOM_PROMISCUOUS dev=virbr0-nic prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
systemd-udevd[7994]: Using default interface naming scheme 'v243'.
systemd-udevd[7995]: Using default interface naming scheme 'v243'.
systemd-udevd[7994]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
systemd-udevd[7995]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
NetworkManager[640]: <info>  [1584349179.9267] manager: (virbr0-nic): new Tun device (/org/freedesktop/NetworkManager/Devices/11)
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --insert LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: No chain/target/match by that name.
libvirtd[733]: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --insert LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: No chain/target/match by that name.
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 --destination 224.0.0.0/24 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 --destination 255.255.255.255/32 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 -p tcp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 -p udp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: Bad rule (does a matching rule exist in that chain?).
NetworkManager[640]: <info>  [1584349179.9985] device (virbr0-nic): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
NetworkManager[640]: <info>  [1584349180.0011] device (virbr0-nic): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external')
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 ! --destination 192.168.122.0/24 --jump MASQUERADE' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWI --destination 192.168.122.0/24 --out-interface virbr0 --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWO --source 192.168.122.0/24 --in-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWX --in-interface virbr0 --out-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWI --out-interface virbr0 --jump REJECT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWO --in-interface virbr0 --jump REJECT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_INP --in-interface virbr0 --protocol udp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_OUT --out-interface virbr0 --protocol udp --destination-port 68 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_INP --in-interface virbr0 --protocol udp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
kernel: device virbr0-nic left promiscuous mode
kernel: virbr0: port 1(virbr0-nic) entered disabled state
kernel: audit: type=1700 audit(1584349180.080:531): dev=virbr0-nic prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
audit: ANOM_PROMISCUOUS dev=virbr0-nic prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
NetworkManager[640]: <info>  [1584349180.0930] device (virbr0-nic): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed')
NetworkManager[640]: <info>  [1584349180.0951] device (virbr0-nic): released from master device virbr0


Version-Release number of selected component (if applicable):

libvirt-daemon-5.6.0-5.fc31.x86_64
firewalld-0.7.3-1.fc31.noarch
iptables-1.8.3-7.fc31.x86_64

Also confirmed on current F32:

libvirt-daemon-6.1.0-1.fc32.x86_64
firewalld-0.8.1-1.fc32.noarch
iptables-nft-1.8.4-7.fc32.x86_64
nftables-0.9.3-2.fc32.x86_64


How reproducible: Always

Comment 1 Martin Pitt 2020-03-16 09:10:30 UTC

I think I found a workaround: When restarting libvirtd.service *after* starting firewalld, it works:

# systemctl start firewalld; systemctl try-restart libvirtd; virsh net-start default

(restarting it before starting firewalld it doesn't work). This is good enough for our integration tests.

Comment 2 Richard W.M. Jones 2020-04-14 08:36:13 UTC

Is firewalld segfaulting?  It could be duplicate of
https://bugzilla.redhat.com/show_bug.cgi?id=1810473

Comment 3 Martin Pitt 2020-04-15 06:38:03 UTC

@Richard: No, it's not crashing. The above journal dump is complete, and coredumpctl also does not show any crash. The reproducer above still works in current F31.

Comment 4 Laine Stump 2020-05-08 01:53:44 UTC

This is due to an optimization in libvirt's network driver. Because 1) attempting to create a private iptables chain when it already exists results in an error, and 2) repeatedly looking to see if a chain exists for every network that is started is very inefficient and makes for a slow startup time, the network driver only calls the function that creates the private chains once per libvirtd run. This was done based on the assumption that firewalld would keep all of the iptables chains created by libvirt intact, which is in fact not the case.

Instead we need to recreate the private chains any time we are notified (via dbus) that firewalld has restarted.

Comment 5 Laine Stump 2020-05-08 02:53:52 UTC

I just posted these two patches upstream:

https://www.redhat.com/archives/libvir-list/2020-May/msg00344.html

Comment 6 Laine Stump 2020-05-19 17:41:06 UTC

The following two patches will be in upstream libvirt-6.4.0. We should backport them to both the F31 and F32 builds of libvirt.

commit de110f110fb917a31b9f33ad8e4b3c1d3284766a
Author: Laine Stump <laine>
Date:   Thu May 7 22:32:59 2020 -0400

    network: make it safe to call networkSetupPrivateChains() multiple times

commit f5418b427e7d2f26803880309478de9103680826
Author: Laine Stump <laine>
Date:   Thu May 7 21:54:39 2020 -0400

    network: force re-creation of iptables private chains on firewalld restart

Comment 7 Fedora Update System 2020-05-28 12:58:46 UTC

FEDORA-2020-5cd83efda7 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-5cd83efda7

Comment 8 Fedora Update System 2020-05-29 03:03:09 UTC

FEDORA-2020-5cd83efda7 has been pushed to the Fedora 31 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-5cd83efda7`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-5cd83efda7

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 9 Fedora Update System 2020-06-16 01:18:33 UTC

libvirt-5.6.0-7.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.