Description of problem: In a stock Fedora 31 or 32 one often cannot restart libvirt's "default" network (virbr0). Right after booting, in a standard install, the default network is active and registered as a zone in firewalld: # virsh net-list --all; firewall-cmd --get-active-zones Name State Autostart Persistent -------------------------------------------- default active yes yes libvirt interfaces: virbr0 public interfaces: eth0 While firewalld stays running, I can `virsh net-destroy default` and `virsh net-start default`, and everything works. I can also `systemctl stop firewalld` and `systemctl start firewalld`, and everything is still in order. But I can't do both together: # virsh net-destroy default; systemctl stop firewalld # virsh net-list --all; firewall-cmd --get-active-zones Name State Autostart Persistent ---------------------------------------------- default inactive yes yes FirewallD is not running # systemctl start firewalld; virsh net-start default error: Failed to start network default error: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --insert LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: No chain/target/match by that name. This isn't a race condition -- you can wait between starting firewalld and the default network, or retry the latter, always the same result. There are a lot of errors in the journal during `virsh net-start default`: NetworkManager[640]: <info> [1584349179.8789] manager: (virbr0): new Bridge device (/org/freedesktop/NetworkManager/Devices/10) kernel: virbr0: port 1(virbr0-nic) entered blocking state kernel: virbr0: port 1(virbr0-nic) entered disabled state kernel: device virbr0-nic entered promiscuous mode kernel: kauditd_printk_skb: 8 callbacks suppressed kernel: audit: type=1700 audit(1584349179.887:530): dev=virbr0-nic prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295 audit: ANOM_PROMISCUOUS dev=virbr0-nic prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295 systemd-udevd[7994]: Using default interface naming scheme 'v243'. systemd-udevd[7995]: Using default interface naming scheme 'v243'. systemd-udevd[7994]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. systemd-udevd[7995]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. NetworkManager[640]: <info> [1584349179.9267] manager: (virbr0-nic): new Tun device (/org/freedesktop/NetworkManager/Devices/11) firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --insert LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: No chain/target/match by that name. libvirtd[733]: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --insert LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: No chain/target/match by that name. firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 --destination 224.0.0.0/24 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?). firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 --destination 255.255.255.255/32 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?). firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 -p tcp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: Bad rule (does a matching rule exist in that chain?). firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 -p udp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: Bad rule (does a matching rule exist in that chain?). NetworkManager[640]: <info> [1584349179.9985] device (virbr0-nic): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external') NetworkManager[640]: <info> [1584349180.0011] device (virbr0-nic): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external') firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table nat --delete LIBVIRT_PRT --source 192.168.122.0/24 ! --destination 192.168.122.0/24 --jump MASQUERADE' failed: iptables: Bad rule (does a matching rule exist in that chain?). firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWI --destination 192.168.122.0/24 --out-interface virbr0 --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWO --source 192.168.122.0/24 --in-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWX --in-interface virbr0 --out-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWI --out-interface virbr0 --jump REJECT' failed: iptables: Bad rule (does a matching rule exist in that chain?). firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_FWO --in-interface virbr0 --jump REJECT' failed: iptables: Bad rule (does a matching rule exist in that chain?). firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_INP --in-interface virbr0 --protocol udp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_OUT --out-interface virbr0 --protocol udp --destination-port 68 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_INP --in-interface virbr0 --protocol udp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). firewalld[7706]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --delete LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). kernel: device virbr0-nic left promiscuous mode kernel: virbr0: port 1(virbr0-nic) entered disabled state kernel: audit: type=1700 audit(1584349180.080:531): dev=virbr0-nic prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295 audit: ANOM_PROMISCUOUS dev=virbr0-nic prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295 NetworkManager[640]: <info> [1584349180.0930] device (virbr0-nic): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') NetworkManager[640]: <info> [1584349180.0951] device (virbr0-nic): released from master device virbr0 Version-Release number of selected component (if applicable): libvirt-daemon-5.6.0-5.fc31.x86_64 firewalld-0.7.3-1.fc31.noarch iptables-1.8.3-7.fc31.x86_64 Also confirmed on current F32: libvirt-daemon-6.1.0-1.fc32.x86_64 firewalld-0.8.1-1.fc32.noarch iptables-nft-1.8.4-7.fc32.x86_64 nftables-0.9.3-2.fc32.x86_64 How reproducible: Always
I think I found a workaround: When restarting libvirtd.service *after* starting firewalld, it works: # systemctl start firewalld; systemctl try-restart libvirtd; virsh net-start default (restarting it before starting firewalld it doesn't work). This is good enough for our integration tests.
Is firewalld segfaulting? It could be duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1810473
@Richard: No, it's not crashing. The above journal dump is complete, and coredumpctl also does not show any crash. The reproducer above still works in current F31.
This is due to an optimization in libvirt's network driver. Because 1) attempting to create a private iptables chain when it already exists results in an error, and 2) repeatedly looking to see if a chain exists for every network that is started is very inefficient and makes for a slow startup time, the network driver only calls the function that creates the private chains once per libvirtd run. This was done based on the assumption that firewalld would keep all of the iptables chains created by libvirt intact, which is in fact not the case. Instead we need to recreate the private chains any time we are notified (via dbus) that firewalld has restarted.
I just posted these two patches upstream: https://www.redhat.com/archives/libvir-list/2020-May/msg00344.html
The following two patches will be in upstream libvirt-6.4.0. We should backport them to both the F31 and F32 builds of libvirt. commit de110f110fb917a31b9f33ad8e4b3c1d3284766a Author: Laine Stump <laine> Date: Thu May 7 22:32:59 2020 -0400 network: make it safe to call networkSetupPrivateChains() multiple times commit f5418b427e7d2f26803880309478de9103680826 Author: Laine Stump <laine> Date: Thu May 7 21:54:39 2020 -0400 network: force re-creation of iptables private chains on firewalld restart
FEDORA-2020-5cd83efda7 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-5cd83efda7
FEDORA-2020-5cd83efda7 has been pushed to the Fedora 31 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-5cd83efda7` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-5cd83efda7 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
libvirt-5.6.0-7.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.