Description of problem: When trying to launch a VM with vagrant after booting I get: "Error while activating network: Call to virNetworkCreate failed: internal error: Failed to apply firewall rules /usr/sbin/ip6tables -w --table filter --insert LIBVIRT_FWO --in-interface virbr2 --jump REJECT: ip6tables: No chain/target/match by that name." Version-Release number of selected component (if applicable): libvirt-5.1.0-4.fc30.x86_64 iptables-1.8.0-5.fc30.x86_64 iptables-services-1.8.0-5.fc30.x86_64 How reproducible: Every time I have rebooted so far Steps to Reproduce: 1. Install Fedora 30 2. Enable iptables, ip6tables, and libvirtd serviecs 3. Reboot 4. Try to start a VM with libvirt Actual results: Error message above. Expected results: VM starts Additional info: Once I manually sudo systemctl restart libvirtd everything works normally.
Thanks for the report. So I take it firewalld is not installed/enabled? Laine have you heard of anything similar?
Yes, installed, but not running. If enabling firewalld ends up being the fix I'm ok with that. I've been upgrading this system for years now and I've just been lazy to switch. rpm -q firewalld firewalld-0.6.3-2.fc30.noarch $ systemctl status firewalld iptables ip6tables ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: man:firewalld(1) ● iptables.service - IPv4 firewall with iptables Loaded: loaded (/usr/lib/systemd/system/iptables.service; enabled; vendor preset: disabled) Active: active (exited) since Tue 2019-04-09 11:19:37 EDT; 4h 11min ago Process: 1104 ExecStart=/usr/libexec/iptables/iptables.init start (code=exited, status=0/SUCCESS) Main PID: 1104 (code=exited, status=0/SUCCESS) Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable. ● ip6tables.service - IPv6 firewall with ip6tables Loaded: loaded (/usr/lib/systemd/system/ip6tables.service; enabled; vendor preset: disabled) Active: active (exited) since Tue 2019-04-09 11:19:36 EDT; 4h 11min ago Process: 1102 ExecStart=/usr/libexec/iptables/ip6tables.init start (code=exited, status=0/SUCCESS) Main PID: 1102 (code=exited, status=0/SUCCESS) Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
This is related to the new private chains that libvirt's network iptables rules are being put into. There have been several BZes filed due to missing IPv6 in the kernel causing these chains to not be added, but I haven't previously heard of one where the problem magically disappeared after restarting libvirtd. Are there any other error messages in the system log from during the initial startup of libvirt during boot? Also, there is this patch that is in 5.2.0, but not 5.1.0, and I don't know if Dan has backported it into F30: https://www.redhat.com/archives/libvir-list/2019-March/msg01218.html Not sure if that will have an effect on this specific situation or not.
I disabled ip6tables and rebooted and everything just worked. Started ip6tables and it broke. Restarted libvirtd and it worked. grep ExecStart /lib/systemd/system/ip6tables.service ExecStart=/usr/libexec/iptables/ip6tables.init start It looks like that script runs ip6tables-restore to restore the rules from /etc/sysconfig/ip6tables. I'm familiar with the sysconfig file, not the ip6tables-restore command, but given -t has a -n/--noflush option I'd guess by default it flushes everything before restoring by default. So far I don't see why this is happening with ip6tables and not iptables though. I don't see any Before/After type stuff called out for iptables in service files in /etc/systemd or /lib/systemd. I don't see anything different in where the services are linked to start (basic.target.wants) # ip6tables -nvL Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 LIBVIRT_INP all * * ::/0 ::/0 Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 LIBVIRT_FWX all * * ::/0 ::/0 0 0 LIBVIRT_FWI all * * ::/0 ::/0 0 0 LIBVIRT_FWO all * * ::/0 ::/0 Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 LIBVIRT_OUT all * * ::/0 ::/0 Chain LIBVIRT_FWI (1 references) pkts bytes target prot opt in out source destination 0 0 REJECT all * virbr2 ::/0 ::/0 reject-with icmp6-port-unreachable Chain LIBVIRT_FWO (1 references) pkts bytes target prot opt in out source destination 0 0 REJECT all virbr2 * ::/0 ::/0 reject-with icmp6-port-unreachable Chain LIBVIRT_FWX (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT all virbr2 virbr2 ::/0 ::/0 Chain LIBVIRT_INP (1 references) pkts bytes target prot opt in out source destination Chain LIBVIRT_OUT (1 references) pkts bytes target prot opt in out source destination # systemctl start ip6tables # ip6tables -nvL Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT all * * ::/0 ::/0 state RELATED,ESTABLISHED 0 0 ACCEPT icmpv6 * * ::/0 ::/0 0 0 ACCEPT all lo * ::/0 ::/0 0 0 ACCEPT tcp * * ::/0 ::/0 state NEW tcp dpt:22 0 0 ACCEPT udp * * ::/0 fe80::/64 udp dpt:546 state NEW 0 0 REJECT all * * ::/0 ::/0 reject-with icmp6-adm-prohibited Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 REJECT all * * ::/0 ::/0 reject-with icmp6-adm-prohibited Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination # systemctl restart libvirtd # ip6tables -nvL Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 LIBVIRT_INP all * * ::/0 ::/0 0 0 ACCEPT all * * ::/0 ::/0 state RELATED,ESTABLISHED 0 0 ACCEPT icmpv6 * * ::/0 ::/0 0 0 ACCEPT all lo * ::/0 ::/0 0 0 ACCEPT tcp * * ::/0 ::/0 state NEW tcp dpt:22 0 0 ACCEPT udp * * ::/0 fe80::/64 udp dpt:546 state NEW 0 0 REJECT all * * ::/0 ::/0 reject-with icmp6-adm-prohibited Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 LIBVIRT_FWX all * * ::/0 ::/0 0 0 LIBVIRT_FWI all * * ::/0 ::/0 0 0 LIBVIRT_FWO all * * ::/0 ::/0 0 0 REJECT all * * ::/0 ::/0 reject-with icmp6-adm-prohibited Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 LIBVIRT_OUT all * * ::/0 ::/0 Chain LIBVIRT_FWI (1 references) pkts bytes target prot opt in out source destination 0 0 REJECT all * virbr2 ::/0 ::/0 reject-with icmp6-port-unreachable Chain LIBVIRT_FWO (1 references) pkts bytes target prot opt in out source destination 0 0 REJECT all virbr2 * ::/0 ::/0 reject-with icmp6-port-unreachable Chain LIBVIRT_FWX (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT all virbr2 virbr2 ::/0 ::/0 Chain LIBVIRT_INP (1 references) pkts bytes target prot opt in out source destination Chain LIBVIRT_OUT (1 references) pkts bytes target prot opt in out source destination
It looks like it's just a race condition and not specific to ipv6. After a couple more reboots: Error while activating network: Call to virNetworkCreate failed: internal error: Failed to apply firewall rules /usr/sbin/iptables -w --table filter --insert LIBVIRT_INP --in-interface virbr2 --protocol tcp --destination-port 67 --jump ACCEPT: iptables: No chain/target/match by that name. . I guess this wasn't a problem in Fedora 29 beucase it was only adding rules to the FORWARD chain and other existing chains on the nat table?
I added these to the libvirtd service file and after a few reboots it looking consistently correct. After=firewalld.service After=iptables.service After=ip6tables.service I originally tried this to avoid listing out the two iptables services individually, but it didn't work as I expected. After=basic.target
Having those deps makes sense, as we definitely want any firewall setup to have been run before libvirtd starts.
This message is a reminder that Fedora 30 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '30'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 30 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Somehow nobody got around to posting a patch with the libvirtd.service change suggested/tested by Jason in Comment 6 :-(. Better late than never, though, I just pushed this upstream: commit 0756415f147dda15a417bd79eef9a62027d176e6 Author: Laine Stump <laine> Date: Fri May 1 00:05:50 2020 -0400 systemd: start libvirtd after firewalld/iptables services
FEDORA-2020-bce67bc520 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-bce67bc520
FEDORA-2020-bce67bc520 has been pushed to the Fedora 32 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-bce67bc520` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-bce67bc520 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-0fc9a55b49 has been pushed to the Fedora 32 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-0fc9a55b49` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-0fc9a55b49 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-0fc9a55b49 has been pushed to the Fedora 32 stable repository. If problem still persists, please make note of it in this bug report.