Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: The following command is failing if executed on a machine with no global IPv6 address: # firewall-cmd --direct --passthrough ipv6 -I OUTPUT 1 -p udp --dport 53 -j ACCEPT Version-Release number of selected component (if applicable): firewalld-1.0.0-1.fc35.noarch How reproducible: Always with my vagrant box. Steps to Reproduce: Create a Vagrantfile with the following content: $script = <<-SCRIPT dnf update -y systemctl unmask firewalld systemctl enable firewalld --now SCRIPT Vagrant.configure("2") do |config| config.ssh.username = "root" config.vm.synced_folder "./", "/vagrant", type: "sshfs", sshfs_opts_append: "-o cache=no" config.vm.box = "freeipa/ci-master-frawhide" config.vm.box_version = "0.4.2" config.vm.provider "libvirt" do |domain, override| domain.cpus = 1 domain.memory = 2750 # Nested virtualization options domain.nested = true domain.cpu_mode = "host-passthrough" # Disable graphics domain.graphics_type = "none" domain.volume_cache = "unsafe" end config.vm.provision "shell", inline: $script end 1. vagrant up 2. vagrant ssh -c "firewall-cmd --direct --passthrough ipv6 -I OUTPUT 1 -p udp --dport 53 -j ACCEPT" Actual results: Error: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid backend or is unavailable Expected results: the command should succeed Additional info: - The vagrant box doesn't set any global IPv6 address (only link-local one): # ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:d7:dc:d6 brd ff:ff:ff:ff:ff:ff altname enp0s4 altname ens4 inet 192.168.122.25/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0 valid_lft 3123sec preferred_lft 3123sec inet6 fe80::3aae:6f43:da31:8b83/64 scope link noprefixroute valid_lft forever preferred_lft forever - with the previous version of firewalld (firewalld-0.9.3-6.fc35.noarch), the issue does not happen (replace the line dnf update -y with dnf update -y --excludepkg=firewalld in the vagrantfile). This issue is causing a test failure in FreeIPA nightly tests on rawhide (see for instance http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/563d5e66-f556-11eb-84e1-fa163e684033/report.html).
Can you verify the VM has a usable ip6tables? Running `ip6tables-save` should be enough.
The command does not produce any output but exits successfully: [root@fedora ~]# ip6tables-save [root@fedora ~]# echo $? 0
I can't reproduce this. Can you check /var/log/firewalld for other errors/warnings? Also check dmesg for any SELinux/AVC issues. --->8--- [root@vm-bos-fedora-1 ~]# cat /etc/os-release NAME="Fedora Linux" VERSION="35 (Rawhide Prerelease)" [root@vm-bos-fedora-1 ~]# dnf info firewalld Installed Packages Name : firewalld Version : 1.0.0 Release : 1.fc35 [root@vm-bos-fedora-1 ~]# ip addr [..] 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff altname enp0s3 inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute ens3 valid_lft 86229sec preferred_lft 86229sec [root@vm-bos-fedora-1 ~]# ip6tables -L OUTPUT Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@vm-bos-fedora-1 ~]# firewall-cmd --direct --passthrough ipv6 -I OUTPUT 1 -p udp --dport 53 -j ACCEPT success [root@vm-bos-fedora-1 ~]# ip6tables -L OUTPUT Chain OUTPUT (policy ACCEPT) target prot opt source destination ACCEPT udp anywhere anywhere udp dpt:domain
With "--debug 10" added to /etc/sysconfig/firewalld, I only see the following in /var/log/firewalld after the firewall-cmd command: 2021-08-09 16:46:11 DEBUG2: Introspect() 2021-08-09 16:46:11 DEBUG2: config.Introspect() 2021-08-09 16:46:11 DEBUG2: config.Introspect() 2021-08-09 16:46:11 DEBUG2: config.Introspect() 2021-08-09 16:46:11 DEBUG1: direct.passthrough('ipv6', '-I','OUTPUT','1','-p','udp','--dport','53','-j','ACCEPT') 2021-08-09 16:46:11 DEBUG2: INVALID_IPV: 'ipv6' is not a valid backend or is unavailable 2021-08-09 16:46:11 WARNING: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid backend or is unavailable But earlier logs show: 2021-08-09 15:53:36 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables -t security -L -n 2021-08-09 15:53:36 DEBUG1: ipv6 table 'security' does not exist (or not enough permission to check). 2021-08-09 15:53:36 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables -t raw -L -n 2021-08-09 15:53:36 DEBUG1: ipv6 table 'raw' does not exist (or not enough permission to check). 2021-08-09 15:53:36 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables -t mangle -L -n 2021-08-09 15:53:36 DEBUG1: ipv6 table 'mangle' does not exist (or not enough permission to check). 2021-08-09 15:53:36 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables -t nat -L -n 2021-08-09 15:53:36 DEBUG1: ipv6 table 'nat' does not exist (or not enough permission to check). 2021-08-09 15:53:36 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables -t filter -L -n 2021-08-09 15:53:36 DEBUG1: ipv6 table 'filter' does not exist (or not enough permission to check). 2021-08-09 15:53:36 ip6tables is not usable. The issue is always reproduced using my vagrant image, and the system is running in permissive mode but displays the following AVC when firewalld is started: ---- time->Mon Aug 9 15:53:36 2021 type=AVC msg=audit(1628524416.105:1470): avc: denied { setpcap } for pid=19607 comm="firewalld" capability=8 scontext=system_u:system_r:firewalld_t:s0 tcontext=system_u:system_r:firewalld_t:s0 tclass=capability permissive=1 ---- time->Mon Aug 9 15:53:36 2021 type=AVC msg=audit(1628524416.105:1471): avc: denied { setcap } for pid=19607 comm="firewalld" scontext=system_u:system_r:firewalld_t:s0 tcontext=system_u:system_r:firewalld_t:s0 tclass=process permissive=1
(In reply to Florence Blanc-Renaud from comment #4) [..] > But earlier logs show: > 2021-08-09 15:53:36 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: > /usr/sbin/ip6tables -t security -L -n > 2021-08-09 15:53:36 DEBUG1: ipv6 table 'security' does not exist (or not > enough permission to check). > 2021-08-09 15:53:36 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: > /usr/sbin/ip6tables -t raw -L -n > 2021-08-09 15:53:36 DEBUG1: ipv6 table 'raw' does not exist (or not enough > permission to check). > 2021-08-09 15:53:36 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: > /usr/sbin/ip6tables -t mangle -L -n > 2021-08-09 15:53:36 DEBUG1: ipv6 table 'mangle' does not exist (or not > enough permission to check). > 2021-08-09 15:53:36 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: > /usr/sbin/ip6tables -t nat -L -n > 2021-08-09 15:53:36 DEBUG1: ipv6 table 'nat' does not exist (or not enough > permission to check). > 2021-08-09 15:53:36 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: > /usr/sbin/ip6tables -t filter -L -n > 2021-08-09 15:53:36 DEBUG1: ipv6 table 'filter' does not exist (or not > enough permission to check). > 2021-08-09 15:53:36 ip6tables is not usable. That means when firewalld is calling ip6tables it's unusable. You tried `ip6tables-save` in comment 2. What about `ip6tables -L` ? Are you by chance using a custom kernel? > The issue is always reproduced using my vagrant image, and the system is > running in permissive mode but displays the following AVC when firewalld is > started: > ---- > time->Mon Aug 9 15:53:36 2021 > type=AVC msg=audit(1628524416.105:1470): avc: denied { setpcap } for > pid=19607 comm="firewalld" capability=8 > scontext=system_u:system_r:firewalld_t:s0 > tcontext=system_u:system_r:firewalld_t:s0 tclass=capability permissive=1 > ---- > time->Mon Aug 9 15:53:36 2021 > type=AVC msg=audit(1628524416.105:1471): avc: denied { setcap } for > pid=19607 comm="firewalld" scontext=system_u:system_r:firewalld_t:s0 > tcontext=system_u:system_r:firewalld_t:s0 tclass=process permissive=1 These AVC warning are due to bug 1985494.
(In reply to Eric Garver from comment #5) > That means when firewalld is calling ip6tables it's unusable. You tried > `ip6tables-save` in comment 2. What about `ip6tables -L` ? [root@fedora ~]# ip6tables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@fedora ~]# > > Are you by chance using a custom kernel? No, it's a vagrant image built with the standard kernel.
This bug appears to have been reported against 'rawhide' during the Fedora 35 development cycle. Changing version to 35.
With some effort I was able to reproduce this with you Vagrantfile. When firewalld starts it probes iptables/ip6tables for the supported wait options. This can be seen in the log: 2021-08-11 13:25:26 DEBUG2: <class 'firewall.core.ipXtables.ip4tables'>: /usr/sbin/iptables will be using -w10 option. 2021-08-11 13:25:26 DEBUG2: <class 'firewall.core.ipXtables.ip4tables'>: /usr/sbin/iptables-restore will be using -w option. 2021-08-11 13:25:26 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables-restore will be using -w option. 2021-08-11 13:25:26 DEBUG2: <class 'firewall.core.ebtables.ebtables'>: /usr/sbin/ebtables-restore /run/firewalld/temp._r_kpkve: 0 The absence of `ip6tables` means none of the wait options were detected. Later on we see basic ip6tables command fail: 2021-08-11 13:25:29 DEBUG2: <class 'firewall.core.ipXtables.ip4tables'>: /usr/sbin/iptables -w10 -t security -L -n 2021-08-11 13:25:29 DEBUG2: <class 'firewall.core.ipXtables.ip4tables'>: /usr/sbin/iptables -w10 -t raw -L -n 2021-08-11 13:25:29 DEBUG2: <class 'firewall.core.ipXtables.ip4tables'>: /usr/sbin/iptables -w10 -t mangle -L -n 2021-08-11 13:25:29 DEBUG2: <class 'firewall.core.ipXtables.ip4tables'>: /usr/sbin/iptables -w10 -t nat -L -n 2021-08-11 13:25:29 DEBUG2: <class 'firewall.core.ipXtables.ip4tables'>: /usr/sbin/iptables -w10 -t filter -L -n 2021-08-11 13:25:29 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables -t security -L -n 2021-08-11 13:25:29 DEBUG1: ipv6 table 'security' does not exist (or not enough permission to check). 2021-08-11 13:25:29 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables -t raw -L -n 2021-08-11 13:25:29 DEBUG1: ipv6 table 'raw' does not exist (or not enough permission to check). 2021-08-11 13:25:29 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables -t mangle -L -n 2021-08-11 13:25:29 DEBUG1: ipv6 table 'mangle' does not exist (or not enough permission to check). 2021-08-11 13:25:29 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables -t nat -L -n 2021-08-11 13:25:29 DEBUG1: ipv6 table 'nat' does not exist (or not enough permission to check). 2021-08-11 13:25:29 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables -t filter -L -n 2021-08-11 13:25:29 DEBUG1: ipv6 table 'filter' does not exist (or not enough permission to check). 2021-08-11 13:25:29 ip6tables is not usable. Note how `iptables` (ipv4) is okay. Likely because -w is in use. Restarting firewalld worked for me. I guess during the restart ip6tables worked correctly when probed. We can see this because the wait probe works as expected. 2021-08-11 13:33:07 DEBUG2: <class 'firewall.core.ipXtables.ip4tables'>: /usr/sbin/iptables will be using -w10 option. 2021-08-11 13:33:07 DEBUG2: <class 'firewall.core.ipXtables.ip4tables'>: /usr/sbin/iptables-restore will be using -w option. 2021-08-11 13:33:07 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables will be using -w10 option. 2021-08-11 13:33:07 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables-restore will be using -w option. 2021-08-11 13:33:07 DEBUG2: <class 'firewall.core.ebtables.ebtables'>: /usr/sbin/ebtables-restore /run/firewalld/temp.fkf01fzc: 0 See how `ip6tables` is present here. ## What's the cause? I don't know yet. At the moment it looks like an issue with ip6tables. The intermittent failure makes me wonder if it's related to bug 1986588 comment 7. The wait probe uses args `-w -L -n`. @psutter, any ideas?
Update to comment 8. The vagrant image is using the -legacy variants. As such it's not a surprise that ip6tables fails without the -w. Any contention on the xtables lock will cause the command to fail. [root@fedora ~]# ip6tables --version ip6tables v1.8.7 (legacy) e.g. [root@fedora ~]# while true; do ip6tables -L -n >/dev/null || break; done Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
Eric, thanks for your investigation. The vagrant image was built based on Fedora-Rawhide-20210620.n.0 (according to https://app.vagrantup.com/freeipa/boxes/ci-master-frawhide) and I don't think we modify the ip6tables version. If we rebuild the vagrant image with a newer compose, do you think the issue should be fixed (i.e. will iptables use the nf_tables version)?
(In reply to Florence Blanc-Renaud from comment #10) > If we rebuild the vagrant image with a newer compose, do you think the issue > should be fixed (i.e. will iptables use the nf_tables version)? Yes and No. Yes. Your CI may pass as expected with the -nft variant. IIRC, the -nft variant ignores "-w" as the nftables kernel API guarantees atomicity. So no userspace locking is needed. No, because Fedora has alternatives support for iptables. It should work with both variants. Let me investigate more. --- Interesting that the VM uses -nft variants for ebtables and arptables, but -legacy for iptables. [root@fedora ~]# alternatives --list |grep table ebtables auto /usr/sbin/ebtables-nft iptables auto /usr/sbin/iptables-legacy arptables auto /usr/sbin/arptables-nft
Firewalld's interaction with ip6tables fails initially, but after invoking `ip6tables` on the command line it works fine. I've tried this many times and the behavior is always the same. 1. boot up 2. this fails: firewall-cmd --direct --passthrough ipv6 -I OUTPUT 1 -p udp --dport 53 -j ACCEPT 3. restart firewalld 4. this fails: firewall-cmd --direct --passthrough ipv6 -I OUTPUT 1 -p udp --dport 53 -j ACCEPT 5. run command: ip6tables -w -L -n 6. restart firewalld (recall that the probe happens when firewalld starts) 7. this WORKS: firewall-cmd --direct --passthrough ipv6 -I OUTPUT 1 -p udp --dport 53 -j ACCEPT --->8--- [root@vm-local-fedora tmp]# vagrant ssh ==> default: Libvirt Provider: volume_cache is deprecated. Use disk_driver :cache => 'unsafe' instead. Last login: Wed Aug 11 17:13:09 2021 from 192.168.122.1 [root@fedora ~]# ls -al /var/run/xtables.lock -rw-------. 1 root root 0 Aug 11 17:16 /var/run/xtables.lock [root@fedora ~]# firewall-cmd --direct --passthrough ipv6 -I OUTPUT 1 -p udp --dport 53 -j ACCEPT Error: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid backend or is unavailable [root@fedora ~]# systemctl restart firewalld [root@fedora ~]# firewall-cmd --direct --passthrough ipv6 -I OUTPUT 1 -p udp --dport 53 -j ACCEPT Error: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid backend or is unavailable [root@fedora ~]# systemctl restart firewalld [root@fedora ~]# firewall-cmd --direct --passthrough ipv6 -I OUTPUT 1 -p udp --dport 53 -j ACCEPT Error: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid backend or is unavailable [root@fedora ~]# ip6tables -w -L -n Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@fedora ~]# ls -al /var/run/xtables.lock -rw-------. 1 root root 0 Aug 11 17:16 /var/run/xtables.lock [root@fedora ~]# firewall-cmd --direct --passthrough ipv6 -I OUTPUT 1 -p udp --dport 53 -j ACCEPT Error: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid backend or is unavailable [root@fedora ~]# systemctl restart firewalld [root@fedora ~]# firewall-cmd --direct --passthrough ipv6 -I OUTPUT 1 -p udp --dport 53 -j ACCEPT success
Adding some debug to the code reveals the problem. The implicit module load is being blocked. This is interesting because SELinux is set to permissive. [root@fedora ~]# getenforce Permissive --->8--- 2021-08-11 17:41:16 DEBUG2: <class 'firewall.core.ipXtables.ip6tables'>: /usr/sbin/ip6tables probe (-w -L -n): ret = 3, output = "modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted ip6tables v1.8.7 (legacy): can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?) Perhaps ip6tables or your kernel needs to be upgraded. "
Solved. firewalld v1.0.0 added support for dropping linux capabilities. However, it dropped CAP_SYS_MODULE which is needed to load modules. The ip6tables userspace will attempt to modprobe the kernel modules. Fix is simple. I'll probably cut a new upstream release and update Fedora ASAP. Thanks for being patient.
Upstream PR: https://github.com/firewalld/firewalld/pull/836
New build started for package update: https://koji.fedoraproject.org/koji/taskinfo?taskID=73738461 We're not yet at bodhi activation yet so the update should propagate quickly. However, we're in post-branch freeze so it may take a couple days. As such, closing this bug with NEXTRELEASE since the bug is against f35 and not rawhide. You can grab the build directly from koji once it's complete.