After configuring the netavark firewall_driver to "nftables", I've noticed that the published ports of the containers become unreachable if the containers are restarted. Reproducible: Always Steps to Reproduce: 1. Configure netavark to use the nftables firewall driver in `/etc/containers/containers.conf.d/50-netavark-nftables.conf`: ~~~ [network] firewall_driver="nftables" ~~~ 2. Reboot the host 3. Run a container with some published ports. In this example I'm using port 53 TCP and UDP. The service is reachable. 4. Restart the container: the published ports are unreachable Actual Results: after the container restart, new nftables rules are added, but the previous rules are not removed: ~~~ --- netavark-before.nft 2024-10-27 19:42:39.401751448 +0000 +++ netavark-after.nft 2024-10-27 19:43:13.056798909 +0000 @@ -37,6 +37,10 @@ tcp dport 53 jump nv_d9dc9108_10_89_0_0_nm24_dnat udp dport 53 jump nv_d9dc9108_fdbd-32d0-22-7ad5--_nm64_dnat tcp dport 53 jump nv_d9dc9108_fdbd-32d0-22-7ad5--_nm64_dnat + tcp dport 53 jump nv_d9dc9108_10_89_0_0_nm24_dnat + udp dport 53 jump nv_d9dc9108_10_89_0_0_nm24_dnat + udp dport 53 jump nv_d9dc9108_fdbd-32d0-22-7ad5--_nm64_dnat + tcp dport 53 jump nv_d9dc9108_fdbd-32d0-22-7ad5--_nm64_dnat } chain NETAVARK-HOSTPORT-SETMARK { @@ -74,6 +78,12 @@ ip saddr 10.89.0.0/24 ip daddr 192.168.3.2 tcp dport 53 jump NETAVARK-HOSTPORT-SETMARK ip saddr 127.0.0.1 ip daddr 192.168.3.2 tcp dport 53 jump NETAVARK-HOSTPORT-SETMARK ip daddr 192.168.3.2 tcp dport 53 dnat ip to 10.89.0.5:53 + ip saddr 10.89.0.0/24 ip daddr 192.168.3.2 tcp dport 53 jump NETAVARK-HOSTPORT-SETMARK + ip saddr 127.0.0.1 ip daddr 192.168.3.2 tcp dport 53 jump NETAVARK-HOSTPORT-SETMARK + ip daddr 192.168.3.2 tcp dport 53 dnat ip to 10.89.0.6:53 + ip saddr 10.89.0.0/24 ip daddr 192.168.3.2 udp dport 53 jump NETAVARK-HOSTPORT-SETMARK + ip saddr 127.0.0.1 ip daddr 192.168.3.2 udp dport 53 jump NETAVARK-HOSTPORT-SETMARK + ip daddr 192.168.3.2 udp dport 53 dnat ip to 10.89.0.6:53 } chain nv_d9dc9108_fdbd-32d0-22-7ad5--_nm64_dnat { @@ -81,6 +91,10 @@ ip6 daddr fddc:f797:78ef:30::2 udp dport 53 dnat ip6 to [fdbd:32d0:22:7ad5::5]:53 ip6 saddr fdbd:32d0:22:7ad5::/64 ip6 daddr fddc:f797:78ef:30::2 tcp dport 53 jump NETAVARK-HOSTPORT-SETMARK ip6 daddr fddc:f797:78ef:30::2 tcp dport 53 dnat ip6 to [fdbd:32d0:22:7ad5::5]:53 + ip6 saddr fdbd:32d0:22:7ad5::/64 ip6 daddr fddc:f797:78ef:30::2 udp dport 53 jump NETAVARK-HOSTPORT-SETMARK + ip6 daddr fddc:f797:78ef:30::2 udp dport 53 dnat ip6 to [fdbd:32d0:22:7ad5::6]:53 + ip6 saddr fdbd:32d0:22:7ad5::/64 ip6 daddr fddc:f797:78ef:30::2 tcp dport 53 jump NETAVARK-HOSTPORT-SETMARK + ip6 daddr fddc:f797:78ef:30::2 tcp dport 53 dnat ip6 to [fdbd:32d0:22:7ad5::6]:53 } chain nv_d9dc9108_10_89_0_0_nm24 { ~~~ Expected Results: Ports reachable ~~~ # podman network inspect adguard-home [ { "name": "adguard-home", "id": "d9dc91084f02e5b0643205dff66f5bb65aa2e0c2ef18d11d818ecd31f51b1eca", "driver": "bridge", "network_interface": "podman1", "created": "2024-10-27T17:25:08.460159391Z", "subnets": [ { "subnet": "10.89.0.0/24", "gateway": "10.89.0.1" }, { "subnet": "fdbd:32d0:22:7ad5::/64", "gateway": "fdbd:32d0:22:7ad5::1" } ], "ipv6_enabled": true, "internal": false, "dns_enabled": false, "options": { "isolate": "true" }, "ipam_options": { "driver": "host-local" }, "containers": { "9d8e0c3ff7bca3dab2d30a119269150b60fa5ac691cd0cccc19dd965802e0b63": { "name": "adguard-home", "interfaces": { "eth0": { "subnets": [ { "ipnet": "10.89.0.6/24", "gateway": "10.89.0.1" }, { "ipnet": "fdbd:32d0:22:7ad5::6/64", "gateway": "fdbd:32d0:22:7ad5::1" } ], "mac_address": "32:fa:cb:46:19:38" } } } } } ] ~~~
Created attachment 2054027 [details] netavark debug log Attaching debug logs of the container. Initially it's started and works as expected, then it's restarted and the ports are unreachable. There's these errors when cleaning up the nftables rules: ~~~ oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory oct 28 08:11:51 adguard-home[1230]: time="2024-10-28T08:11:51Z" level=error msg="Unable to clean up network for container 8392cd6c7e271e27d1d7a434dcb3adaf94378518add0acd934dd28131c3860c1: \"netavark (exit code 1): nftables error: nft did not return successfully while applying ruleset\"" ~~~
After more investigation, it looks the issue is caused by publishing the same port in UDP and TCP. There's already a bug in upstream about this: https://github.com/containers/netavark/issues/1068 Reproduced in Fedora 41 with: netavark-1.12.2-1.fc41.x86_64 podman-5.2.5-1.fc41.x86_64 kernel-core-6.11.5-300.fc41.x86_64 kernel-modules-core-6.11.5-300.fc41.x86_64 Reproducer: /etc/containers/containers.conf.d/50-netavark-nftables.conf: ~~~ [network] firewall_driver="nftables" ~~~ /etc/containers/systemd/nginx.network: ~~~ [Network] NetworkName=nginx ~~~ /etc/containers/systemd/nginx.container: ~~~ [Container] ContainerName=nginx Image=docker.io/library/nginx:latest Network=nginx.network PublishPort=80:80/tcp PublishPort=443:443/tcp PublishPort=443:443/udp GlobalArgs=--log-level=debug [Install] WantedBy=multi-user.target default.target ~~~ curl to port 80 doesn't work after 2nd container start.
Will be fixed in the next release of netavark expected upstream soon.
Closing. We recently had netavark v1.14.0. Please reopen if the issue persists.