Bug 2322021

Summary: netavark leftover nftables rules make published container ports unreachable after container restart
Product: [Fedora] Fedora Reporter: Juan Orti Alcaine <jortialc>
Component: netavarkAssignee: Lokesh Mandvekar <lsm5>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 41CC: bbaude, dwalsh, jnovy, jorti, lsm5, mheon, tsweeney
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-03-03 13:26:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
netavark debug log none

Description Juan Orti Alcaine 2024-10-27 19:57:39 UTC 
After configuring the netavark firewall_driver to "nftables", I've noticed that the published ports of the containers become unreachable if the containers are restarted.

Reproducible: Always

Steps to Reproduce:
1. Configure netavark to use the nftables firewall driver in `/etc/containers/containers.conf.d/50-netavark-nftables.conf`:

~~~
[network]
firewall_driver="nftables"
~~~

2. Reboot the host
3. Run a container with some published ports. In this example I'm using port 53 TCP and UDP. The service is reachable.
4. Restart the container: the published ports are unreachable
Actual Results:  
after the container restart, new nftables rules are added, but the previous rules are not removed:

~~~
--- netavark-before.nft	2024-10-27 19:42:39.401751448 +0000
+++ netavark-after.nft	2024-10-27 19:43:13.056798909 +0000
@@ -37,6 +37,10 @@
 		tcp dport 53 jump nv_d9dc9108_10_89_0_0_nm24_dnat
 		udp dport 53 jump nv_d9dc9108_fdbd-32d0-22-7ad5--_nm64_dnat
 		tcp dport 53 jump nv_d9dc9108_fdbd-32d0-22-7ad5--_nm64_dnat
+		tcp dport 53 jump nv_d9dc9108_10_89_0_0_nm24_dnat
+		udp dport 53 jump nv_d9dc9108_10_89_0_0_nm24_dnat
+		udp dport 53 jump nv_d9dc9108_fdbd-32d0-22-7ad5--_nm64_dnat
+		tcp dport 53 jump nv_d9dc9108_fdbd-32d0-22-7ad5--_nm64_dnat
 	}
 
 	chain NETAVARK-HOSTPORT-SETMARK {
@@ -74,6 +78,12 @@
 		ip saddr 10.89.0.0/24 ip daddr 192.168.3.2 tcp dport 53 jump NETAVARK-HOSTPORT-SETMARK
 		ip saddr 127.0.0.1 ip daddr 192.168.3.2 tcp dport 53 jump NETAVARK-HOSTPORT-SETMARK
 		ip daddr 192.168.3.2 tcp dport 53 dnat ip to 10.89.0.5:53
+		ip saddr 10.89.0.0/24 ip daddr 192.168.3.2 tcp dport 53 jump NETAVARK-HOSTPORT-SETMARK
+		ip saddr 127.0.0.1 ip daddr 192.168.3.2 tcp dport 53 jump NETAVARK-HOSTPORT-SETMARK
+		ip daddr 192.168.3.2 tcp dport 53 dnat ip to 10.89.0.6:53
+		ip saddr 10.89.0.0/24 ip daddr 192.168.3.2 udp dport 53 jump NETAVARK-HOSTPORT-SETMARK
+		ip saddr 127.0.0.1 ip daddr 192.168.3.2 udp dport 53 jump NETAVARK-HOSTPORT-SETMARK
+		ip daddr 192.168.3.2 udp dport 53 dnat ip to 10.89.0.6:53
 	}
 
 	chain nv_d9dc9108_fdbd-32d0-22-7ad5--_nm64_dnat {
@@ -81,6 +91,10 @@
 		ip6 daddr fddc:f797:78ef:30::2 udp dport 53 dnat ip6 to [fdbd:32d0:22:7ad5::5]:53
 		ip6 saddr fdbd:32d0:22:7ad5::/64 ip6 daddr fddc:f797:78ef:30::2 tcp dport 53 jump NETAVARK-HOSTPORT-SETMARK
 		ip6 daddr fddc:f797:78ef:30::2 tcp dport 53 dnat ip6 to [fdbd:32d0:22:7ad5::5]:53
+		ip6 saddr fdbd:32d0:22:7ad5::/64 ip6 daddr fddc:f797:78ef:30::2 udp dport 53 jump NETAVARK-HOSTPORT-SETMARK
+		ip6 daddr fddc:f797:78ef:30::2 udp dport 53 dnat ip6 to [fdbd:32d0:22:7ad5::6]:53
+		ip6 saddr fdbd:32d0:22:7ad5::/64 ip6 daddr fddc:f797:78ef:30::2 tcp dport 53 jump NETAVARK-HOSTPORT-SETMARK
+		ip6 daddr fddc:f797:78ef:30::2 tcp dport 53 dnat ip6 to [fdbd:32d0:22:7ad5::6]:53
 	}
 
 	chain nv_d9dc9108_10_89_0_0_nm24 {
~~~

Expected Results:  
Ports reachable

~~~
# podman network inspect adguard-home
[
     {
          "name": "adguard-home",
          "id": "d9dc91084f02e5b0643205dff66f5bb65aa2e0c2ef18d11d818ecd31f51b1eca",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2024-10-27T17:25:08.460159391Z",
          "subnets": [
               {
                    "subnet": "10.89.0.0/24",
                    "gateway": "10.89.0.1"
               },
               {
                    "subnet": "fdbd:32d0:22:7ad5::/64",
                    "gateway": "fdbd:32d0:22:7ad5::1"
               }
          ],
          "ipv6_enabled": true,
          "internal": false,
          "dns_enabled": false,
          "options": {
               "isolate": "true"
          },
          "ipam_options": {
               "driver": "host-local"
          },
          "containers": {
               "9d8e0c3ff7bca3dab2d30a119269150b60fa5ac691cd0cccc19dd965802e0b63": {
                    "name": "adguard-home",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.89.0.6/24",
                                        "gateway": "10.89.0.1"
                                   },
                                   {
                                        "ipnet": "fdbd:32d0:22:7ad5::6/64",
                                        "gateway": "fdbd:32d0:22:7ad5::1"
                                   }
                              ],
                              "mac_address": "32:fa:cb:46:19:38"
                         }
                    }
               }
          }
     }
]
~~~
Comment 1 Juan Orti Alcaine 2024-10-28 08:24:24 UTC 
Created attachment 2054027 [details]
netavark debug log

Attaching debug logs of the container. Initially it's started and works as expected, then it's restarted and the ports are unreachable.
There's these errors when cleaning up the nftables rules:

~~~
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1268]: internal:0:0-0: Error: Could not process rule: No such file or directory
oct 28 08:11:51 adguard-home[1230]: time="2024-10-28T08:11:51Z" level=error msg="Unable to clean up network for container 8392cd6c7e271e27d1d7a434dcb3adaf94378518add0acd934dd28131c3860c1: \"netavark (exit code 1): nftables error: nft did not return successfully while applying ruleset\""
~~~
Comment 2 Juan Orti Alcaine 2024-10-28 09:05:50 UTC 
After more investigation, it looks the issue is caused by publishing the same port in UDP and TCP. There's already a bug in upstream about this:

https://github.com/containers/netavark/issues/1068

Reproduced in Fedora 41 with:

  netavark-1.12.2-1.fc41.x86_64
  podman-5.2.5-1.fc41.x86_64
  kernel-core-6.11.5-300.fc41.x86_64
  kernel-modules-core-6.11.5-300.fc41.x86_64


Reproducer:

/etc/containers/containers.conf.d/50-netavark-nftables.conf:

~~~
[network]
firewall_driver="nftables"
~~~

/etc/containers/systemd/nginx.network:

~~~
[Network]
NetworkName=nginx
~~~

/etc/containers/systemd/nginx.container:

~~~
[Container]
ContainerName=nginx
Image=docker.io/library/nginx:latest
Network=nginx.network
PublishPort=80:80/tcp
PublishPort=443:443/tcp
PublishPort=443:443/udp
GlobalArgs=--log-level=debug

[Install]
WantedBy=multi-user.target default.target
~~~

curl to port 80 doesn't work after 2nd container start.
Comment 3 Lokesh Mandvekar 2024-10-29 11:02:15 UTC 
Will be fixed in the next release of netavark expected upstream soon.
Comment 4 Lokesh Mandvekar 2025-03-03 13:26:12 UTC 
Closing. We recently had netavark v1.14.0. Please reopen if the issue persists.