Bug 2321943

I have several containers started with quadlet as root. A few days ago I discovered that I can no longer start most of my containers with the following netavark messages:

~~~
oct 26 20:31:39 systemd[1]: Starting nextcloud.service - Nextcloud...
oct 26 20:31:40 podman[5999]: 2024-10-26 20:31:40.255709251 +0200 CEST m=+0.380560182 image pull 0d4b47b97bf0b1c1637a6058dd084e042ef26e6a74f23684a45b565c1c6fbd43 docker.io/library/nextcloud:30
oct 26 20:31:44 podman[5999]: 2024-10-26 20:31:44.661588204 +0200 CEST m=+4.786439111 container create 2c79bc96e4e7ae39117764a7be4693d149d1a52508524d5df4678f04e756dbff (image=docker.io/library/nextcloud:30, name=nextcloud, traefik.http.middlewares.nextcloud-redirect-well-known.redirectregex.permanent=true, traefik.ht>
oct 26 20:31:53 podman[5999]: 2024-10-26 20:31:53.822017855 +0200 CEST m=+13.946868766 container init 2c79bc96e4e7ae39117764a7be4693d149d1a52508524d5df4678f04e756dbff (image=docker.io/library/nextcloud:30, name=nextcloud, traefik.http.routers.nextcloud.rule=Host(`REDACTED`), traefik.http.middlewares.nextcloud-r>
oct 26 20:31:56 nextcloud[5999]: time="2024-10-26T20:31:56+02:00" level=error msg="IPAM error: failed to get ips for container ID 2c79bc96e4e7ae39117764a7be4693d149d1a52508524d5df4678f04e756dbff on network nextcloud"
oct 26 20:31:56 nextcloud[13412]: [ERROR netavark::network::bridge] failed to parse ipam options: no static ips provided
oct 26 20:31:56 nextcloud[5999]: time="2024-10-26T20:31:56+02:00" level=error msg="IPAM error: failed to find ip for subnet 10.89.15.0/24 on network nextcloud"
oct 26 20:31:56 nextcloud[5999]: time="2024-10-26T20:31:56+02:00" level=error msg="Unable to clean up network for container 2c79bc96e4e7ae39117764a7be4693d149d1a52508524d5df4678f04e756dbff: \"netavark: netavark encountered multiple errors:\\n\\t- remove aardvark entries: IO error: No such file or directory (os error >
oct 26 20:31:56 podman[13662]: 2024-10-26 20:31:56.574373638 +0200 CEST m=+0.064938863 container remove 2c79bc96e4e7ae39117764a7be4693d149d1a52508524d5df4678f04e756dbff (image=docker.io/library/nextcloud:30, name=nextcloud, traefik.http.middlewares.nextcloud-redirect-well-known.redirectregex.replacement=https://cloud>
oct 26 20:31:56 nextcloud[5999]: Error: netavark: unable to append rule '-j MARK  --set-xmark 0x2000/0x2000' to table 'nat': code: 2, msg: Warning: Extension MARK revision 0 not supported, missing kernel module?
oct 26 20:31:56 nextcloud[5999]: ip6tables v1.8.10 (nf_tables): unknown option "--set-xmark"
oct 26 20:31:56 nextcloud[5999]: Try `ip6tables -h' or 'ip6tables --help' for more information.
oct 26 20:31:56 systemd[1]: nextcloud.service: Main process exited, code=exited, status=126/n/a
oct 26 20:31:56 systemd[1]: nextcloud.service: Failed with result 'exit-code'.
oct 26 20:31:56 systemd[1]: Failed to start nextcloud.service - Nextcloud.
~~~

The container is using this network:

~~~
[
     {
          "name": "nextcloud",
          "id": "5f159c6dbe3fe36ca41e0a593377110611568700ce2fcccfed25dccce17ed9c0",
          "driver": "bridge",
          "network_interface": "podman16",
          "created": "2024-10-16T20:44:38.663060312+02:00",
          "subnets": [
               {
                    "subnet": "10.89.15.0/24",
                    "gateway": "10.89.15.1"
               },
               {
                    "subnet": "fdc5:a811:9e5a:d82d::/64",
                    "gateway": "fdc5:a811:9e5a:d82d::1"
               }
          ],
          "ipv6_enabled": true,
          "internal": false,
          "dns_enabled": true,
          "options": {
               "isolate": "true"
          },
          "ipam_options": {
               "driver": "host-local"
          },
          "containers": {}
      }
]
~~~

Surprisingly some containers that are connected to internal networks did actually start.

Versions used:

netavark-1.12.2-1.fc40.x86_64
podman-5.2.3-1.fc40.x86_64
crun-1.17-1.fc40.x86_64
kernel-6.11.4-201.fc40.x86_64

Reproducible: Always

Steps to Reproduce:
1. As root, create a quadlet file to start a container conencted to a non-internal network.
2. Start the container

Actual Results:  
Container doesn't start. I get this error:

Error: netavark: unable to append rule '-j MARK  --set-xmark 0x2000/0x2000' to table 'nat': code: 2, msg: Warning: Extension MARK revision 0 not supported, missing kernel module?

Expected Results:  
Container starts

Creating the file `/etc/containers/containers.conf.d/50-netavark-nftables.conf` with the content below and rebooting fixes the issue:

~~~
[network]
firewall_driver="nftables"
~~~