This is also described in the following Debian bug: 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970267

(In reply to Andreas Karis from comment #0)
> This happens when injecting an iptables rule with -m state, then saving a
> ruleset with `nft list ruleset` and dumping it to a file, then reloading
> from the file with `nft -f`. iptables will complain that `table `filter` is
> incompatible.

That's a sure way to kill one's ruleset: iptables-nft uses xtables extensions
for most matches/targets. nft might be able to translate them using libxtables
but that's not guaranteed. In doubt it merely prints the extension's name in a
comment at rule end, so restoring such a ruleset would just drop the respective
matches/targets.

In this specific case, a translation is available. So restoring the ruleset
turns the xtables extension into a native nftables expression which
iptables-nft in turn isn't able to interpret correctly.

Is this just a case of wrong usage or are you relying upon this behaviour
somehow? In general, one should treat iptables-nft like it wasn't related to
nftables and use iptables-nft-save etc. Mixing the tools is rarely sane.

Further clarification (hopefully) in the private comments above.

What I don't understand is this:
~~~
[root@rhel8 ~]# nft flush ruleset
[root@rhel8 ~]# iptables -t filter -I FORWARD -o virbr0 --dst 192.168.122.0/24 -m state --state ESTABLISHED,RELATED -j ACCEPT
[root@rhel8 ~]# nft list ruleset
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
		oifname "virbr0" @nh,128,24 12626042 ct state related,established counter packets 0 bytes 0 accept
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}
}
~~~

Hence, the rule translates cleanly to:
~~~
table ip filter {
	chain FORWARD {
		oifname "virbr0" @nh,128,24 12626042 ct state related,established counter packets 0 bytes 0 accept
	}
}
~~~

And it can be shown:
~~~
[root@rhel8 ~]# iptables -t filter -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             192.168.122.0/24     state RELATED,ESTABLISHED

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination 
~~~

Why does this break if I insert the rule with nft, even though the `nft list ruleset` looks exactly the same?
~~~
[root@rhel8 ~]# nft flush ruleset
[root@rhel8 ~]# iptables -t filter -L FORWARD
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
[root@rhel8 ~]# nft flush ruleset
[root@rhel8 ~]# nft list ruleset
[root@rhel8 ~]# iptables -t filter -L FORWARD
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
[root@rhel8 ~]# nft list ruleset
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}
}
[root@rhel8 ~]# iptables -t filter -L FORWARD
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
[root@rhel8 ~]# nft add rule ip filter FORWARD oifname "virbr0" @nh,128,24 12626042 ct state related,established counter packets 0 bytes 0 accept
[root@rhel8 ~]# nft list ruleset
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
		oifname "virbr0" @nh,128,24 12626042 ct state established,related counter packets 0 bytes 0 accept
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}
}
[root@rhel8 ~]# iptables -t filter -L FORWARD
iptables v1.8.4 (nf_tables): chain `FORWARD' in table `filter' is incompatible, use 'nft' tool.
~~~

Something there must be happening under the hood which is invisible to the end user. Because if I compare that to the earlier output, `nft list ruleset` is exactly the same in both cases, but iptables can translate the rules in the first case, but not in the latter case.

The same if I leave out the @nh... part:

[root@rhel8 ~]# nft flush ruleset
[root@rhel8 ~]# iptables -t filter -L FORWARD
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
[root@rhel8 ~]#  nft list ruleset
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}
}
[root@rhel8 ~]# iptables -t filter -L FORWARD
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
[root@rhel8 ~]# nft list ruleset
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}
}
[root@rhel8 ~]# iptables -t filter -L FORWARD
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
[root@rhel8 ~]#  nft add rule ip filter FORWARD oifname "virbr0" ct state related,established counter packets 0 bytes 0 accept
[root@rhel8 ~]# nft list ruleset
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
		oifname "virbr0" ct state established,related counter packets 0 bytes 0 accept
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}
}
[root@rhel8 ~]# iptables -t filter -L FORWARD
iptables v1.8.4 (nf_tables): chain `FORWARD' in table `filter' is incompatible, use 'nft' tool.

(In reply to Andreas Karis from comment #5)
> Further clarification (hopefully) in the private comments above.
> 
> What I don't understand is this:
> ~~~
> [root@rhel8 ~]# nft flush ruleset
> [root@rhel8 ~]# iptables -t filter -I FORWARD -o virbr0 --dst
> 192.168.122.0/24 -m state --state ESTABLISHED,RELATED -j ACCEPT
> [root@rhel8 ~]# nft list ruleset
> table ip filter {
> 	chain INPUT {
> 		type filter hook input priority filter; policy accept;
> 	}
> 
> 	chain FORWARD {
> 		type filter hook forward priority filter; policy accept;
> 		oifname "virbr0" @nh,128,24 12626042 ct state related,established counter
> packets 0 bytes 0 accept

Here we see an older nft tool that doesn't understand iptables-nft's optimized
prefix match yet. Hence why you see '@nh,128,24' (for "look at network header
offset 128bits length 24 bits").

The reasone why it breaks iptables after restoring with nft is because the 'ct
state' match you see above is actually a compat expression calling xt_CONNTRACK
extension. nft has xtables support built in and uses libxtables' xlate callback
to translate the extension into native nftables syntax. So effectively it
prints '-m state --state ESTABLISHED,RELATED' as 'ct state related,established'
although it is not the same (technically). When you restore the output, nft
creates a "real" nftables ct state expression. And iptables-nft can't handle
that, therefore the error message.

> Something there must be happening under the hood which is invisible to the
> end user. Because if I compare that to the earlier output, `nft list
> ruleset` is exactly the same in both cases, but iptables can translate the
> rules in the first case, but not in the latter case.


Yes, that's right. But it's an inherent dilemma: iptables-nft, in order to be
as compatible to legacy iptables as possible, has to call iptables extensions
in kernel for at least some matches/targets. nft in turn does not offer to call
those, so there's no syntax we could use to list those compat expressions
created by iptables-nft. Up until now there are but two possible outcomes: One
is the "# CHECKSUM fill" you see in comment 3, this is nft telling you there's
a compat expression calling an extension that doesn't have a translate callback
in libxtables. The other is the misleading 'ct state' output you see above. It
is nft telling you what the rule does, but still omits the fact that it is a
compat expression internally.

From a technical point of view there is no good solution to this. :(

Hi,

Thank you so much for the detailed explanation. I think I'll have to do some further reading about this but I'll close this BZ as WON'T FIX and I will help the customer clean up their iptables rules and switch to iptables.service instead of nftables.service on system boot and see if that helps. Or other wise hunt down the service that's using nft instead of iptables-nft.

Thanks,

Andreas