The problem is caused by optimized prefix match code generation in iptables-nft, nftables doesn't know how to represent the three-byte payload match. This is fixed by this upstream commit:

commit 8a927c56d83ed0f78785011bd92a53edc25a0ca0
Author: Phil Sutter <phil>
Date:   Tue Oct 27 17:05:25 2020 +0100

    src: Support odd-sized payload matches
    
    When expanding a payload match, don't disregard oversized templates at
    the right offset. A more flexible user may extract less bytes from the
    packet if only parts of a field are interesting, e.g. only the prefix of
    source/destination address. Support that by using the template, but fix
    the length. Later when creating a relational expression for it, detect
    the unusually small payload expression length and turn the RHS value
    into a prefix expression.
    
    Signed-off-by: Phil Sutter <phil>

In order to enable the same optimization in nftables as well, we might want to also backport the following commit:


commit 25338cdb6c77aa2f0977afbbb612571c9d325213
Author: Phil Sutter <phil>
Date:   Tue Oct 27 17:33:15 2020 +0100

    src: Optimize prefix matches on byte-boundaries
    
    If a prefix expression's length is on a byte-boundary, it is sufficient
    to just reduce the length passed to "cmp" expression. No need for
    explicit bitwise modification of data on LHS. The relevant code is
    already there, used for string prefix matches. There is one exception
    though, namely zero-length prefixes: Kernel doesn't accept zero-length
    "cmp" expressions, so keep them in the old code-path for now.
    
    This patch depends upon the previous one to correctly parse odd-sized
    payload matches but has to extend support for non-payload LHS as well.
    In practice, this is needed for "ct" expressions as they allow matching
    against IP address prefixes, too.
    
    Signed-off-by: Phil Sutter <phil>

https://src.osci.redhat.com/rpms/nftables/pull-request/4

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nftables bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4465