Bug 1651813 - nft tproxy support not working as documented
Summary: nft tproxy support not working as documented
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 30
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-20 22:57 UTC by Trever Adams
Modified: 2019-10-08 19:39 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:


Attachments (Terms of Use)
Patch for x86_64 config (446 bytes, patch)
2018-11-20 23:21 UTC, Trever Adams
no flags Details | Diff
nftaables spec file updated to have necessary BuildRequire and include additional provided files. (6.86 KB, text/plain)
2019-04-12 22:17 UTC, Trever Adams
trever: review-
Details
Updated libnftnl spec for version. (5.59 KB, text/plain)
2019-04-12 22:18 UTC, Trever Adams
trever: review-
Details
Photo of oops (3.70 MB, image/jpeg)
2019-04-13 04:44 UTC, Trever Adams
no flags Details
Updated Patch for x86_64 config for 5.2.9 (354 bytes, patch)
2019-08-23 17:47 UTC, Trever Adams
no flags Details | Diff
Photo 1 of oops on 5.2.9 (1.22 MB, image/jpeg)
2019-08-29 15:25 UTC, Trever Adams
no flags Details
Photo 2 of oops on 5.2.9 (1.19 MB, image/jpeg)
2019-08-29 15:25 UTC, Trever Adams
no flags Details

Description Trever Adams 2018-11-20 22:57:17 UTC
Description of problem:
Per https://www.systutorials.com/linux-kernels/771204/netfilter-doc-add-nf_tables-part-in-tproxy-txt-linux-4-19/
# nft add table filter
# nft add chain filter divert "{ type filter hook prerouting priority -150; }"
# nft add rule filter divert meta l4proto tcp socket transparent 1 meta mark set 1 accept
# nft add rule filter divert tcp dport 80 tproxy to :50080 meta mark set 1 accept

should work

The first two work, but the last yields:

# nft add rule filter divert meta l4proto tcp socket transparent 1 meta mark set 1 accept
Error: Could not process rule: No such file or directory
add rule filter divert meta l4proto tcp socket transparent 1 meta mark set 1 accept
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

# nft add rule filter divert tcp dport 80 tproxy to :50080 meta mark set 1 accept
Error: syntax error, unexpected to
add rule filter divert tcp dport 80 tproxy to :50080 meta mark set 1 accept
                                           ^^


Version-Release number of selected component (if applicable):
kernel-4.19.2-300.fc29.x86_64
nftables-0.9.0-2.fc29.x86_64

Comment 1 Trever Adams 2018-11-20 23:21:57 UTC
Created attachment 1507473 [details]
Patch for x86_64 config

This is why it doesn't work. I have supplied the patch for x86_64. I assume changing file names will fix the other arches.

Comment 2 Trever Adams 2018-11-22 06:23:40 UTC
Even with the modules nft_tproxy and nft_socket compiled and loaded, the complaint about unexpected to remains.

Comment 3 Justin M. Forbes 2019-01-29 16:13:01 UTC
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There are a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs.

Fedora 29 has now been rebased to 4.20.5-200.fc29.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you experience different issues, please open a new bug report for those.

Comment 4 Trever Adams 2019-01-30 02:01:41 UTC
This appears to still be a valid bug.

Comment 5 Fernando F. Mancera 2019-03-31 12:04:05 UTC
Hi, I am trying to solve this issue. The first rule works fine but the second one doesn't.

# nft add table filter
# nft add chain filter divert "{ type filter hook prerouting priority -150; }"
# nft add rule filter divert meta l4proto tcp socket transparent 1 meta mark set 1 accept
# nft list ruleset
table ip filter {
	chain divert {
		type filter hook prerouting priority mangle; policy accept;
		meta l4proto tcp socket transparent 1 meta mark set 0x00000001 accept
	}
}

Could you provide more information? (nft version, kernel version..) Thank you!

Comment 6 Fernando F. Mancera 2019-03-31 12:42:17 UTC
After compile and load tproxy and socket module it works fine. It is still not working for you?

Comment 7 Trever Adams 2019-03-31 20:57:29 UTC
nftables-0.9.0-2.fc29.x86_64
4.20.x was the last I tried. Currently the system is kernel-5.0.5-200.fc29.x86_64. Are you a Fedora packager? Would you mind sharing x86_64 packages or providing a link to a Fedora build with the tproxy and socket modules included. Last I tried, it crashed when I tried to load one or the other (or both). I gave up at that point.

Comment 8 Fernando F. Mancera 2019-04-01 08:26:25 UTC
No, I am not a Fedora packager, I am a Netfilter contributor. I am going to test it on Fedora but I have been reviewing the tproxy and socket code and it seems fine to me.

Comment 9 Fernando F. Mancera 2019-04-01 17:15:51 UTC
Hi, the nftables fedora package is based on the last nftables stable release which is http://ftp.netfilter.org/pub/nftables/nftables-0.9.0.tar.bz2. This version doesn't include tproxy support.

You can wait for a new stable release or compile an unstable version from https://git.netfilter.org/nftables. Thank you!

Comment 10 Laura Abbott 2019-04-09 20:45:38 UTC
We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs.
 
Fedora XX has now been rebased to 5.0.6  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.
 
If you have moved on to Fedora 30, and are still experiencing this issue, please change the version to Fedora 30.
 
If you experience different issues, please open a new bug report for those.

Comment 11 Trever Adams 2019-04-12 22:17:01 UTC
The kernel config patch is not current, but it is what I am doing. I am providing more information on what I am doing right now to test things (still waiting on kernel to compile, but the others are done).

Attaching rpm spec for nftables and libnftnl. These use assumed names for next release. libnftnl is git commit 05123215d4825d25f6069addab5de23067b2ccf7 (HEAD at pull). nftables is git commit 2bb74a7796ea6d7a9df64bb9d3ef57fc31b8d7b7 (HEAD at pull).

Besides file versions, there are changes in nftables spec.

Comment 12 Trever Adams 2019-04-12 22:17:55 UTC
Created attachment 1554882 [details]
nftaables spec file updated to have necessary BuildRequire and include additional provided files.

Comment 13 Trever Adams 2019-04-12 22:18:24 UTC
Created attachment 1554883 [details]
Updated libnftnl spec for version.

Comment 14 Trever Adams 2019-04-13 04:37:39 UTC
Alright, all this time, I thought the crash was coming from the socket module, it is from the tproxy. The following is where it seems to crash (please note, I have tried with and without the second to see if it is that, it doesn't matter):

	$NFT add rule inet filter divert ip saddr { LIST OF NETWORKS } ip daddr != @myipaddresses tcp dport 80 tproxy ip to :3129 meta mark set 1 counter accept
#	$NFT add rule inet filter divert ip6 saddr { LIST OF NETWORKS } ip6 daddr != @myipaddresses6 tcp dport 80 tproxy ip6 to :3129 meta mark set 1 counter accept

myipaddresses and myipaddresses6 are the addresses of the machine doing the tproxy.

This is with kernel 5.0.7 and the libnftnl and nftables as listed above.

Comment 15 Trever Adams 2019-04-13 04:44:39 UTC
Created attachment 1554886 [details]
Photo of oops

Please, forgive the photo. I know I would make serious mistakes transcribing. I hope this will help track this down.

Comment 16 Justin M. Forbes 2019-08-20 17:42:41 UTC
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There are a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs.

Fedora 29 has now been rebased to 5.2.9-100.fc29.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 30, and are still experiencing this issue, please change the version to Fedora 30.

If you experience different issues, please open a new bug report for those.

Comment 17 Trever Adams 2019-08-23 17:47:04 UTC
Created attachment 1607463 [details]
Updated Patch for x86_64 config for 5.2.9

Comment 18 Trever Adams 2019-08-23 17:49:48 UTC
I am recompiling 5.2.9 fc30 now. I will test it and give feed back as soon as I can (it may be tomorrow).

Comment 19 Trever Adams 2019-08-26 20:10:55 UTC
Alright, I am running that kernel. TPROXY is working, but the divert/socket part MUST be the old iptables. The second I turn it on with nft, the system goes down.

The tproxy.txt part of the kernel documentation says:

# iptables -t mangle -N DIVERT
# iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
# iptables -t mangle -A DIVERT -j MARK --set-mark 1
# iptables -t mangle -A DIVERT -j ACCEPT

Alternatively you can do this in nft with the following commands:

# nft add table filter
# nft add chain filter divert "{ type filter hook prerouting priority -150; }"
# nft add rule filter divert meta l4proto tcp socket transparent 1 meta mark set 1 accept

Mine is:

nft add chain inet filter divert "{ type filter hook prerouting priority -150; }"
nft add rule inet filter divert meta l4proto tcp socket transparent 1 meta mark set 1 counter accept # This is the line that crashes the system within 30 seconds, usually about 3.

The difference is that have inet instead of handling ipv4 or ipv6 separately. I also have a counter added. I doubt that is the problem.

Is the problem with divert socket transparent that it is being handed potentially both ipv4 and ipv6 from the same rule? Does it only handle one or the other period?

Comment 20 Trever Adams 2019-08-26 20:14:33 UTC
The first two attachments, reviewed -, can be marked obsolete or not relevant any more.

Comment 21 Trever Adams 2019-08-26 20:16:54 UTC
The line  I said is causing the crash happens with or without the tproxy lines.

Comment 22 Fernando F. Mancera 2019-08-26 21:47:04 UTC
Hi Trever,

thank you for all the efforts explaining the issue. I am going to try to reproduce it tomorrow and give the feedback to you. If I am able to reproduce it I will write a patch soon as possible.

Thanks again :-)

Comment 23 Trever Adams 2019-08-26 21:55:44 UTC
I tried to refine the rules as below (using two) to see if it is the IPv4/IPv6 handling being off. It still crashes the second one or both of those installed (or within just a few seconds later). This with the equivalent ip(6)tables rules not being present.

nft add rule inet filter divert meta protocol ip meta l4proto tcp socket transparent 1 meta mark set 1 counter accept
nft add rule inet filter divert meta protocol ip6 meta l4proto tcp socket transparent 1 meta mark set 1 counter accept

Comment 24 Fernando F. Mancera 2019-08-26 22:03:25 UTC
Okay,right now I am running the following ruleset.

table inet filter {
        chain divert {
                type filter hook prerouting priority mangle; policy accept;
                meta l4proto tcp socket transparent 1 meta mark set 0x00000001 counter packets 0 bytes 0 accept
        }
}

It has been running for some minutes and the kernel did not crash. In the photo that you attached you can see that the problem is on the tproxy module. Is that crashing in your machine (without any other rules)?. Thanks!

Comment 25 Trever Adams 2019-08-26 22:18:30 UTC
libnftnl-1.1.3-1.fc30.x86_64
nftables-0.9.1-2.fc30.x86_64

kernel 5.2.9-200.fc30.x86_64 (exact config to this, except NFT_TUNNEL, NFT_SOCKET, NFT_TPROXY enabled as modules).

Comment 26 Trever Adams 2019-08-26 22:25:19 UTC
This currently works

table inet filter {
	chain divert { # handle 150
		type filter hook prerouting priority mangle; policy accept;
		tcp dport 80 ether saddr @media_players counter packets 0 bytes 0 accept # handle 151
		ip saddr { local_ipv4_networks } ip daddr != @myipaddresses tcp dport 80 tproxy ip to :3129 meta mark set 0x00000001 counter packets 1056 bytes 56398 accept # handle 153
		ip6 saddr { local_ipv6_networks } ip6 daddr != @myipaddresses6 tcp dport 80 tproxy ip6 to :3129 meta mark set 0x00000001 counter packets 21582 bytes 1554419 accept # handle 155
		tcp dport 443 ether saddr @media_players counter packets 101 bytes 12923 accept # handle 156
		ip saddr { local_ipv4_networks } ip daddr != @myipaddresses tcp dport 443 tproxy ip to :3130 meta mark set 0x00000001 counter packets 15450 bytes 1666569 accept # handle 158
		ip6 saddr { local_ipv6_networks } ip6 daddr != @myipaddresses6 tcp dport 443 tproxy ip6 to :3130 meta mark set 0x00000001 counter packets 5068 bytes 672756 accept # handle 160
	}
}

iptables -t mangle --list -v

Chain DIVERT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
84106   57M MARK       all      any    any     anywhere             anywhere             MARK set 0x1
84106   57M ACCEPT     all      any    any     anywhere             anywhere      

ip6tables -t mangle --list -v

Chain DIVERT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 108K  158M MARK       all  --  any    any     anywhere             anywhere             MARK set 0x1
 108K  158M ACCEPT     all  --  any    any     anywhere             anywhere          

The image is from quite a while back. Unfortunately, I do not have an oops from what is going on now. Even if the nftables inet filter divert chain is nonexistent or empty, it crashes when I do the socket rule(s). I do have these.... might they be the problem?

table ip nat { # handle 2
	chain prerouting { # handle 1
		type nat hook prerouting priority filter; policy accept;
	}

	chain postrouting { # handle 2
		type nat hook postrouting priority srcnat; policy accept;
		oifname "ppp0" ip saddr { local_ipv4_networks } counter packets 1119 bytes 79447 masquerade # handle 6
	}
}
table inet mangle { # handle 3
	chain prerouting { # handle 1
		type filter hook prerouting priority mangle; policy accept;
	}

	chain forward { # handle 2
		type filter hook forward priority mangle; policy accept;
		oifname { "ppp0", "ipv6-tunnel-interface" } tcp flags syn counter packets 18 bytes 1360 tcp option maxseg size set rt mtu # handle 4
	}
}


I don't think anything else is using anything odd. Basic matching and accepting, drop, counter, reject, etc.

Comment 27 Trever Adams 2019-08-26 22:26:37 UTC
The photo is old. It may not apply. It would have been from 5.0.7 or something like that.

Comment 28 Trever Adams 2019-08-26 22:28:33 UTC
Is that crashing in your machine (without any other rules)?. Thanks!

Sorry, unfortunately, this is a production machine and I don't have any other setup I can test it on, so the other rules have to stay in place.

Comment 29 Fernando F. Mancera 2019-08-26 22:53:28 UTC
Could you attach an updated photo? I am not able to reproduce it, tomorrow I will review the socket and tproxy code. I am going to give priority to this ticket, thanks.

Comment 30 Fernando F. Mancera 2019-08-29 00:01:18 UTC
Hi Trever,

I have been trying to reproduce your problem using a similar system. For this test I've used:

- kernel 5.2.9-200.fc30.x86_64 (tproxy, socket and tunnel enabled)
- libnftnl-1.1.3-1.fc30.x86_64
- nftables-0.9.1-2.fc30.x86_64

That is my result:

table inet filter {
	chain divert {
		type filter hook prerouting priority mangle; policy accept;
		meta l4proto tcp socket transparent 1 meta mark set 0x00000001 accept
		tcp dport 7777 tproxy to :8080 meta mark set 0x00000001 counter packets 4 bytes 320 accept
	}
}

That works for me and is handling ipv4 and ipv6 at the same time. I don't have any other rules in my firewall (iptables/nftables). I would like to have a photo of the kernel panic (like the ooops photo). I'm going to analyze it and check where the error occurs.

Thanks again!

Comment 31 Trever Adams 2019-08-29 15:25:16 UTC
Created attachment 1609466 [details]
Photo 1 of oops on 5.2.9

These two photos are from the current kernel and config posted. I am afraid I don't have the camera that I used in the past. Nothing in system logs. It locks hard immediately. Hopefully, between these two, things are clear enough.

Comment 32 Trever Adams 2019-08-29 15:25:56 UTC
Created attachment 1609467 [details]
Photo 2 of oops on 5.2.9

Again, same oops. Hoping for good data between the two as it is not easy to photograph this screen.

Comment 33 Trever Adams 2019-08-29 15:26:35 UTC
You are correct. I do not understand why nft tproxy works with xt_socket and not nft_socket. At least in my setup.

Comment 34 Trever Adams 2019-08-31 01:17:09 UTC
Thanks to Fernando F. Mancera for working with me along with Pablo Neira working with him, it appears this bug is fixed. Please, consider applying the configuration patch I have provided as part of this bug report and releasing fixed kernels as soon as the patch has been committed upstream (preferably released in a kernel as well, but I am ready to use it now!).

Thank you to all who have helped track this down and fix it.

Comment 35 Fernando F. Mancera 2019-08-31 10:29:01 UTC
Here is the proposed patch. I will update the ticket when it got merged. Thanks! http://patchwork.ozlabs.org/patch/1156147/

Comment 36 Trever Adams 2019-09-19 01:24:22 UTC
The patch landed for 5.3 final. So, the only thing left for Fedora, is to enable the three modules referenced in the patch I provided for the config. Unfortunately, this patch no longer applies as there is a SYNPROXY now as well. But, it isn't hard to enable NFT_TPROXY, NFT_SOCKET, NFT_TUNNEL.

Please, consider doing this for the next kernel release in Fedora.

Comment 37 Eliezer Croitoru 2019-10-07 13:54:32 UTC
(In reply to Trever Adams from comment #36)
> The patch landed for 5.3 final. So, the only thing left for Fedora, is to
> enable the three modules referenced in the patch I provided for the config.
> Unfortunately, this patch no longer applies as there is a SYNPROXY now as
> well. But, it isn't hard to enable NFT_TPROXY, NFT_SOCKET, NFT_TUNNEL.
> 
> Please, consider doing this for the next kernel release in Fedora.

Can you update on a specific testable Fedora kernel?

Comment 38 Laura Abbott 2019-10-07 18:00:15 UTC
we generally prefer to have options enabled in rawhide for a bit before they go into the stable kernels. I enabled the options in rawhide so they should show up in the git1 build of -rc2. They should make their way to stable kernels when we rebase for 5.4 in a few months.

Comment 39 Trever Adams 2019-10-08 19:39:27 UTC
Eliezer Croitoru, you have to compile it yourself right now.

Laura, thank you.


Note You need to log in before you can comment on or make changes to this bug.