Description of problem: Per https://www.systutorials.com/linux-kernels/771204/netfilter-doc-add-nf_tables-part-in-tproxy-txt-linux-4-19/ # nft add table filter # nft add chain filter divert "{ type filter hook prerouting priority -150; }" # nft add rule filter divert meta l4proto tcp socket transparent 1 meta mark set 1 accept # nft add rule filter divert tcp dport 80 tproxy to :50080 meta mark set 1 accept should work The first two work, but the last yields: # nft add rule filter divert meta l4proto tcp socket transparent 1 meta mark set 1 accept Error: Could not process rule: No such file or directory add rule filter divert meta l4proto tcp socket transparent 1 meta mark set 1 accept ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # nft add rule filter divert tcp dport 80 tproxy to :50080 meta mark set 1 accept Error: syntax error, unexpected to add rule filter divert tcp dport 80 tproxy to :50080 meta mark set 1 accept ^^ Version-Release number of selected component (if applicable): kernel-4.19.2-300.fc29.x86_64 nftables-0.9.0-2.fc29.x86_64
Created attachment 1507473 [details] Patch for x86_64 config This is why it doesn't work. I have supplied the patch for x86_64. I assume changing file names will fix the other arches.
Even with the modules nft_tproxy and nft_socket compiled and loaded, the complaint about unexpected to remains.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There are a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs. Fedora 29 has now been rebased to 4.20.5-200.fc29. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you experience different issues, please open a new bug report for those.
This appears to still be a valid bug.
Hi, I am trying to solve this issue. The first rule works fine but the second one doesn't. # nft add table filter # nft add chain filter divert "{ type filter hook prerouting priority -150; }" # nft add rule filter divert meta l4proto tcp socket transparent 1 meta mark set 1 accept # nft list ruleset table ip filter { chain divert { type filter hook prerouting priority mangle; policy accept; meta l4proto tcp socket transparent 1 meta mark set 0x00000001 accept } } Could you provide more information? (nft version, kernel version..) Thank you!
After compile and load tproxy and socket module it works fine. It is still not working for you?
nftables-0.9.0-2.fc29.x86_64 4.20.x was the last I tried. Currently the system is kernel-5.0.5-200.fc29.x86_64. Are you a Fedora packager? Would you mind sharing x86_64 packages or providing a link to a Fedora build with the tproxy and socket modules included. Last I tried, it crashed when I tried to load one or the other (or both). I gave up at that point.
No, I am not a Fedora packager, I am a Netfilter contributor. I am going to test it on Fedora but I have been reviewing the tproxy and socket code and it seems fine to me.
Hi, the nftables fedora package is based on the last nftables stable release which is http://ftp.netfilter.org/pub/nftables/nftables-0.9.0.tar.bz2. This version doesn't include tproxy support. You can wait for a new stable release or compile an unstable version from https://git.netfilter.org/nftables. Thank you!
We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs. Fedora XX has now been rebased to 5.0.6 Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 30, and are still experiencing this issue, please change the version to Fedora 30. If you experience different issues, please open a new bug report for those.
The kernel config patch is not current, but it is what I am doing. I am providing more information on what I am doing right now to test things (still waiting on kernel to compile, but the others are done). Attaching rpm spec for nftables and libnftnl. These use assumed names for next release. libnftnl is git commit 05123215d4825d25f6069addab5de23067b2ccf7 (HEAD at pull). nftables is git commit 2bb74a7796ea6d7a9df64bb9d3ef57fc31b8d7b7 (HEAD at pull). Besides file versions, there are changes in nftables spec.
Created attachment 1554882 [details] nftaables spec file updated to have necessary BuildRequire and include additional provided files.
Created attachment 1554883 [details] Updated libnftnl spec for version.
Alright, all this time, I thought the crash was coming from the socket module, it is from the tproxy. The following is where it seems to crash (please note, I have tried with and without the second to see if it is that, it doesn't matter): $NFT add rule inet filter divert ip saddr { LIST OF NETWORKS } ip daddr != @myipaddresses tcp dport 80 tproxy ip to :3129 meta mark set 1 counter accept # $NFT add rule inet filter divert ip6 saddr { LIST OF NETWORKS } ip6 daddr != @myipaddresses6 tcp dport 80 tproxy ip6 to :3129 meta mark set 1 counter accept myipaddresses and myipaddresses6 are the addresses of the machine doing the tproxy. This is with kernel 5.0.7 and the libnftnl and nftables as listed above.
Created attachment 1554886 [details] Photo of oops Please, forgive the photo. I know I would make serious mistakes transcribing. I hope this will help track this down.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There are a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs. Fedora 29 has now been rebased to 5.2.9-100.fc29. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 30, and are still experiencing this issue, please change the version to Fedora 30. If you experience different issues, please open a new bug report for those.
Created attachment 1607463 [details] Updated Patch for x86_64 config for 5.2.9
I am recompiling 5.2.9 fc30 now. I will test it and give feed back as soon as I can (it may be tomorrow).
Alright, I am running that kernel. TPROXY is working, but the divert/socket part MUST be the old iptables. The second I turn it on with nft, the system goes down. The tproxy.txt part of the kernel documentation says: # iptables -t mangle -N DIVERT # iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT # iptables -t mangle -A DIVERT -j MARK --set-mark 1 # iptables -t mangle -A DIVERT -j ACCEPT Alternatively you can do this in nft with the following commands: # nft add table filter # nft add chain filter divert "{ type filter hook prerouting priority -150; }" # nft add rule filter divert meta l4proto tcp socket transparent 1 meta mark set 1 accept Mine is: nft add chain inet filter divert "{ type filter hook prerouting priority -150; }" nft add rule inet filter divert meta l4proto tcp socket transparent 1 meta mark set 1 counter accept # This is the line that crashes the system within 30 seconds, usually about 3. The difference is that have inet instead of handling ipv4 or ipv6 separately. I also have a counter added. I doubt that is the problem. Is the problem with divert socket transparent that it is being handed potentially both ipv4 and ipv6 from the same rule? Does it only handle one or the other period?
The first two attachments, reviewed -, can be marked obsolete or not relevant any more.
The line I said is causing the crash happens with or without the tproxy lines.
Hi Trever, thank you for all the efforts explaining the issue. I am going to try to reproduce it tomorrow and give the feedback to you. If I am able to reproduce it I will write a patch soon as possible. Thanks again :-)
I tried to refine the rules as below (using two) to see if it is the IPv4/IPv6 handling being off. It still crashes the second one or both of those installed (or within just a few seconds later). This with the equivalent ip(6)tables rules not being present. nft add rule inet filter divert meta protocol ip meta l4proto tcp socket transparent 1 meta mark set 1 counter accept nft add rule inet filter divert meta protocol ip6 meta l4proto tcp socket transparent 1 meta mark set 1 counter accept
Okay,right now I am running the following ruleset. table inet filter { chain divert { type filter hook prerouting priority mangle; policy accept; meta l4proto tcp socket transparent 1 meta mark set 0x00000001 counter packets 0 bytes 0 accept } } It has been running for some minutes and the kernel did not crash. In the photo that you attached you can see that the problem is on the tproxy module. Is that crashing in your machine (without any other rules)?. Thanks!
libnftnl-1.1.3-1.fc30.x86_64 nftables-0.9.1-2.fc30.x86_64 kernel 5.2.9-200.fc30.x86_64 (exact config to this, except NFT_TUNNEL, NFT_SOCKET, NFT_TPROXY enabled as modules).
This currently works table inet filter { chain divert { # handle 150 type filter hook prerouting priority mangle; policy accept; tcp dport 80 ether saddr @media_players counter packets 0 bytes 0 accept # handle 151 ip saddr { local_ipv4_networks } ip daddr != @myipaddresses tcp dport 80 tproxy ip to :3129 meta mark set 0x00000001 counter packets 1056 bytes 56398 accept # handle 153 ip6 saddr { local_ipv6_networks } ip6 daddr != @myipaddresses6 tcp dport 80 tproxy ip6 to :3129 meta mark set 0x00000001 counter packets 21582 bytes 1554419 accept # handle 155 tcp dport 443 ether saddr @media_players counter packets 101 bytes 12923 accept # handle 156 ip saddr { local_ipv4_networks } ip daddr != @myipaddresses tcp dport 443 tproxy ip to :3130 meta mark set 0x00000001 counter packets 15450 bytes 1666569 accept # handle 158 ip6 saddr { local_ipv6_networks } ip6 daddr != @myipaddresses6 tcp dport 443 tproxy ip6 to :3130 meta mark set 0x00000001 counter packets 5068 bytes 672756 accept # handle 160 } } iptables -t mangle --list -v Chain DIVERT (1 references) pkts bytes target prot opt in out source destination 84106 57M MARK all any any anywhere anywhere MARK set 0x1 84106 57M ACCEPT all any any anywhere anywhere ip6tables -t mangle --list -v Chain DIVERT (1 references) pkts bytes target prot opt in out source destination 108K 158M MARK all -- any any anywhere anywhere MARK set 0x1 108K 158M ACCEPT all -- any any anywhere anywhere The image is from quite a while back. Unfortunately, I do not have an oops from what is going on now. Even if the nftables inet filter divert chain is nonexistent or empty, it crashes when I do the socket rule(s). I do have these.... might they be the problem? table ip nat { # handle 2 chain prerouting { # handle 1 type nat hook prerouting priority filter; policy accept; } chain postrouting { # handle 2 type nat hook postrouting priority srcnat; policy accept; oifname "ppp0" ip saddr { local_ipv4_networks } counter packets 1119 bytes 79447 masquerade # handle 6 } } table inet mangle { # handle 3 chain prerouting { # handle 1 type filter hook prerouting priority mangle; policy accept; } chain forward { # handle 2 type filter hook forward priority mangle; policy accept; oifname { "ppp0", "ipv6-tunnel-interface" } tcp flags syn counter packets 18 bytes 1360 tcp option maxseg size set rt mtu # handle 4 } } I don't think anything else is using anything odd. Basic matching and accepting, drop, counter, reject, etc.
The photo is old. It may not apply. It would have been from 5.0.7 or something like that.
Is that crashing in your machine (without any other rules)?. Thanks! Sorry, unfortunately, this is a production machine and I don't have any other setup I can test it on, so the other rules have to stay in place.
Could you attach an updated photo? I am not able to reproduce it, tomorrow I will review the socket and tproxy code. I am going to give priority to this ticket, thanks.
Hi Trever, I have been trying to reproduce your problem using a similar system. For this test I've used: - kernel 5.2.9-200.fc30.x86_64 (tproxy, socket and tunnel enabled) - libnftnl-1.1.3-1.fc30.x86_64 - nftables-0.9.1-2.fc30.x86_64 That is my result: table inet filter { chain divert { type filter hook prerouting priority mangle; policy accept; meta l4proto tcp socket transparent 1 meta mark set 0x00000001 accept tcp dport 7777 tproxy to :8080 meta mark set 0x00000001 counter packets 4 bytes 320 accept } } That works for me and is handling ipv4 and ipv6 at the same time. I don't have any other rules in my firewall (iptables/nftables). I would like to have a photo of the kernel panic (like the ooops photo). I'm going to analyze it and check where the error occurs. Thanks again!
Created attachment 1609466 [details] Photo 1 of oops on 5.2.9 These two photos are from the current kernel and config posted. I am afraid I don't have the camera that I used in the past. Nothing in system logs. It locks hard immediately. Hopefully, between these two, things are clear enough.
Created attachment 1609467 [details] Photo 2 of oops on 5.2.9 Again, same oops. Hoping for good data between the two as it is not easy to photograph this screen.
You are correct. I do not understand why nft tproxy works with xt_socket and not nft_socket. At least in my setup.
Thanks to Fernando F. Mancera for working with me along with Pablo Neira working with him, it appears this bug is fixed. Please, consider applying the configuration patch I have provided as part of this bug report and releasing fixed kernels as soon as the patch has been committed upstream (preferably released in a kernel as well, but I am ready to use it now!). Thank you to all who have helped track this down and fix it.
Here is the proposed patch. I will update the ticket when it got merged. Thanks! http://patchwork.ozlabs.org/patch/1156147/
The patch landed for 5.3 final. So, the only thing left for Fedora, is to enable the three modules referenced in the patch I provided for the config. Unfortunately, this patch no longer applies as there is a SYNPROXY now as well. But, it isn't hard to enable NFT_TPROXY, NFT_SOCKET, NFT_TUNNEL. Please, consider doing this for the next kernel release in Fedora.
(In reply to Trever Adams from comment #36) > The patch landed for 5.3 final. So, the only thing left for Fedora, is to > enable the three modules referenced in the patch I provided for the config. > Unfortunately, this patch no longer applies as there is a SYNPROXY now as > well. But, it isn't hard to enable NFT_TPROXY, NFT_SOCKET, NFT_TUNNEL. > > Please, consider doing this for the next kernel release in Fedora. Can you update on a specific testable Fedora kernel?
we generally prefer to have options enabled in rawhide for a bit before they go into the stable kernels. I enabled the options in rawhide so they should show up in the git1 build of -rc2. They should make their way to stable kernels when we rebase for 5.4 in a few months.
Eliezer Croitoru, you have to compile it yourself right now. Laura, thank you.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There are a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 30 kernel bugs. Fedora 30 has now been rebased to 5.5.7-100.fc30. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 31, and are still experiencing this issue, please change the version to Fedora 31. If you experience different issues, please open a new bug report for those.
This is fixed.
This has been appropriately fixed.