Created attachment 1859521 [details] Output of "conntrack -E" on Debian and Fedora 35 Created attachment 1859521 [details] Output of "conntrack -E" on Debian and Fedora 35 1. Please describe the problem: Fedora based router internet <-> LAN WLAN <-> LAN router without NAT connected to LAN. Route to WLAN on Fedora. Route from WLAN to LAN . Direct on subnet Route from LAN to WLAN: Default DHCP route via Fedora.Asymmetric route Ping from WLAN to LAN works TCP connection from WLAN to LAN,e.g. SSH or HTTP hangs. 2. What is the Version-Release number of the kernel: 5.16.5-200 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : The bug is introduced between 5.15.4-200.fc35.x86_64 (good) and 5.15.15-200.fc35.x86_64 (error present) 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: Yes. Additional router on LAN net. Attach second no-nat router to the LAN Define route to subnet behind second router on Fedora Leave only default route on the rest of the LAN clients Type on the second subnet "ping <client on main subnet>" OK. Type on the second subnet "ssh <client on main subnet>" Hangup. Workarounds: Workaround 1: Activate ICMP-redirect instruction clients to take the better route Workaround 2: Block conntrack for LAN-LAN2 traffic by iptables -t raw -A PREROUTING -j CT --notrack Workaround 3: set /proc/sys/net/netfilter/nf_conntrack_tcp_loose to zero. Reason: As shown in conntrack -E, packets are sent out to another port without any NAT rule in place. Due to the asymmetry in the routing, the sender is not aware of this and waits for the correct packet. The output of conntrack -E is attached for a working Debian system and a failing Fedora 35 system. dport is changed rightside, and sent-out as seen by Wireshark. 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: Yes, same result. 6. Are you running any modules that not shipped with directly Fedora's kernel?: No. 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. No releavant logs, the information is in the attachmen
The problem is present in 5.16.8 and 5.16.9 The problem is NOT present in Archlinux with kernel 5.16.8 It is also present in 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64 I see a file patch patch-5.17-redhat.patch in the SOURCES folder with some patches in the conntrack/nat system, could there be a possibility that this bug is a side effect of this patch? (no knowledge of the code, only the patch seen... )
If I compiled everything correctly, the problem is related to patches in nf_conntrack_core.c and nf_nat_core.c. The answer packets passing the ethernet interface without having the original SYN packets seen because of the routing asymmetry change port. Creating nf_conntrack.ko and nf_nat.ko using the original files from the kernel tar file, port 22 does not change upon a SSH connection.
Not working for Fedora 5.16 kernels (should have started with 5.15.15) and working on Arch and Debian means this is likely tied to "netfilter: nat: force port remap to prevent shadowing well-known ports" which is a fix for CVE-2021-3773 and is now upstream with 5.17 (meaning all distros 5.17 kernels should exhibit similar behavior). I have pointed the author of that patch to this bug so that we can see what the best solution is here.
Thanks for looking into it. Funny is that if you connect to a ncat server listening at port 16384, there is no problem, but port 16383 fails. This points directly to location(s) calling the function tuple_port_force_remap in nf_nat_core.c. So to be clear: Client is on subnet 2, routed to subnet 1 by no-nat wifi router, connects to server on subnet1 without being seen by Fedora. Server on subnet1 has only default route to Fedora, is routed there on same eth to subnet2 via wifi router's subnet1 address.
hi all, just my 2 cents: this patch renders all (open)vpn infrastructures unusable. for example ssh does not work, because the router changes the port of reply message from port 22. for me it's a scary prospect that this is included in kernels 5.17 onwards, as it means we have to recompile the kernel for the router if/when patching it. pretty please consider modifying this patch so that a router does not alter the ports. thanks, zfelleg
(In reply to zfelleg from comment #6) > hi all, > > just my 2 cents: this patch renders all (open)vpn infrastructures unusable. How so? The patch results in an implicit "-j MASQUERADE --random", this is safe on normal nat routers. Could you elaborate on how anything breaks? > for example ssh does not work, because the router changes the port of reply > message from port 22. I don't see how thats possible, as the relevant code is only executed for the very first packet, i.e. the syn packet (highport -> 22), so no port rewrite would take place. Please consider sending a patch to https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/netfilter/nft_nat.sh to add a test for the non-working scenario that you describe above, that would help for sure. > kernels 5.17 onwards, as it means we have to recompile the kernel for the > router if/when patching it. pretty please consider modifying this patch so > that a router does not alter the ports. Pure router does NOT alter ports, even with the patch. The patched code sits in the nat area, if you have a pure router (no nat rules), no rewrite takes place. If you have NAT rules (snat, masquerade), then source port alternation is part of normal operations, with or without this patch.
I get a bit lost, sorry, I'm no specialist > The patch results in an implicit "-j MASQUERADE --random" : Do you mean : random is added as fine-tuning for MASQUERADE or that "-j MASQUERADE --random" is default? If I have understood it correctly, changing source port to whatever is an essential process in masquerading because unique connection parameters for the return packets are created. BUT: In the original asymmetric case there is no NAT at all in the data path, so that's why is was very surprised about this port change. Even: NAT is fatal, you connect to a host on the LAN via the direct link and the client gets back the router's address in the return packet. The original problem is that conntrack does not see the client-to-server path and has to decide something with packets coming in unexpectedly from the server to the client into the router. But no idea why it suddenly starts changing ports while no NAT at all is in place in this path. If the connection is set-up symmetric, so both the client-server and server-client packets routed via the Fedora router, there is no problem. So may be: As @zfellig has the same symptoms as I had,may be the problem is the same: some routing issues in the VPN setup causing conntrack seeing the packets in only one direction. Personally, I have not had any problem with OpenVPN or IPSec with the patch installed, but this was only with a simple setup.
(In reply to Florian Westphal from comment #7) hi, first of all, sorry for my english, it's clearly not my native language. i'll try to describe the situation we are in in detail. however, i don't know what patch i should send and where to. if you still need it after this, please elaborate. so: we have a containerized environment running on a fedora 35 server. the container infrastructure is lxc. up until kernel 5.5.15 (and here i mean the host kernel) everything went smoothly, after that we could not establish _any_ connections through openvpn. the infrastructure is as follows: we have an external firewall/gateway (efg for short), a dmz network, an internal firewall/gateway (ifg for short), and an internal network. the openvpn server sits in the internal network. this is the crude visualization: Internet | +-----+ | efg | +-----+ | +------ dmz network (irrelevant for us) | +-----+ | ifg | +-----+ | +---+-------+--- internal network | | +-----+ +-----+ | vpn | | tgt | +-----+ +-----+ let's follow an ssh packet destined for tgt, over openvpn. step 1.: the openvpn encapsulated ssh packet arrives at efg step 2.: efg dnats it to vpn, through ifg step 3.: ifg forwards it to vpn step 4.: vpn de-encapsulates(?) it, from now on it is an ssh packet (originated from the vpn network, which is not the same as the internal network) step 5.: vpn sends it to the tgt server so far so good. now comes the problem. step 6.: tgt reponds from port 22, but this response is sent to ifg instead of the vpn server, as tgt's default gateway is ifg, and the origin of the request is not on the internal network. step 7.: ifg forwards the packet to the vpn server (it has a route for the vpn network through the vpn server), and this is the point where port 22 is randomized. (again, from kernel 5.5.15 onwards) step 8.: vpn encapsulates the ssh packet, from now on it is an openvpn packet, and sends it on its merry way... (the first two steps are irrelevant for us i think, just included them for the sake of accuracy.) so what we see is this: from kernel 5.5.15 on (where this patch gets applied by the fedora rpm build process), if a router gets a package, and it forwards/sends it _on the same interface it arrived on_, the port gets randomized. and it breaks openvpn, and i assume all vpn connections, as any (we have tried it with ssh and dns) request gets a reply from a port randomly changed, which of course is _not_ treated as a legit reply. this packet is definitely not nat-ted, but still its port gets rewrited somehow. thanks, zfelleg
@zfelleg, From the bug starter: I think you have in a more complex context exactly the same situation as I reported in the original report. Best is to distribute the route for the VPN server over the internal network as DHCP route option, then ifg is no longer involved, but may be easier is an iptables rule on ifg like: /sbin/iptables -t raw -A PREROUTING -s <intranet subnet> -d <VPN subnet> -j CT --notrack. Then connection tracking for those packets is disabled and ifg just forwards them to the VPN server.
(In reply to H.Janssen from comment #10) > @zfelleg, > > From the bug starter: > > I think you have in a more complex context exactly the same situation as I > reported in the original report. > Best is to distribute the route for the VPN server over the internal network > as DHCP route option, then ifg is no longer involved, but may be easier is > an iptables rule on ifg like: > > /sbin/iptables -t raw -A PREROUTING -s <intranet subnet> -d <VPN subnet> -j > CT --notrack. > > Then connection tracking for those packets is disabled and ifg just forwards > them to the VPN server. thanks. i'll look into the firewall rule, it seems promising. (just have to translate it to nft, as we have switched over from iptables.) the dhcp route option is not suitable for us, as we have quite a few servers with static ip addresses, and i'm very reluctant to add a route to each and every one of them. but i still don't get it: if the port rewrite takes place only at the first packet with SYN set, which is not seen by the router(!), then why and how does it rewrite the reply, which is the only packet it sees? (but i must admit, my knowledge about SYN, ACK, and other flags are very rusty.)
I understand, statically configured servers are a problem. I do not understand too what's happening exactly, it looks like a not foreseen state. If you send a datafile from client to a nc server listening on 80: client sends SYN to server server sends syn,ack to ifg ifg sends syn,ack to client as out-of-order packet. client sends ACK client sends DATA which arrive on server server sends ACK to ifg ifg sends ACK to client with wrong source port number client does not like that and sends RST, hangs waiting for correct one. So probably the first unexpected syn,ack from the server triggers the port change process. You can also try on the VPN: ip route add <intranet> via <ifg> dev <vpnintranetdev> table 100 ip rule add iif tunx table 100 This forces traffic coming from openvpn via ifg instead of directly to the intranet
(In reply to H.Janssen from comment #12) > I understand, statically configured servers are a problem. > I do not understand too what's happening exactly, it looks like a not > foreseen state. > > If you send a datafile from client to a nc server listening on 80: > > client sends SYN to server > server sends syn,ack to ifg > ifg sends syn,ack to client as out-of-order packet. for us, ifg changes the port of this packet, so the communication halts here. the client does not send the following ack message, it just sits there and waits for the server's syn,ack. but now it makes sense: this handshake reply packet has the SYN flag set, the router sees this as the first packet of the communication, so it changes the port. would it be feasible to change the port of only those packets which have the SYN flag set, but not the ACK? that would solve the problem. (at least for the TCP communication, i don't know about UDP.) > client sends ACK > client sends DATA which arrive on server > server sends ACK to ifg > ifg sends ACK to client with wrong source port number > client does not like that and sends RST, hangs waiting for correct one. > > So probably the first unexpected syn,ack from the server triggers the port > change process. > > > You can also try on the VPN: > ip route add <intranet> via <ifg> dev <vpnintranetdev> table 100 > ip rule add iif tunx table 100 this is even better than adding a rule to the firewall. thanks! > > This forces traffic coming from openvpn via ifg instead of directly to the > intranet
(In reply to zfelleg from comment #11) > (In reply to H.Janssen from comment #10) > but i still don't get it: if the port rewrite takes place only at the first > packet with SYN set, which is not seen by the router(!), then why and how > does it rewrite the reply, which is the only packet it sees? The rewrite takes place for the first packet it sees, in your case thats the syn-ack. We will fix this either by applying a variant of this patch: https://lore.kernel.org/netfilter-devel/20220302105908.GA5852@breakpoint.cc/ or by a revert of the CVE fix. I suspect its going to be the latter.
The two patches have been reverted upstream: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=ee0a4dc9f317fb9a97f20037d219802ca8de939b https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=a82c25c366b0963d33ddf699196e6cf57f6d89b1
(In reply to Florian Westphal from comment #15) > The two patches have been reverted upstream: > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/ > ?id=ee0a4dc9f317fb9a97f20037d219802ca8de939b > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/ > ?id=a82c25c366b0963d33ddf699196e6cf57f6d89b1 Reverts were released with 5.17 kernel.
Hello Florian: We found a similar issue in [1], kernel-4.18.0-362.el8. The SSH traffic is SNATed; the client can connect but the connection is stuck. We have tested several versions and we found the issue appears in [2], just when the commented patches were merged. Is it possible to have those reverts in 4.18? Do I need to open another BZ? Thank you in advance and regards. [1]https://bugzilla.redhat.com/show_bug.cgi?id=2065504 [2]http://pkgs.devel.redhat.com/cgit/rpms/kernel/commit/?h=rhel-8.6.0&id=72aa3ceac59e8732bc2c4df70e761ef6071ed32a
(In reply to Rodolfo Alonso from comment #17) > Hello Florian: > > We found a similar issue in [1], kernel-4.18.0-362.el8. The SSH traffic is > SNATed; the client can connect but the connection is stuck. We have tested > several versions and we found the issue appears in [2], just when the > commented patches were merged. > > Is it possible to have those reverts in 4.18? Do I need to open another BZ? They were reverted in kernel-4.18.0-397.el8. They were not backported to 8.4 and are also reverted in the 8.5 and 8.6 branches.
Thanks a lot Florian!
This message is a reminder that Fedora Linux 35 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '35'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 35 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Fedora Linux 35 entered end-of-life (EOL) status on 2022-12-13. Fedora Linux 35 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora Linux please feel free to reopen this bug against that version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see the version field. If you are unable to reopen this bug, please file a new report against an active release. Thank you for reporting this bug and we are sorry it could not be fixed.