Bug 642388
Summary: | ip_nat_ftp not working if ack for "227 Enter Passive mode" packet is lost | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 5 | Reporter: | Veaceslav Falico <vfalico> | ||||||||
Component: | kernel | Assignee: | Thomas Graf <tgraf> | ||||||||
Status: | CLOSED ERRATA | QA Contact: | Network QE <network-qe> | ||||||||
Severity: | high | Docs Contact: | |||||||||
Priority: | medium | ||||||||||
Version: | 5.5 | CC: | awestbro, bturner, cye, dennisml, dtian, fleite, herrold, hjia, peterm, qcai, rkhan, rom | ||||||||
Target Milestone: | rc | ||||||||||
Target Release: | --- | ||||||||||
Hardware: | All | ||||||||||
OS: | Linux | ||||||||||
Whiteboard: | |||||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||||
Doc Text: | Story Points: | --- | |||||||||
Clone Of: | |||||||||||
: | 676900 (view as bug list) | Environment: | |||||||||
Last Closed: | 2011-07-21 10:22:55 UTC | Type: | --- | ||||||||
Regression: | --- | Mount Type: | --- | ||||||||
Documentation: | --- | CRM: | |||||||||
Verified Versions: | Category: | --- | |||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||
Embargoed: | |||||||||||
Attachments: |
|
Description
Veaceslav Falico
2010-10-12 18:46:57 UTC
I spent some hours looking at the code in the ip_nat_ftp and ip_conntrack_ftp modules, and I don't think it was designed with the idea of retransmissions of mangled packets in mind. I don't really get why is it that the second PASV packet goes by un-mangled, is it because earlier in the call chain the generic connection tracking code detects it has gone over that SEQ number already? Can we use that information to somehow replay some cached contents? or re-mangle the packet contents but being careful to NOT re-do the other side-effects, such as affecting the SEQ number offset or the related connection expectation. The problem is that once the 227 packet has been processed by ftp conntrack. The seq number is updated in update_nl_seq(). When the same sequence number is processed again it will fail the "are we directly after a '\n' check" and the packet will be accepted without mangling. At least that's what I think is happening. This bug affects any packets being retransmitted, e.g. a retransmitted "PORT" packet would also cause problems. Actually, I think I finally understand that find_nl/update_nl code, and the call to update_nl_seq() at the end of help(). It is supposed to cache the SKB that was mangled, so that the conntrack code can eventually just replay it if needed, by calling ip_conntrack_event_cache(). Perhaps this is what is failing. So apparently the module mangles the SKB and caches it for future replay, but this cache fails to be used later on? I checked and we do build our kernels with CONFIG_IP_NF_CONNTRACK_EVENTS, so the code should be there. Created attachment 453327 [details]
potential fix
This patch may fix the problem. update_nl_seq() contains a bug in RHEL5 which leads to not always updating the older of the two remembered sequence numbers.
This patch fixes this and ensures that the "smallest" of the two sequence numbers being remembered is overwritten by the latest sequence to be expected. This should allow for a retransmitted packet to be accepted because the old sequence number is still around. Every packet matching either of the two remebered sequence numbers will be accepted and mangled.
I just finished setting up a network of 3 KVM guests, where a client guest connects to a router guest which NATs the FTP port to a server guest. Then on the NAT router I apply 5% of packet drops with the netem qdisc. The NATing is subtle, from 192.168.122.101 to .102, but at least it is easy to follow on a capture at the virtual bridge. After a few minutes manually testing, I have a failed PASV response, although it does not quite seem to be the same situation: 569.147952 client -> router FTP Request: PASV 569.148427 router -> server FTP Request: PASV 569.148847 server -> router FTP Response: 227 Entering Passive Mode (192,168,122,101,250,120) 569.149032 router -> client FTP Response: 227 Entering Passive Mode (192,168,122,102,250,120) 569.151381 client -> router FTP Request: LIST 569.151706 router -> server FTP Request: LIST 569.151881 server -> router FTP Response: 150 Here comes the directory listing. 569.152055 server -> router FTP Response: 226 Directory send OK. 569.152218 router -> client FTP [TCP Previous segment lost] Response: 226 Directory send OK. 569.355627 server -> router FTP [TCP Retransmission] Response: 150 Here comes the directory listing. 569.355805 router -> client FTP [TCP Retransmission] Response: 150 Here comes the directory listing. 579.692039 client -> router FTP Request: PASV 579.692281 router -> server FTP Request: PASV 579.692774 server -> router FTP Response: 227 Entering Passive Mode (192,168,122,101,176,156) 579.693025 router -> client FTP Response: 227 Entering Passive Mode (192,168,122,101,176,156) [ Notice that on the packets above there was no NAT of the 227 contents ] 579.696357 client -> router FTP Request: LIST 579.696513 router -> server FTP Request: LIST 579.696841 server -> router FTP Response: 425 Security: Bad IP connecting. 579.697238 router -> client FTP Response: 425 Security: Bad IP connecting. On this capture I was ignoring pure-ack packets, just to trim the packet flow and see if I could actually reproduce something similar. Now that I know we're almost there I'll start looking at all packets, so that we're sure to spot the drops and retransmissions and see about reproducing the exact situation. Cheers, Fábio Olivé 370.826272 client -> router FTP Request: PASV 370.826429 router -> server FTP Request: PASV 370.826796 server -> router FTP Response: 227 Entering Passive Mode (192,168,122,101,132,156) 371.027328 server -> router FTP [TCP Retransmission] Response: 227 Entering Passive Mode (192,168,122,101,132,156) 371.027499 router -> client FTP Response: 227 Entering Passive Mode (192,168,122,101,132,156) I believe this was a valid reproduction. The router dropped the first mangled 227 response, so 200ms later the server retransmitted it, and it went to the client as .101. I'll see about testing your patched kernel ASAP. That is a valid reproduction. My patch does not fix the original problem completely. A retransmitted 227 is still not translated. It does solve the connection abortion issues in my 50% packet loss testing setup though. It seems the patch fixes the issue after all. I can no longer reproduce the problem. I've run hundreds of ftp session with 50% of simulated packet loss and even if the same packet is lost multiple times the session recovers and all transfers completed successfully. Can you confirm Fabio? With an ftpd running on 192.168.101.197 behind 192.168.100.130: 60 35.916735 192.168.100.130 192.168.100.212 FTP Response: 227 Entering Passive Mode (192,168,100,130,139,218) 65 36.119986 192.168.100.130 192.168.100.212 FTP [TCP Retransmission] Response: 227 Entering Passive Mode (192,168,100,130,139,218) Hi Thomas! Indeed, I have been unable to cause the issue even at 10% loss. One case was particularly nasty as both PASV and 227 and their ACKs were lost and retransmitted, and I never got any errors. 54.914910 server -> router FTP Response: 227 Entering Passive Mode (192,168,122,101,87,125) 55.116488 server -> router FTP [TCP Retransmission] Response: 227 Entering Passive Mode (192,168,122,101,87,125) 55.116728 router -> client FTP Response: 227 Entering Passive Mode (192,168,122,102,87,125) 213.049225 server -> router FTP Response: 227 Entering Passive Mode (192,168,122,101,161,242) 213.255426 server -> router FTP [TCP Retransmission] Response: 227 Entering Passive Mode (192,168,122,101,161,242) 213.255699 router -> client FTP Response: 227 Entering Passive Mode (192,168,122,102,161,242) 261.787753 server -> router FTP Response: 227 Entering Passive Mode (192,168,122,101,76,222) 261.992034 server -> router FTP [TCP Retransmission] Response: 227 Entering Passive Mode (192,168,122,101,76,222) 261.992223 router -> client FTP Response: 227 Entering Passive Mode (192,168,122,102,76,222) 283.440550 client -> router FTP Request: PASV 283.440698 router -> server FTP Request: PASV 283.441184 server -> router FTP Response: 227 Entering Passive Mode (192,168,122,101,145,169) 283.645299 client -> router FTP [TCP Retransmission] Request: PASV 283.645472 router -> server FTP [TCP Retransmission] Request: PASV 283.645943 server -> router FTP [TCP Retransmission] Response: 227 Entering Passive Mode (192,168,122,101,145,169) 283.646160 router -> client FTP [TCP Out-Of-Order] Response: 227 Entering Passive Mode (192,168,122,102,145,169) 414.987642 server -> router FTP Response: 227 Entering Passive Mode (192,168,122,101,45,176) 415.192745 server -> router FTP [TCP Retransmission] Response: 227 Entering Passive Mode (192,168,122,101,45,176) 415.192983 router -> client FTP Response: 227 Entering Passive Mode (192,168,122,102,45,176) 698.482338 server -> router FTP Response: 227 Entering Passive Mode (192,168,122,101,29,230) 698.687647 server -> router FTP [TCP Retransmission] Response: 227 Entering Passive Mode (192,168,122,101,29,230) 698.687877 router -> client FTP Response: 227 Entering Passive Mode (192,168,122,102,29,230) Thanks! Fábio Olivé Not sure if this belongs in it's own bug report or not, but I think it's definitely related, as I was able to reproduce the bug without packet loss. 5 Centos 5 machines. xenhost1 running vfw, and www2 xenhost2 running www3 www2 and www3 both have private IP addresses, and use vfw as a firewall. vfw has two virtual NICs bridged to two different VLANS, one used as internal network, one used as a external/public network. All Xen networking is bridging to VLANs on a physical NIC If I run wireshark on vfw, and do a FTP from a outside host to www3, all works as expected, no retransmissions, no CRC errors, no packet loss. FTP client get's passed the external IP address of www2, and all works as it should. If I run wireshark on vfw, and do a FTP from a outside host to www2, I get CRC errors on every single packet that originates from www2 (packets from outside host have correct CRC), and I get retransmissions with the PASV handshake, tripping the above noted bug where retransmitted packet doesn't get mangled like it should be. No packet loss (verified with a second wireshark on the host running the ftp client). FTP client get's passed the internal IP address of www2, and fails. Despite the CRC errors between vfw & www2, any traffic that does not need to be mangled by ip_nat_ftp works just fine. I am guessing that ip_nat_ftp is causing the retransmission of the PASV handshake due to the CRC errors, in effect self generating the case that triggers the bug discussed above (previous notes). So... Should the CRC error trigger ip_nat_ftp to cause a retransmission be a separate bug report, and just consider this as documentation of this bug being triggered by another bug and not packet loss? Or is it enough related to handle it in the bug report? From the research I've done looks like I do have a work around available to me by running the following command on www2 (turning off tcp checksum offload): ethtool -K eth0 tx off I have not yet tried the workaround though. This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. Hi Fred, (In reply to comment #24) > > If I run wireshark on vfw, and do a FTP from a outside host to www2, I get CRC > errors on every single packet that originates from www2 (packets from outside > host have correct CRC), This is because you are capturing on the same physical host and those packets will only have their checksums calculated when they leave the box through the NIC, as "your ethtool -K eth0 tx off" workaround also implies. It is normal for wireshark to complain about those, since it cannot know if those are incoming or outgoing packets. You can disable TCP checksum verification if you want to silence those warnings. > and I get retransmissions with the PASV handshake, > tripping the above noted bug where retransmitted packet doesn't get mangled > like it should be. No packet loss (verified with a second wireshark on the > host running the ftp client). Hmm I believe you are indeed having packet loss, or at least some form of delay that is large enough to trigger a retransmission. How many copies of the PASV packets do you see on the client? If you only see one, and you see two on the host/firewall capture, then it means one of them got dropped somewhere on the network. If you see two, then it likely means the client ACK got dropped or delayed, thus also causing a retransmission from the server. Hope this helps! Fábio Olivé Created attachment 478095 [details]
Wireshark Packet Capture Screenshots (censored)
(In reply to comment #26) -- cut -- > This is because you are capturing on the same physical host and those packets > will only have their checksums calculated when they leave the box through the > NIC, as "your ethtool -K eth0 tx off" workaround also implies. It is normal for > wireshark to complain about those, since it cannot know if those are incoming > or outgoing packets. You can disable TCP checksum verification if you want to > silence those warnings. Understood. The incorrect checksums where the only difference I could find that may have caused the retransmission. -- cut -- > > Hmm I believe you are indeed having packet loss, or at least some form of delay > that is large enough to trigger a retransmission. > > How many copies of the PASV packets do you see on the client? > > If you only see one, and you see two on the host/firewall capture, then it > means one of them got dropped somewhere on the network. If you see two, then it > likely means the client ACK got dropped or delayed, thus also causing a > retransmission from the server. I think the packet capture I attached will show that there was no packet loss, packet count on vfw1 and the ftp client match exactly. As for delayed, I don't really think so either, unless it is a delay caused by ip_nat_ftp. This packet capture is 100% repeatable with ftp connections to www2, and the retransmission only happens on the PASV handshake (every time). Packet loss / delay should make where the retransmission happens random, not repeatable. This ftp issue also did not start until vfw1 and www2 where on the same physical host, and the change of physical host was the only thing changed. Somehow I think ip_nat_ftp is being tripped up by the CRC issue, and this would explain why the retransmission only happens on the PASV handshake. Olá! (In reply to comment #28) > > I think the packet capture I attached will show that there was no packet loss, > packet count on vfw1 and the ftp client match exactly. As for delayed, I don't > really think so either, unless it is a delay caused by ip_nat_ftp. This packet > capture is 100% repeatable with ftp connections to www2, and the retransmission > only happens on the PASV handshake (every time). Packet loss / delay should > make where the retransmission happens random, not repeatable. You are correct. This is a fairly interesting new bug to investigate. > Somehow I think ip_nat_ftp is being tripped up by the CRC issue, and this would > explain why the retransmission only happens on the PASV handshake. There must be some bit of interaction with the virtualization setup. Perhaps the packet is mangled and set for checksum recalculation, and this is delayed to be done by the NIC, but since it is targeted to a VM on the same host it does not leave through the NIC and is sent to the VM directly without having it recalculated? I guess it would be dropped in this case? Wild guess, of course... :) It could be confirmed with disabling TX checksum offload, as in this case the VMs would [re]calculate the cksums themselves. I believe it may be interesting to open a new bug for this investigation, or clone this one to preserve the history and context. Fábio Olivé I can't reproduce this with kvm so far. Can you please report this issue via its own bugzilla? This is definitely a separate issue. Created attachment 484442 [details]
proposed patch
(In reply to comment #32) > I can't reproduce this with kvm so far. Can you please report this issue via > its own bugzilla? This is definitely a separate issue. Already did. https://bugzilla.redhat.com/show_bug.cgi?id=676900 Patch(es) available in kernel-2.6.18-250.el5 Detailed testing feedback is always welcomed. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-1065.html After this patch ip_nat_ftp seems to break FTP connections. See: https://bugzilla.redhat.com/show_bug.cgi?id=740399 Removing ip_nat_ftp from the iptables-config files makes the problem go away. |