| Summary: | IPVS does not handle passive FTP correctly | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Jan Tluka <jtluka> |
| Component: | kernel | Assignee: | Jesper Brouer <jbrouer> |
| Status: | CLOSED NOTABUG | QA Contact: | Jan Tluka <jtluka> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 6.1 | CC: | rohara, uwe.knop |
| Target Milestone: | rc | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2012-03-29 13:37:32 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
I've enabled debugging and got following during passive ftp connection: (from one terminal) # telnet 192.168.122.6 21 Trying 192.168.122.6... Connected to 192.168.122.6. Escape character is '^]'. 220 (vsFTPd 2.2.2) user anonymous 331 Please specify the password. pass 230 Login successful. pasv 227 Entering Passive Mode (192,168,122,6,40,175). list 425 Failed to establish connection. (from another terminal) # telnet 192.168.122.6 10415 Trying 192.168.122.6... Connected to 192.168.122.6. Escape character is '^]'. 220 (vsFTPd 2.2.2) Sep 8 14:26:43 dhcp-27-143 kernel: IPVS: ip_vs_wlc_schedule(): Scheduling... Sep 8 14:26:43 dhcp-27-143 kernel: IPVS: WLC: server 192.168.200.40:21 activeconns 0 refcnt 2 weight 1 overhead 1 Sep 8 14:26:43 dhcp-27-143 kernel: IPVS: Bind-dest TCP c:192.168.122.1:45931 v:192.168.122.6:21 d:192.168.200.40:21 fwd:M s:0 conn->flags:100 conn->refcnt:1 dest->refcnt:3 Sep 8 14:26:43 dhcp-27-143 kernel: IPVS: Schedule fwd:M c:192.168.122.1:45931 v:192.168.122.6:21 d:192.168.200.40:21 conn->flags:140 conn->refcnt:2 Sep 8 14:26:43 dhcp-27-143 kernel: IPVS: TCP input [S...] 192.168.200.40:21->192.168.122.1:45931 state: NONE->SYN_RECV conn->refcnt:2 Sep 8 14:26:43 dhcp-27-143 kernel: IPVS: TCP input [..A.] 192.168.200.40:21->192.168.122.1:45931 state: SYN_RECV->ESTABLISHED conn->refcnt:2 Sep 8 14:27:21 dhcp-27-143 kernel: IPVS: ip_vs_wlc_schedule(): Scheduling... Sep 8 14:27:21 dhcp-27-143 kernel: IPVS: WLC: server 192.168.200.40:21 activeconns 1 refcnt 3 weight 1 overhead 257 Sep 8 14:27:21 dhcp-27-143 kernel: IPVS: Bind-dest TCP c:192.168.122.1:50417 v:192.168.122.6:10415 d:192.168.200.40:21 fwd:M s:0 conn->flags:100 conn->refcnt:1 dest->refcnt:4 Sep 8 14:27:21 dhcp-27-143 kernel: IPVS: Schedule fwd:M c:192.168.122.1:50417 v:192.168.122.6:10415 d:192.168.200.40:21 conn->flags:140 conn->refcnt:2 Sep 8 14:27:21 dhcp-27-143 kernel: IPVS: TCP input [S...] 192.168.200.40:21->192.168.122.1:50417 state: NONE->SYN_RECV conn->refcnt:2 Sep 8 14:27:21 dhcp-27-143 kernel: IPVS: TCP input [..A.] 192.168.200.40:21->192.168.122.1:50417 state: SYN_RECV->ESTABLISHED conn->refcnt:2 Sep 8 14:27:22 dhcp-27-143 kernel: IPVS: Unbind-dest TCP c:192.168.122.1:42665 v:192.168.122.6:21 d:192.168.200.40:21 fwd:M s:5 conn->flags:100 conn->refcnt:1 dest->refcnt:4 Since RHEL 6.2 External Beta has begun, and this bug remains unresolved, it has been rejected as it is not proposed as exception or blocker. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux. I'll start working on this... change status to "ASSIGNED" I have successfully recreated your setup and verified that I can provoke the bug. I'll start working on a solution. Packet dumps on the FTP server, show: 1. The correct and working FTP control connection: 192.168.122.1:47678 -> 192.168.200.40:ftp 2. The problem occur, when the passive data connection, is being established: 192.168.122.1:59807 -> 192.168.122.6:15553 Notice the "wrong" dst '192.168.122.6', this is the "public" IPVS/LVS IP-address. It should have been remapped to '192.168.200.40'. That's the basic problem. Now, we need to figure out, why ipvsadm doesn't perform this remapping! SOLUTION I have found a solution to this bug, and its actually NOT-A-BUG in ipvsadm, but a bug in the test-setup config Notice the FTP configuration related to LVS: # grep pasv vsftpd.conf pasv_min_port=10000 pasv_max_port=20000 pasv_address=192.168.122.6 The bug fix is to remove the line: pasv_address=192.168.122.6 The FTP server should just send its own IP as the pasv address, its ipvsadm/ip_vs_ftp's job to perform the remapping. IPVS was actually performing remapping, we just instructed it to use the wrong IP... PITFALLS Extra setup info, if someone tries to reproduce this bug: (1) Remember to load kernel module "nf_conntrack_ftp" on the ftp client machine, to make "active-mode" work (else the clients firewall will drop the incomming ftp-data port packet):: modprobe nf_conntrack_ftp (2) Remember to load kernel module "ip_vs_ftp" on the IPVS "master". On the this RHEL6.2 server, the module was not automatically loaded, which perhaps is a bug it self(?). modprobe ip_vs_ftp (In reply to comment #8) > SOLUTION > > I have found a solution to this bug, and its actually NOT-A-BUG in ipvsadm, but > a bug in the test-setup config > > Notice the FTP configuration related to LVS: > # grep pasv vsftpd.conf > pasv_min_port=10000 > pasv_max_port=20000 > pasv_address=192.168.122.6 > > The bug fix is to remove the line: > pasv_address=192.168.122.6 > > The FTP server should just send its own IP as the pasv address, its > ipvsadm/ip_vs_ftp's job to perform the remapping. IPVS was actually performing > remapping, we just instructed it to use the wrong IP... I'm not convinced this is a solution and the bug is real. I was following the documentation for Load Balancer on RHEL: http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Load_Balancer_Administration/s3-fwm-passive-VSA.html Further, I don't think the FTP server should advertise it's own IP because the address is behind NAT - how a client can send packets to this IP? FTP server must advertise the virtual IP to be able to route packets behind NAT. (In reply to comment #11) > I'm not convinced this is a solution and the bug is real. Could you please, try to reproduce with the pasv_address addr remove? > I was following the documentation for Load Balancer on RHEL: > http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Load_Balancer_Administration/s3-fwm-passive-VSA.html (Thanks for the doc link.) I think there is a bug in the documentation then... > Further, I don't think the FTP server should advertise it's own IP because the > address is behind NAT - how a client can send packets to this IP? FTP server > must advertise the virtual IP to be able to route packets behind NAT. This is handled by the ipvs-system. It will not reveal the IP, but will rewrite the PASV response to use the Virtual-IP. Thus the ftp client only sees the VIP. I have two tcpdumps, that show this action going on. Do you want me to attach these to the case? Jesper, thanks for clarification. Now I see what's been wrong with my test setup. I'll try to run the test using your suggestion. I'll close the bug when I'm done. Thanks, Jan (In reply to comment #12) > (In reply to comment #11) > > > I'm not convinced this is a solution and the bug is real. > > Could you please, try to reproduce with the pasv_address addr remove? > > > > I was following the documentation for Load Balancer on RHEL: > > http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Load_Balancer_Administration/s3-fwm-passive-VSA.html > > (Thanks for the doc link.) > I think there is a bug in the documentation then... > > > > Further, I don't think the FTP server should advertise it's own IP because the > > address is behind NAT - how a client can send packets to this IP? FTP server > > must advertise the virtual IP to be able to route packets behind NAT. > > This is handled by the ipvs-system. It will not reveal the IP, but will > rewrite the PASV response to use the Virtual-IP. Thus the ftp client only sees > the VIP. > > I have two tcpdumps, that show this action going on. Do you want me to attach > these to the case? I confirm that this is not a bug in IPVS. Removing the pasv_address option from vsftpd configuration file fixed my problem. Closing as not a bug. I'll open a bug for Load Balancer documentation as we need to let users know about following: 1. loading ip_vs_ftp module 2. remove pasv_address option to have passive FTP working. Load Balancer Administration documentation bug has been created: https://bugzilla.redhat.com/show_bug.cgi?id=808083 |
Description of problem: It seems that IPVS is not correctly handling ftp-data connections in passive mode. The control connection is forwarded properly: 12:02:54.925703 IP 192.168.200.10.58450 > 192.168.200.40.21: Flags [S], seq 508011187, win 5840, options [mss 1460,sackOK,TS val 7212495 ecr 0,nop,wscale 5], length 0 Once the client tries to open a connection to server for ftp-data, I see that address is not NATted: 12:12:53.286971 IP 192.168.122.1.51027 > 192.168.122.6.19845: Flags [S], seq 2432213335, win 5840, options [mss 1460,sackOK,TS val 181337926 ecr 0,nop,wscale 7], length 0 I'm using following rules in magle table to mark FTP-related packets: # iptables -t mangle -nvL Chain PREROUTING (policy ACCEPT 8859 packets, 816K bytes) pkts bytes target prot opt in out source destination 43 2395 MARK tcp -- * * 0.0.0.0/0 192.168.122.6 tcp dpt:21 MARK set 0x15 10 592 MARK tcp -- * * 0.0.0.0/0 192.168.122.6 tcp dpts:10000:20000 MARK set 0x15 I'm using followin LVS setup: # cat /etc/sysconfig/ha/lvs.cf serial_no = 30 primary = 192.168.122.10 primary_private = 192.168.200.10 service = lvs backup_active = 1 backup = 192.168.122.20 backup_private = 192.168.200.20 heartbeat = 1 heartbeat_port = 539 keepalive = 6 deadtime = 18 network = nat nat_router = 192.168.200.5 eth2:1 nat_nmask = 255.255.255.0 debug_level = NONE monitor_links = 1 syncdaemon = 0 virtual FTP { active = 1 address = 192.168.122.6 eth1:2 vip_nmask = 255.255.255.0 fwmark = 21 port = 21 send = "quit" expect = "220" use_regex = 0 load_monitor = none scheduler = wlc protocol = tcp timeout = 6 reentry = 15 quiesce_server = 0 server s2 { address = 192.168.200.40 active = 1 weight = 1 } } I'm using setup with virtual guests for testing. Network 192.168.122.0/24 is acting as Internet, 192.168.200.0/24 is used as private network. LVS primary is: 192.168.122.10 (public iface), 192.168.200.10 (private iface) LVS backup is: 192.168.122.20, 192.168.200.20 FTP server is: 192.168.200.40 FTP client is connecting from 192.168.122.1 FTP configuration related to LVS: # grep pasv vsftpd.conf pasv_min_port=10000 pasv_max_port=20000 pasv_address=192.168.122.6 Version-Release number of selected component (if applicable): 2.6.32-131.0.15.el6.x86_64 How reproducible: Everytime Steps to Reproduce: 1. See configuration in description above Actual results: ftp-data packets are not forwarded properly, in my case the IP address is not translated to real server IP Expected results: ftp-data packets are forwarded properly, in my case the packets should be forwarded to IP address 192.168.200.40