Description of problem: I've setup a small server, which mirrors the Fedora repositories (extras, updates, etc.) and provide the packages to few PCs. The server is using vsftpd from xinetd (actually in daemon mode the problem shows up even more frequently) in order to provide the rpms to the clients. Sometimes, when doing something like "yum update" (from a client) involving the transfer of several (more than 20) packages, some failure occurs. Typically something like "the file xyz is not found, no more mirrors to try...". Actually the file is there (on the server), in fact a second "yum update" run download the missing file and the update goes on. Of course this is not nice. Looking at the /var/log/messages on the server, the following is reported by audit (extracted from the logs): Dec 4 17:11:05 rain2 kernel: audit(1165248665.743:5): avc: denied { name_bind } for pid=11060 comm="vsftpd" src=9281 scontext=system_u:system_r:ftpd_t:s0 tcontext=system_u:object_r:hplip_port_t:s0 tclass=tcp_socket Dec 5 09:55:27 rain2 kernel: audit(1165308927.682:5): avc: denied { name_bind } for pid=9182 comm="vsftpd" src=6017 scontext=system_u:system_r:ftpd_t:s0 tcontext=system_u:object_r:xserver_port_t:s0 tclass=tcp_socket Dec 5 10:01:49 rain2 kernel: audit(1165309309.480:6): avc: denied { name_bind } for pid=10308 comm="vsftpd" src=6017 scontext=system_u:system_r:ftpd_t:s0 tcontext=system_u:object_r:xserver_port_t:s0 tclass=tcp_socket Dec 5 10:05:09 rain2 kernel: audit(1165309509.723:7): avc: denied { name_bind } for pid=11747 comm="vsftpd" src=9050 scontext=root:system_r:ftpd_t:s0 tcontext=system_u:object_r:tor_port_t:s0 tclass=tcp_socket Dec 5 10:05:13 rain2 kernel: audit(1165309513.474:8): avc: denied { name_bind } for pid=11772 comm="vsftpd" src=9280 scontext=root:system_r:ftpd_t:s0 tcontext=system_u:object_r:hplip_port_t:s0 tclass=tcp_socket Dec 5 10:05:17 rain2 kernel: audit(1165309517.544:9): avc: denied { name_bind } for pid=11866 comm="vsftpd" src=5149 scontext=root:system_r:ftpd_t:s0 tcontext=system_u:object_r:cluster_port_t:s0 tclass=tcp_socket Dec 5 10:07:03 rain2 kernel: audit(1165309623.288:10): avc: denied { name_bind } for pid=12965 comm="vsftpd" src=6667 scontext=root:system_r:ftpd_t:s0 tcontext=system_u:object_r:ircd_port_t:s0 tclass=tcp_socket Dec 5 14:05:31 rain2 kernel: audit(1165323931.701:11): avc: denied { name_bind } for pid=19938 comm="vsftpd" src=5432 scontext=root:system_r:ftpd_t:s0 tcontext=system_u:object_r:postgresql_port_t:s0 tclass=tcp_socket Dec 12 09:04:51 rain2 kernel: audit(1165910691.474:7): avc: denied { name_bind } for pid=25081 comm="vsftpd" src=8081 scontext=system_u:system_r:ftpd_t:s0 tcontext=system_u:object_r:transproxy_port_t:s0 tclass=tcp_socket I've absolutely no understanding of the failure, but from the behaviour of yum, it looks like an ftp connection is opened (and closed) for each file and, at a certain point, a failure occurs, like avoiding a DoS attack (but this is not the case, of course). As mentioned above, vsftpd is started from xinetd (which slows down the operations). In standalone daemon mode, the problem occurs more often. The selinux installation is the default one, this is a clean FC6 install, so it is what it is, I've no idea, I did not touch anything. Version-Release number of selected component (if applicable): 2.4.6-1.fc6 How reproducible: Well, not really always, it seems randomly happening when a couple of tenth files are transferred by yum. Steps to Reproduce: 1. yum update on a client with many updates to do... 2. 3. Actual results: Sometimes one (or more) file transfer fails. Expected results: All transfer should work properly. Additional info: I've the impression this problem occured after some updates. I did not see it at the beginning, after the initial FC6 installation.
In the current RHEL5 policy selinux-policy-2.4.6-9.el5, these name_bind calls are dontaudits. What is going on here, I beleive, is vsftpd randomly picks a port above 1024 and tries to bind to it, If the kernel denies it because it is defined for another application the bind will fail and vsftpd will try another random port. This generates an AVC message if SELinux denied it but vsftpd will continue to work properly. So by dontauditng these messages, I believe we can ignore them.
The problem is that vsftpd fails completely. The file requested, remotely by yum, is not transferred and yum itself fails as a consequence. So, vsftpd does not pick up anther port and it does not continue, I guess, because in this case yum, on the other side, will not have problems. Now, if you say this is a vsftpd problem, we could re-assign the bug to it. I assumed this was a selinux issue, due to the audit report, maybe I was wrong, but somewhere something does not work properly. Any suggestions? Should this be reassigned?
mbarabas is this expected behaviour. IE If ftpd tries to bind to a port for the alternate channel and the bind fails, shouldn't it continue to retry?
Ok, after doing some more talking and investigating, I believe this is a kernel/selinux bug. Basically the bind call without specifying a port is supposed to grab a random port >1024 thaat is not bound. I think one section of the kernel/clib looks for ports that are not currently bount and then hands it back to the bind command. The bind command then actually attempts to bind to the port and SELinux says no. Thus bind fails and vsftp fails. So the kernel only return ports that the domain can bind to. Hopefully this explains what I think is going on. For now I can code around this by allowing ftp to bind to all ports > 1024. But this is not an good solution in the long run.
I'm not sure I understand why this isn't a good solution. Isn't the ftp server supposed to be able to bind to anything >1024 ? Shouldn't this be allowed in policy anyway? So since policy is 'arbitrarily' limiting what ports >1024 it can use do I understand that you want the kernel/glibc to find an open port and then to check if policy would allow bind to that port before passing it back to the application? Stretching this just a little bit more, are we next going to say "it should also check to make sure it can read and write to that port in case the application ask for something it can't use?" I'm a little frightend of such intelligent decision making, I can't put my finger on the reason why right now, but it just isn't sitting right with me on first thought. The way I see it the application ask for anything >1024. It got something >1024. The policy says it can't have anything >1024 it can only have some things >1024. Personally I feel like either the app should be changed or policy...
Ideally one would like to limit each service to only bind to its own well-defined ports and to transient ports that are not well-defined for other services (to prevent masquerading as another service). Well-defined services may live on ports > 1024 as well as ports < 1024. SELinux name_bind checking is currently only applied on explicit bind(2) calls in the generic socket layer, so it would only show up if the application (or library) specified a non-zero port in the bind(2) call (not upon any automatic binding by the kernel). Hence, it makes no sense to have SELinux policy try to control the local port range (as defined by /proc/sys/net/ipv4/ip_local_port_range) since we cannot provide any real guarantees there. But we can control ports > 1024 that are outside of that range. The range can vary a bit based on available memory, but as an example, it is 32768-61000 by default. It doesn't appear that the ports above fall into that range, so they aren't being auto selected by the kernel. vsftpd would have to already handle a failure on bind(2) calls since that can happen anyway if the port is already in use. Only thing I can think of is that it is bailing because the errno is EACCES rather than EADDRINUSE. bindresvport(3) in glibc has a similar issue in that it probes for an available reserved port (< 1024) by scanning through them in sequence starting from a "random" port computed based on pid; this can trigger a lot of name_bind denials even for ports that are in use because the SELinux hook is reached before the in-use check (which happens in the inet layer, one layer down from the generic socket layer). But that just requires dontaudit's, as Dan noted. strace and/or ltrace of vsftpd including the failed bind() calls would be helpful. And looking at the source code to see exactly what it is doing. If we find a similar issue in many applications, then we could have the SELinux bind hook return EADDRINUSE instead of EACCES to ensure that applications still proceed to scan for other ports, but then the app has no way to distinguish permission denial from in-use port.
Took a quick look at the vsftpd sources. In handle_pasv() in postlogin.c, there are a couple of interesting points: 1) At most it will only try binding 10 times (bind_retries) before aborting, so if it happens to start at a point in the port space that has many well-defined ports, it could easily fail. Of course, the same issue could happen even without SELinux if you are actually running all of those services and those ports are all in use. 2) It checks the errno from the bind call (all wrapped by its sysutil functions) and aborts immediately if it is anything other than EADDRINUSE. So this probing logic won't work so well in the presence of SELinux. bindresvport() in glibc is more resilient for reserved ports; unfortunate that there isn't an equivalent libc routine for unreserved ports - or is there? I don't know how pervasive such logic is in applications. Suggest taking up this issue on selinux list, as it has implications either for policy (allowing name_bind to all unreserved ports for domains like vsftpd) or for the kernel (changing selinux_socket_bind to return EADDRINUSE rather than EACCES for name_bind and node_bind failures).
One other item to investigate - why doesn't vsftpd just call bind(2) with a zero port to ask the kernel to auto select one from the local port range? That is more efficient (kernel can scan the range internally rather than requiring multiple bind calls) and avoids any SELinux name_bind checking at all. Only difference is that it is limited to the local port range rather than all unreserved ports, so it could fail if the entire local port range is in use, but vsftpd could always fall back to its own scan at that point.
Oh, I see - the pasv min/max ports are tunable items for vsftpd to help with firewall rules, so they don't want to have the kernel scanning the entire local port range. Elsewhere, in ftpdataio.c, they do call bind(2) with a zero port to let the kernel select for them, but not for pasv. Pity one can't ask the kernel to scan a particular range in a single bind() call.
Obviously one could also patch vsftpd to handle EACCES in the same manner as EADDRINUSE, if we think the problem is fairly localized to vsftpd and isn't pervasive among similar applications.
This is the same bug, as # 198677 (but on rhel5) : https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=198677
I the last week, after setting selinux to permissive, I extensively used the local yum-by-ftp procedure and I could not see the problem anymore (in the logs, of course). If I understand correctly, selinux in permissive mode should still report denials, shouldn't it? There were a couple of selinux-policies updates, could be one of these had something to do with the problem?
Please, try to test package vsftpd-2.0.5-11 from rawhide. Thanks
(In reply to comment #13) > Please, try to test package vsftpd-2.0.5-11 from rawhide. Thanks Nah, well, I'll do it, if I could reproduce the problem. As I wrote in comment #12, I do not see the issue anymore, despite the heavy ftp load. I'll try to set selinux back to enforce and I'll see what's happening. If the problem returns, I'll test the vsftpd from rawhide. Is this OK?
(In reply to comment #14) > (In reply to comment #13) > > Please, try to test package vsftpd-2.0.5-11 from rawhide. Thanks > > Nah, well, I'll do it, if I could reproduce the problem. > As I wrote in comment #12, I do not see the issue anymore, despite the heavy ftp > load. > I'll try to set selinux back to enforce and I'll see what's happening. If the > problem returns, I'll test the vsftpd from rawhide. > > Is this OK? Yes, thanks.
Hi, unfortunately I cannot reproduce the problem anymore. Maybe the latest 2 or 3 selinux updates somehow changed something and the issue seems to be solved or not visible anymore. How should we proceed? Leave it like it is and wait if it happens again? Thanks.
I close this bug, please reopen it when it happens again. Thanks