Bug 219349 - selinux does not seem to play well with vsftpd
selinux does not seem to play well with vsftpd
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
6
All Linux
medium Severity medium
: ---
: ---
Assigned To: Eric Paris
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2006-12-12 13:51 EST by Piergiorgio Sartor
Modified: 2007-11-30 17:11 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-01-24 04:15:23 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Piergiorgio Sartor 2006-12-12 13:51:45 EST
Description of problem:
I've setup a small server, which mirrors the Fedora repositories (extras,
updates, etc.) and provide the packages to few PCs.
The server is using vsftpd from xinetd (actually in daemon mode the problem
shows up even more frequently) in order to provide the rpms to the clients.
Sometimes, when doing something like "yum update" (from a client) involving the
transfer of several (more than 20) packages, some failure occurs. Typically
something like "the file xyz is not found, no more mirrors to try...".
Actually the file is there (on the server), in fact a second "yum update" run
download the missing file and the update goes on.
Of course this is not nice.

Looking at the /var/log/messages on the server, the following is reported by
audit (extracted from the logs):

Dec  4 17:11:05 rain2 kernel: audit(1165248665.743:5): avc:  denied  { name_bind
} for  pid=11060 comm="vsftpd" src=9281 scontext=system_u:system_r:ftpd_t:s0
tcontext=system_u:object_r:hplip_port_t:s0 tclass=tcp_socket
Dec  5 09:55:27 rain2 kernel: audit(1165308927.682:5): avc:  denied  { name_bind
} for  pid=9182 comm="vsftpd" src=6017 scontext=system_u:system_r:ftpd_t:s0
tcontext=system_u:object_r:xserver_port_t:s0 tclass=tcp_socket
Dec  5 10:01:49 rain2 kernel: audit(1165309309.480:6): avc:  denied  { name_bind
} for  pid=10308 comm="vsftpd" src=6017 scontext=system_u:system_r:ftpd_t:s0
tcontext=system_u:object_r:xserver_port_t:s0 tclass=tcp_socket
Dec  5 10:05:09 rain2 kernel: audit(1165309509.723:7): avc:  denied  { name_bind
} for  pid=11747 comm="vsftpd" src=9050 scontext=root:system_r:ftpd_t:s0
tcontext=system_u:object_r:tor_port_t:s0 tclass=tcp_socket
Dec  5 10:05:13 rain2 kernel: audit(1165309513.474:8): avc:  denied  { name_bind
} for  pid=11772 comm="vsftpd" src=9280 scontext=root:system_r:ftpd_t:s0
tcontext=system_u:object_r:hplip_port_t:s0 tclass=tcp_socket
Dec  5 10:05:17 rain2 kernel: audit(1165309517.544:9): avc:  denied  { name_bind
} for  pid=11866 comm="vsftpd" src=5149 scontext=root:system_r:ftpd_t:s0
tcontext=system_u:object_r:cluster_port_t:s0 tclass=tcp_socket
Dec  5 10:07:03 rain2 kernel: audit(1165309623.288:10): avc:  denied  {
name_bind } for  pid=12965 comm="vsftpd" src=6667
scontext=root:system_r:ftpd_t:s0 tcontext=system_u:object_r:ircd_port_t:s0
tclass=tcp_socket
Dec  5 14:05:31 rain2 kernel: audit(1165323931.701:11): avc:  denied  {
name_bind } for  pid=19938 comm="vsftpd" src=5432
scontext=root:system_r:ftpd_t:s0 tcontext=system_u:object_r:postgresql_port_t:s0
tclass=tcp_socket
Dec 12 09:04:51 rain2 kernel: audit(1165910691.474:7): avc:  denied  { name_bind
} for  pid=25081 comm="vsftpd" src=8081 scontext=system_u:system_r:ftpd_t:s0
tcontext=system_u:object_r:transproxy_port_t:s0 tclass=tcp_socket

I've absolutely no understanding of the failure, but from the behaviour of yum,
it looks like an ftp connection is opened (and closed) for each file and, at a
certain point, a failure occurs, like avoiding a DoS attack (but this is not the
case, of course).
As mentioned above, vsftpd is started from xinetd (which slows down the
operations). In standalone daemon mode, the problem occurs more often.

The selinux installation is the default one, this is a clean FC6 install, so it
is what it is, I've no idea, I did not touch anything.

Version-Release number of selected component (if applicable):
2.4.6-1.fc6

How reproducible:
Well, not really always, it seems randomly happening when a couple of tenth
files are transferred by yum.

Steps to Reproduce:
1.
yum update on a client with many updates to do...
2.
3.
  
Actual results:
Sometimes one (or more) file transfer fails.

Expected results:
All transfer should work properly.

Additional info:
I've the impression this problem occured after some updates. I did not see it at
the beginning, after the initial FC6 installation.
Comment 1 Daniel Walsh 2006-12-12 14:45:01 EST
In the current RHEL5 policy selinux-policy-2.4.6-9.el5, these name_bind calls
are dontaudits.  What is going on here,  I beleive, is vsftpd randomly picks a
port above 1024 and tries to bind to it,  If the kernel denies it because it is
defined for another application the bind will fail and vsftpd will try another
random port.  This generates an AVC message if SELinux denied it but vsftpd will
continue to work properly.  So by dontauditng these messages, I believe we can
ignore them.
Comment 2 Piergiorgio Sartor 2006-12-12 17:14:41 EST
The problem is that vsftpd fails completely. The file requested, remotely by
yum, is not transferred and yum itself fails as a consequence.
So, vsftpd does not pick up anther port and it does not continue, I guess,
because in this case yum, on the other side, will not have problems.
Now, if you say this is a vsftpd problem, we could re-assign the bug to it.
I assumed this was a selinux issue, due to the audit report, maybe I was wrong,
but somewhere something does not work properly.

Any suggestions?
Should this be reassigned?
Comment 3 Daniel Walsh 2006-12-13 15:48:14 EST
mbarabas is this expected behaviour.  IE If ftpd tries to bind to a port for the
alternate channel and the bind fails, shouldn't it continue to retry?
Comment 4 Daniel Walsh 2006-12-21 16:38:14 EST
Ok, after doing some more talking and investigating, I believe this is a
kernel/selinux bug.

Basically the bind call without specifying a port is supposed to grab a random
port >1024 thaat is not bound.
I think one section of the kernel/clib looks for ports that are not currently
bount and then hands it back to the bind command.  The bind command then
actually attempts to bind to the port and SELinux says no.  Thus bind fails and
vsftp fails.

So the kernel only return ports that the domain can bind to.  Hopefully this
explains what I think is going on.  

For now I can code around this by allowing ftp to bind to all ports > 1024.  But
this is not an good solution in the long run.
Comment 5 Eric Paris 2006-12-21 22:59:17 EST
I'm not sure I understand why this isn't a good solution.  Isn't the ftp server
supposed to be able to bind to anything >1024 ?  Shouldn't this be allowed in
policy anyway?

So since policy is 'arbitrarily' limiting what ports >1024 it can use do I
understand that you want the kernel/glibc to find an open port and then to check
if policy would allow bind to that port before passing it back to the
application?  Stretching this just a little bit more, are we next going to say
"it should also check to make sure it can read and write to that port in case
the application ask for something it can't use?"

I'm a little frightend of such intelligent decision making, I can't put my
finger on the reason why right now, but it just isn't sitting right with me on
first thought.  The way I see it the application ask for anything >1024.  It got
something >1024.  The policy says it can't have anything >1024 it can only have
some things >1024.  Personally I feel like either the app should be changed or
policy...
Comment 6 Stephen Smalley 2006-12-22 07:31:45 EST
Ideally one would like to limit each service to only bind to its own
well-defined ports and to transient ports that are not well-defined for other
services (to prevent masquerading as another service).  Well-defined services
may live on ports  > 1024 as well as ports < 1024.

SELinux name_bind checking is currently only applied on explicit bind(2) calls
in the generic socket layer, so it would only show up if the application (or
library) specified a non-zero port in the bind(2) call (not upon any automatic
binding by the kernel).  Hence, it makes no sense to have SELinux policy try to
control the local port range (as defined by
/proc/sys/net/ipv4/ip_local_port_range) since we cannot provide any real
guarantees there.   But we can control ports > 1024 that are outside of that
range.  The range can vary a bit based on available memory, but as an example,
it is 32768-61000 by default.  It doesn't appear that the ports above fall into
that range, so they aren't being auto selected by the kernel.

vsftpd would have to already handle a failure on bind(2) calls since that can
happen anyway if the port is already in use.  Only thing I can think of is that
it is bailing because the errno is EACCES rather than EADDRINUSE.

bindresvport(3) in glibc has a similar issue in that it probes for an available
reserved port (< 1024) by scanning through them in sequence starting from a
"random" port computed based on pid; this can trigger a lot of name_bind denials
even for ports that are in use because the SELinux hook is reached before the
in-use check (which happens in the inet layer, one layer down from the generic
socket layer).  But that just requires dontaudit's, as Dan noted.

strace and/or ltrace of vsftpd including the failed bind() calls would be
helpful.  And looking at the source code to see exactly what it is doing.

If we find a similar issue in many applications, then we could have the SELinux
bind hook return EADDRINUSE instead of EACCES to ensure that applications still
proceed to scan for other ports, but then the app has no way to distinguish
permission denial from in-use port.

Comment 7 Stephen Smalley 2006-12-22 07:55:22 EST
Took a quick look at the vsftpd sources.  In handle_pasv() in postlogin.c, there
are a couple of interesting points:
1) At most it will only try binding 10 times (bind_retries) before aborting, so
if it happens to start at a point in the port space that has many well-defined
ports, it could easily fail.  Of course, the same issue could happen even
without SELinux if you are actually running all of those services and those
ports are all in use.
2) It checks the errno from the bind call (all wrapped by its sysutil functions)
and aborts immediately if it is anything other than EADDRINUSE.

So this probing logic won't work so well in the presence of SELinux. 
bindresvport() in glibc is more resilient for reserved ports; unfortunate that
there isn't an equivalent libc routine for unreserved ports - or is there?
I don't know how pervasive such logic is in applications.

Suggest taking up this issue on selinux list, as it has implications either for
policy (allowing name_bind to all unreserved ports for domains like vsftpd) or
for the kernel (changing selinux_socket_bind to return EADDRINUSE rather than
EACCES for name_bind and node_bind failures). 
Comment 8 Stephen Smalley 2006-12-22 08:13:49 EST
One other item to investigate - why doesn't vsftpd just call bind(2) with a zero
port to ask the kernel to auto select one from the local port range?  That is
more efficient (kernel can scan the range internally rather than requiring
multiple bind calls) and avoids any SELinux name_bind checking at all.  Only
difference is that it is limited to the local port range rather than all
unreserved ports, so it could fail if the entire local port range is in use, but
vsftpd could always fall back to its own scan at that point.
Comment 9 Stephen Smalley 2006-12-22 08:20:13 EST
Oh, I see - the pasv min/max ports are tunable items for vsftpd to help with
firewall rules, so they don't want to have the kernel scanning the entire local
port range.  Elsewhere, in ftpdataio.c, they do call bind(2) with a zero port to
let the kernel select for them, but not for pasv.
Pity one can't ask the kernel to scan a particular range in a single bind() call.
Comment 10 Stephen Smalley 2006-12-22 08:32:17 EST
Obviously one could also patch vsftpd to handle EACCES in the same manner as
EADDRINUSE, if we think the problem is fairly localized to vsftpd and isn't
pervasive among similar applications.


Comment 11 Maros Barabas 2007-01-05 05:27:33 EST
This is the same bug, as # 198677 (but on rhel5) :

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=198677
Comment 12 Piergiorgio Sartor 2007-01-16 12:56:05 EST
I the last week, after setting selinux to permissive, I extensively used the
local yum-by-ftp procedure and I could not see the problem anymore (in the logs,
of course).
If I understand correctly, selinux in permissive mode should still report
denials, shouldn't it?
There were a couple of selinux-policies updates, could be one of these had
something to do with the problem?
Comment 13 Maros Barabas 2007-01-17 09:00:51 EST
Please, try to test package vsftpd-2.0.5-11 from rawhide. Thanks
Comment 14 Piergiorgio Sartor 2007-01-17 15:03:52 EST
(In reply to comment #13)
> Please, try to test package vsftpd-2.0.5-11 from rawhide. Thanks

Nah, well, I'll do it, if I could reproduce the problem.
As I wrote in comment #12, I do not see the issue anymore, despite the heavy ftp
load.
I'll try to set selinux back to enforce and I'll see what's happening. If the
problem returns, I'll test the vsftpd from rawhide.

Is this OK?
Comment 15 Maros Barabas 2007-01-18 09:22:13 EST
(In reply to comment #14)
> (In reply to comment #13)
> > Please, try to test package vsftpd-2.0.5-11 from rawhide. Thanks
> 
> Nah, well, I'll do it, if I could reproduce the problem.
> As I wrote in comment #12, I do not see the issue anymore, despite the heavy ftp
> load.
> I'll try to set selinux back to enforce and I'll see what's happening. If the
> problem returns, I'll test the vsftpd from rawhide.
> 
> Is this OK?

Yes, thanks.
Comment 16 Piergiorgio Sartor 2007-01-23 13:02:24 EST
Hi, unfortunately I cannot reproduce the problem anymore.
Maybe the latest 2 or 3 selinux updates somehow changed something and the issue
seems to be solved or not visible anymore.

How should we proceed?
Leave it like it is and wait if it happens again?
Thanks.
Comment 17 Maros Barabas 2007-01-24 04:15:23 EST
I close this bug, please reopen it when it happens again. Thanks

Note You need to log in before you can comment on or make changes to this bug.