Created attachment 518370 [details]
Patch to init script to start memcached as user specified in sysconfig/memcached

Description of problem:

If the 'MAXCONN' variable in /etc/sysconfig/memcached is set to a value greater than 1024 on a stock RHEL 6.1 install, attempting to start memcached will fail with the message:

failed to set rlimit for open files. Try running as root or requesting smaller maxconns value.

However, when the nofile limits for the memcached user are increased in /etc/security/limits.conf like this:

memcached       hard    nofile          65536
memcached       soft    nofile          65536

The problem continues. Looking closer, there is an avc denial logged:

Aug 15 13:25:26 cache101 audispd: node=cache101 type=AVC msg=audit(1313439926.993:45010): avc:  denied  { sys_resource } for  pid=19815 comm="memcached" capability=24  scontext=unconfined_u:system_r:memcached_t:s0 tcontext=unconfined_u:system_r:memcached_t:s0 tclass=capability

And here's the actual failing syscall (from strace):

[pid 24837] getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
[pid 24837] setrlimit(RLIMIT_NOFILE, {rlim_cur=4*1024, rlim_max=4*1024}) = -1 EPERM (Operation not permitted)
[pid 24837] write(2, "failed to set rlimit for open files. Try running as root or requesting smaller maxconns value.\n", 95failed to set rlimit for open files. Try running as root or requesting smaller maxconns value.
) = 95

A quick look at memcached.c and the included init script confirm that redhat starts the memcached binary as root, and that memcached does a setuid() immediately after the failing setrlimit. The problem here is that since it is started as root, the setrlimit is run as root, and since root's nofiles limit is still 1024, it fails. in the absence of SELinux it would normally succeed, by (ab)using the CAP_SYS_RESOURCE capability to override limits since it's a root owned process. My first thought was just to reloacte the setuid() call before the setrlimit() calls, but since limits.conf is only sourced on a new session, this doesn't help (the setuid() doesn't create a new session, so nofile is still set to 1024).

One possible fix would be to allow memcached the sys_resource capability in the selinux policy, but this seems to be the wrong way to go in general. I'm not entirely clear on RedHat's stance on policy changes, but I would think granting an application additional access that is technically doesn't need would be frowned upon.

A simple fix is adding nofiles entries to limits.conf for root, which is what we've done as a workaround. This works fine, but is unintuitive and likely to confuse some future admin. If this is the correct fix, I'd hope to at least see it documented somewhere.

One other option is simply starting the memcached program as the user specified in /etc/sysconfig/memcached. This requires a very simple change to the init script, and at least with the stock config I don't see any problems with it. If a user wanted to bind memcached to a non-ephemeral port this would be a problem, but that would require selinux policy changes anyway and so changing that may not be a big deal.

I've attached the (1 line) patch described above as a possible fix, and would like to know what Redhat's recommended long-term fix to this is.

Version-Release number of selected component (if applicable):
1.4.4-3.el6

How reproducible:
Always

Steps to Reproduce:
1. add nofiles entries to limits.conf for memcached user
2. increase MAXCONN in /etc/sysconfig/memcached to "4096"
3. attempt to (re)start memcached
  
Actual results:

failed to set rlimit for open files. Try running as root or requesting smaller maxconns value.
[ FAILED ]

Expected results:

[  OK  ]

Additional info: