Hide Forgot
Description of problem: Recently upgraded our servers from cyrus-sasl 2.1.22-4 to cyrus-sasl 2.1.22-5.el5_4.3. saslauthd has begun leaking memory when configured to use kerberos5 as the password checking mechanism. Version-Release number of selected component (if applicable): 2.1.22-5.el5_4.3 How reproducible: Every time Steps to Reproduce: 1. Configure /etc/sysconfig/saslauthd to use MECH=kerberos5 and FLAGS=-n1 2. Run a loop like this one: for i in `perl -e'foreach (1..50) { print "x\n" };'`; do /usr/sbin/testsaslauthd -u alh -p'mypassword' ; ps wwwaux|grep sasl |grep -v grep ; done 3. The VSZ and RSS of the one running saslauthd process grow at a visible rate. Actual results: VSZ and RSS of saslauthd grow Expected results: VSZ and RSS should not grow. Additional info: Misc other RPMs were upgraded at the same time by doing a "yum update" so the problem code might not be in cyrus-sasl. All of the RPMs on the affected servers were brought up to date with the RHEL channel on Nov 20 2011. Example output of loop from Step 2 above: root 15913 1.7 0.0 59904 8636 ? Ss 14:42 0:04 /usr/sbin/saslauthd -m /var/run/saslauthd -a kerberos5 -n1 0: OK "Success." root 15913 1.7 0.0 59924 8652 ? Ss 14:42 0:04 /usr/sbin/saslauthd -m /var/run/saslauthd -a kerberos5 -n1 0: OK "Success." root 15913 1.7 0.0 59944 8672 ? Ss 14:42 0:04 /usr/sbin/saslauthd -m /var/run/saslauthd -a kerberos5 -n1 0: OK "Success." root 15913 1.7 0.0 59964 8692 ? Ss 14:42 0:04 /usr/sbin/saslauthd -m /var/run/saslauthd -a kerberos5 -n1 0: OK "Success." root 15913 1.7 0.0 59984 8712 ? Ss 14:42 0:04 /usr/sbin/saslauthd -m /var/run/saslauthd -a kerberos5 -n1 0: OK "Success." root 15913 1.7 0.0 60004 8732 ? Ss 14:42 0:04 /usr/sbin/saslauthd -m /var/run/saslauthd -a kerberos5 -n1 0: OK "Success." root 15913 1.7 0.0 60032 8752 ? Ss 14:42 0:04 /usr/sbin/saslauthd -m /var/run/saslauthd -a kerberos5 -n1 0: OK "Success." root 15913 1.7 0.0 60052 8772 ? Ss 14:42 0:04 /usr/sbin/saslauthd -m /var/run/saslauthd -a kerberos5 -n1 0: OK "Success."
Further testing on another machine (same OS, same packages) shows that the leak starts when nss is upgraded to nss-3.12.10-7.el5_7.
I'm able to reproduce this issue so far. Persists problem after update to latest nss?
This seems to be related to SELinux support in krb5-libs. There is no leak when there is selinux=0 on kernel command line.
Created attachment 569610 [details] valgrind output for saslauthd
I expect that the cost for labeling is going to be constant, though we never know if it's safe to free the labeling information since it's global and we're using it from a library (this is resolvable with later versions of libselinux, which adds APIs which lets libkrb5 "own" the memory it causes libselinux to allocate). If NSS is contributing to things, then I suspect it's coming in through the PKINIT plugin, which uses NSS, and which is loaded if your KDC advertises support for it. If you don't need that (saslauthd at least doesn't, since it's only in the business of checking passwords), the most expedient thing is to remove the pkinit-nss package, which prevents the client from loading the plugin.
(In reply to comment #4) > Created attachment 569610 [details] > valgrind output for saslauthd Can you re-run this with --show-reachable=yes, so that we can verify where the largest chunk of that memory is being allocated?
Created attachment 572257 [details] valgrind --leak-check=full --show-reachable=yes --log-file=saslauthd-valgrind.txt /usr/sbin/saslauthd -m /var/run/saslauthd -a kerberos5 -n1 -d
The majority of it is in matchpathcon(), which allocates memory globally, which means it's not safe for libkrb5 to free it. Later versions of libselinux add the selabel API, which can be safely cleaned up, for doing the same work, and libkrb5 switches to using that in EL 6. Until then, the more practical option is probably to enable caching with the -c flag, run authentication in a child process to isolate its memory use with -n 0, or both.
I removed pkinit-nss from our test server and that appears to fix the leak.
Okay, since the pkinit implementation is a different one in EL6, I'm going to mark this as resolved in the next release.