After updating to kernel-3.19.1-201, gssproxy doesn't function properly on an NFSv4.2/KRB5 server: Debug Enabled Failed to write to /proc/net/rpc/use-gss-proxy: 12 (Cannot allocate memory) Client connected (fd = 11) (pid = 2174) (uid = 0) (gid = 0) (context = system_u:system_r:kernel_t:s0) Just in case, I've tried with both SELinux enforcing and permissive with the same result. Clients are unable to mount NFS filesystems from the server. Comment added here as well: https://admin.fedoraproject.org/updates/FEDORA-2015-4059/
GSS-Proxy hasn't changed, it seem it can't write to the kernel /proc interface anymore.
(In reply to Simo Sorce from comment #2) > GSS-Proxy hasn't changed, it seem it can't write to the kernel /proc > interface anymore. Do you have a strace trace floating around for this issue ?
(In reply to Roland Mainz from comment #3) > (In reply to Simo Sorce from comment #2) > > GSS-Proxy hasn't changed, it seem it can't write to the kernel /proc > > interface anymore. > > Do you have a strace trace floating around for this issue ? Unfortunately, I don't. I was not able to do much testing before reporting the bug as this server provides the /home directory over NFS so I had to reboot to the older kernel fairly quicky in order to keep up operationally. I may be able to try on another server after 3.19.2 reaches stable for Fedora 21.
Simo, Doesn't this take care of the problem? From 4a478ab15016f4690c9a76bd625eff6bb9b24a31 Mon Sep 17 00:00:00 2001 From: Simo Sorce <simo> Date: Sat, 21 Mar 2015 14:00:17 -0400 Subject: [PATCH] Suppress exported_composite_name for the kernel The kernel makes no use of this data, and ita causes allocation issues in some cases with waste of space on the kernel side. Fixes: https://fedorahosted.org/gss-proxy/ticket/129 --- proxy/src/gp_export.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/proxy/src/gp_export.c b/proxy/src/gp_export.c index b855c74c6b04d7a6a9d61e61a117ed18778b5278..1cd05396ff7b66d57f10b28437b6bf6d26381fd5 100644 --- a/proxy/src/gp_export.c +++ b/proxy/src/gp_export.c @@ -623,6 +623,12 @@ export: if (ret_maj) { goto done; } + /* suppress names exported_composite_name, the kernel doesn't want + * this information */ + xdr_free((xdrproc_t)xdr_gssx_buffer, + (char *)&out->src_name.exported_composite_name); + xdr_free((xdrproc_t)xdr_gssx_buffer, + (char *)&out->targ_name.exported_composite_name); break; default: ret_maj = GSS_S_FAILURE;
(In reply to Steve Dickson from comment #5) > Simo, > > Doesn't this take care of the problem? [snip] > + /* suppress names exported_composite_name, the kernel doesn't want > + * this information */ > + xdr_free((xdrproc_t)xdr_gssx_buffer, > + (char *)&out->src_name.exported_composite_name); > + xdr_free((xdrproc_t)xdr_gssx_buffer, > + (char *)&out->targ_name.exported_composite_name); > break; > default: > ret_maj = GSS_S_FAILURE; AFAIK it does fix that specific problem ...
Steve, I do not know if it is the same problem. Worth testing though.
Just tested with kernel-4.0.0-0.rc4.git0.1.fc22.x86_64 to no avail; the failure appears to be the same.
Terrible workaround seems be to stick echo 1 > /proc/net/rpc/use-gss-proxy in rc.local (and make sure systemd runs that if it doesn't happen by default). Or you can disable selinux. Either appears to work for me. I do not know why disabling selinux works; even in permissive mode it logs exactly one AVC: type=AVC msg=audit(1427225711.123:41): avc: denied { search } for pid=638 comm="gssproxy" name="rpc_xprt" dev="debugfs" ino=11728 scontext=system_u:system_r:gssproxy_t:s0 tcontext=system_u:object_r:debugfs_t:s0 tclass=dir permissive=1 This is an attempt to dig into /sys/kernel/debug/sunrpc/rpc_xprt. Not sure whether that's important somehow. Things also run fine if you run gssproxy under strace. There's a long debugging session in #fedora-kernel which I can provide if someone cares.
CC'ing some selinux folks because somehow disabling selinux makes a difference.
Huh. In fact, that debugfs thing is it. Not something gssproxy is doing, but something the rpc layer does when it creates a new client: a printk added to rpc_xprt_debugfs_register shows the debugfs_create_dir there failing.
So this is probably a regression due to 388f0c776781 "sunrpc: add a debugfs rpc_xprt directory with an info file in it". I'm not sure what it should be doing instead.
Huh, that is odd. I guess the avc message might be a clue. debugfs_create_dir calls start_creating, which tries to do lookup_one_len. I suspect that's failing for some reason with SELinux enabled. It might be worth instrumenting start_creating to see whether that's the problem.
(In reply to Jeff Layton from comment #13) > Huh, that is odd. I guess the avc message might be a clue. > debugfs_create_dir calls start_creating, which tries to do lookup_one_len. I > suspect that's failing for some reason with SELinux enabled. Hm, yes, it does do inode_permission. That seems wrong--a system call that creates some debugfs stuff as a side-effect shouldn't care about the caller's permissions on debugfs. As far as I can tell, proc_mkdir, for example, doesn't do any permission checks. > It might be worth instrumenting start_creating to see whether that's the > problem. But I haven't actually tested anything yet.
When testing with gssproxy-0.4.0-1.fc21 and kernel-3.19.3-200.fc21.x86_64, the issue remains unresolved--gssproxy doesn't work on the NFS server unless I revert to kernel-3.18.9-200.fc21.x86_64.
The new gssproxy wasn't intended to fix this and should have no effect. I don't think this can be reasonably fixed in gssproxy, though I suppose it could be worked around in the selinux policy.
(In reply to Jason Tibbitts from comment #16) > The new gssproxy wasn't intended to fix this and should have no effect. I > don't think this can be reasonably fixed in gssproxy, though I suppose it > could be worked around in the selinux policy. Right, this is more likely a kernel bug, rather than a gssproxy bug, but I wanted to report the updated versions. Jason, I don't have the same experience with SELinux that you do. Whether or not I have SELinux enforcing or permissive, I am unable to write to /proc/net/rpc/use-gss-proxy, even when I try (as root) to echo 1 > /proc/net/rpc/use-gss-proxy Bruce, do you have any suggestions for the mere mortal user?
One fun thing is that if /proc/net/rpc/use-gss-proxy isn't currently set to "-1", you can't write to it no matter what as the kernel prevents it from being changed more than once. So if it somehow gets set to "0", maybe by rpc.svcgssd starting instead of gssproxy, then you will be unable to change it, selinux or not. So be sure to cat it to make sure it's currently -1, and if that's the case and you still can't write to it then something else is going on. And I am far and away a mere mortal user; I just happened to learn a fair bit about it while blowing a few hours trying to understand what was happening. In any case, the hacks/workarounds I tried did both work well and repeatedly for me. Once I'm back in the office I can try making a tiny selinux policy to allow the access and see what happens. (Which, lest anyone doubt my pleb user status, involves a fairly light understanding of how to run audit2allow.)
This works for me: http://lkml.kernel.org/r/20150330142310.GB6901@fieldses.org The lookup logic I copied more-or-less blindly from someone else, I probably need to look at it a little more carefully, and get some review from somebody else.
(In reply to Anthony Messina from comment #17) > Bruce, do you have any suggestions for the mere mortal user? So, I'm pretty sure I've found the problem (I was able to reproduce it and that patch fixed it for me), but confirming that by testing the kernel patch would be useful if possible.
Bruce, could you spin a kernel scratch build for Fedora 21/22 with the patch ?
If someone wants to test immediately, I built a kernel with this patch locally. https://www.math.uh.edu/~tibbs/gssproxy-kernel/ They're signed by my key, which you probably don't trust anyway. The patch does appear to fix the issue, for me at least. The selinux denial is gone, selinux is enforcing, and I'm not manually tweaking /proc/net/rpc/use-gss-proxy anywhere.
So, what are the chances this could make it into a Fedora kernel any time soon? I guess the linux-fsdevel mailing list discussion is still going.
I just posted a v2 version of a patch that should "fix" this to some degree. If that's merged it should work around the problem. I sent it to Trond since he merged the original debugfs patches for sunrpc, but I imagine he won't mind if Bruce wants to shepherd them in.
Is there any point in me rolling another kernel build with Jeff's v2 patch in it?
(In reply to Jason Tibbitts from comment #25) > Is there any point in me rolling another kernel build with Jeff's v2 patch > in it? I'm pretty sure it's right (it fixes the problem for me). It should show up in Linus's tree this week, I think, and then the stable people pick it up and then Fedora usually seems to get the latest stable fairly quickly--but I don't know the schedule. (In reply to Simo Sorce from comment #21) > could you spin a kernel scratch build for Fedora 21/22 with the patch ? I don't think I'll get around to that any faster, unless maybe somebody wants to point me to a simple recipe.
I'll bring it back as soon as it lands in Linus' tree.
I think this landed in Linus's tree: commit f9c72d10d6fbf949558cd088389a42213ed7b12d Author: Jeff Layton <jlayton> Date: Tue Mar 31 12:03:28 2015 -0400 sunrpc: make debugfs file creation failure non-fatal [...] Fixes: 388f0c776781 "sunrpc: add a debugfs rpc_xprt directory..." Cc: stable.org Signed-off-by: Jeff Layton <jeff.layton> Acked-by: Greg Kroah-Hartman <gregkh> Signed-off-by: J. Bruce Fields <bfields> But I don't see it in stable as of 3.19.4 unless I just can't drive git properly.
kernel-3.19.4-200.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/kernel-3.19.4-200.fc21
kernel-3.19.4-100.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/kernel-3.19.4-100.fc20
I pulled 3.19.4-200.fc21 from koji and tested it; this bug appears to be fixed. Karma given.
\o/ Thanks a lot everyone for the testing and solution.
Package kernel-3.19.4-100.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing kernel-3.19.4-100.fc20' as soon as you are able to, then reboot. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2015-6294/kernel-3.19.4-100.fc20 then log in and leave karma (feedback).
kernel-3.19.4-100.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
kernel-3.19.4-200.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.