Bug 1406885 - server supports labeled NFS by default
Summary: server supports labeled NFS by default
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel
Version: 7.3
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: ---
Assignee: J. Bruce Fields
QA Contact: JianHong Yin
Milan Navratil
URL:
Whiteboard:
Depends On: 1449877
Blocks: 1375259
TreeView+ depends on / blocked
 
Reported: 2016-12-21 18:20 UTC by Jason Tibbitts
Modified: 2018-09-07 07:53 UTC (History)
12 users (show)

Fixed In Version: kernel-3.10.0-658.el7
Doc Type: Bug Fix
Doc Text:
Labeled NFS is now turned off by default The SELinux labels on a Red hat Enterprise Linux NFS server are not normally visible to NFS clients. Instead, NFS clients see all files labeled as type `nfs_t` regardless of what label the files have on the server. Since Red Hat Enterprise Linux 7.3, the NFS server has the ability to communicate individual file labels to clients. Sufficiently recent clients, such as recent Fedora clients, see NFS files labeled with the same labels that those files have on the server. This is useful in certain cases, but it can also lead to unexpected access permission problems on recent clients after a server is upgraded to Red Hat Enterprise Linux 7.3 and later. Note that labeled NFS support is turned off by default on the NFS server. You can re-enable labeled NFS support by using the `security_label` export option.
Clone Of:
Environment:
Last Closed: 2017-08-02 05:00:51 UTC
Target Upstream Version:


Attachments (Terms of Use)
nfsd: opt in to labeled nfs per export (4.69 KB, patch)
2017-01-03 20:01 UTC, J. Bruce Fields
no flags Details | Diff
nfs-utils "security_label" patch (2.88 KB, patch)
2017-01-04 16:05 UTC, J. Bruce Fields
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1272142 unspecified CLOSED Turn on server NFS 4.2 by default 2020-10-14 00:28:05 UTC
Red Hat Bugzilla 1375259 high CLOSED NFSv4.1 as default NFS mount protocol 2020-10-14 00:28:05 UTC
Red Hat Bugzilla 1409950 unspecified CLOSED [RFE][exportfs] add new export option "security_label" to enable labeled nfs on server 2020-10-14 00:28:05 UTC
Red Hat Bugzilla 1435899 unspecified CLOSED exportfs: support "security_label" export option 2020-10-14 00:28:05 UTC
Red Hat Product Errata RHSA-2017:1842 normal SHIPPED_LIVE Important: kernel security, bug fix, and enhancement update 2017-08-01 18:22:09 UTC

Internal Links: 1272142 1375259 1409950 1435899

Description Jason Tibbitts 2016-12-21 18:20:02 UTC
Please forgive me if this is a kernel issue instead of an nfs-utils issue.  I'm not sure what actually governs the default set of exported NFS versions.

I just updated several file servers from 7.2 to 7.3 last night.  Everything appeared to be fine, except that it turns out that my users (who all have NFS-mounted home directories) could not log into their Fedora desktops because of selinux issues.  Nothing relevant changed on the clients.  

After debugging, I found that NFS v4.2 is now offered by default in EL7.3.  The clients now see the security label from the server for all mounts from these servers instead of nfs_t.  This was somewhat of a surprise.  I don't see any mention of this in the release notes, though it could be part of the pNFS tech preview or something like that.  In any event, if the enabling of v4.2 was intentional, the issue I'm seeing may be an unintended consequence

I'm running CentOS, but I see nothing which would indicate that this is a CentOS-specific issue.

    Updated     nfs-utils-1:1.3.0-0.21.el7_2.1.x86_64                    @updates
    Update                1:1.3.0-0.33.el7.x86_64                        @base


So it shows up in searches, here's the initial client-side selinux denial which keeps users from logging in (either from a graphical login or on a text VT):

type=AVC msg=audit(1482333895.794:287): avc:  denied  { search } for  pid=4228 comm="login" name="tester2" dev="0:53" ino=32394244 scontext=system_u:system_r:local_login_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=dir permissive=0


In case someone is interested in how I've worked around this:

For now my clients are all in permissive mode.  I haven't yet decided what to do about it, but I'm at least going to experiment with adding a set of selinux equivalencies and doing a big restorecon over all of my exported homedirs.  I have no idea if the RHEL and Linux policies are similar enough for that to actually work well, though.  Guess I'll find out.

Comment 1 Jason Tibbitts 2016-12-22 02:22:50 UTC
If possible, please leave this ticket public so that others might be able to find it and any solutions I might add to it.

Comment 2 JianHong Yin 2016-12-22 03:15:24 UTC
Seems this because the default mount version of nfs is NFSv4.2 in Fedora
  and the "SELinux Labeled NFS support" is enabled in NFSv4.2

see: http://serverfault.com/questions/764140/wrong-selinux-labels-on-nfs-homes/764225

Comment 3 Jason Tibbitts 2016-12-22 14:38:23 UTC
This bug is still private, unfortunately, but for the record:

On each file server I added a big load of selinux fcontext equivalences between each of /export/* and /home, then relabeled the lot.  It appears that the /home file contexts are essentially the same between RHEL7 and current Fedora, so this _should_ be OK and it does appear to be working.

Comment 4 Chuck Lever 2016-12-22 15:43:24 UTC
One area of concern is that updating the NFS server resulted in surprising behavior. Updating from EL 7.2 to 7.3 enabled options that resulted in service outages. There were no warnings about this in the 7.3 Release Notes (thanks for checking on that, Jason!).

Disabling NFSv4.2 on the server after the upgrade (to mitigate the SELinux issue) could also have consequences for clients that have already mounted the server with NFSv4.2. Another option that was considered was remounting the server's exports with the context option, but that is also a disruptive remedy.

Would it be possible to leave NFSv4.2 disabled by default in EL 7.3? Or, disable NFSv4.2 in update scenarios, but enable it only for fresh installs? At the very least, a documentation update (what to expect, how to alleviate the issue) should be considered.

Has there been testing done to see how NFSv4.2-capable clients behave across updates from EL 7.2 to 7.3, and then reverts from 7.3 to 7.2 ?

Comment 5 J. Bruce Fields 2016-12-22 16:04:52 UTC
I can't see why this should be private; fixed.

(In reply to Jason Tibbitts from comment #3)
> On each file server I added a big load of selinux fcontext equivalences
> between each of /export/* and /home, then relabeled the lot.  It appears
> that the /home file contexts are essentially the same between RHEL7 and
> current Fedora, so this _should_ be OK and it does appear to be working.

The server's just a passive container storing the contexts, so I'd think the more reliable approach would be to relabel from a client.

Doesn't this mean we're also imposing a new requirement that all the clients accessing a given filesystem have SELinux policies that are compatible in some way?

Anyway, maybe the correct solution is to keep labeled NFS off by default (and should it be off on the client too?).  I'd rather do that specifically for labeled NFS, as the other 4.2 features should be safe on upgrade, but if we need a quick solution then maybe reverting the default-to-4.2 change is right.

(In reply to Chuck Lever from comment #4)
> Has there been testing done to see how NFSv4.2-capable clients behave across
> updates from EL 7.2 to 7.3, and then reverts from 7.3 to 7.2 ?

Not that I know of.

If a client mounts a 7.3 server with 4.2 and then the server turns off 4.1 support--I doubt any client handles that gracefully.  Then again, I doubt distribution downgrades are something any distribution fully supports anyway, as this probably isn't the only difficulty.

Comment 6 Jason Tibbitts 2016-12-22 16:45:03 UTC
(In reply to J. Bruce Fields from comment #5)
> I can't see why this should be private; fixed.

It got moved to the kernel components, and all tickets there are private by default (and can't be made public by the reporter).

> The server's just a passive container storing the contexts, so I'd think the
> more reliable approach would be to relabel from a client.

I don't know; that opens up all kinds of complexity.  Root can't in general do it, so the users would have to do it themselves or it would have to happen as part of the login process.  And running a relabel over the network can't be particularly fast.

Now, of course, in actual use the client will be the one to decide the policy on newly created files.  However, as far as I understand RFC7204-3.4.1, what matters is basically the set intersection of the policies supported on the client and server.  The client will only set a context it knows about, and the server will only allow that context to be set if it knows about it.

> Doesn't this mean we're also imposing a new requirement that all the clients
> accessing a given filesystem have SELinux policies that are compatible in
> some way?

Yes.  As I mentioned previously, it appears that current RHEL7 and Fedora will label things identically, but that's certainly not guaranteed to always be the case.  I don't know if there's any kind of commitment from the RHEL selinux folks about at least supporting newly added upstream policy labels.  I would also hope that the selinux folks have considered the possibility that the client policy might get too far ahead, and either degrade reasonably or provide mechanisms for supporting file servers with old policies.

It all gets a bit complicated for my limited comprehension, anyway.

> Anyway, maybe the correct solution is to keep labeled NFS off by default
> (and should it be off on the client too?).
 
Maybe the RHEL7 client, but in my case that would be a Fedora thing and I'd hope that they'd leave it on.  It's more about server policy and how the OS there expects to manage it in the face of evolving clients.  But in any case I assume (or at least I sure hope) that the selinux folks have thought long and hard about how they intend to handle this case.

Comment 7 J. Bruce Fields 2016-12-22 17:25:17 UTC
(In reply to Jason Tibbitts from comment #6)
> I don't know; that opens up all kinds of complexity.  Root can't in general
> do it, so the users would have to do it themselves or it would have to
> happen as part of the login process.  And running a relabel over the network
> can't be particularly fast.
> 
> Now, of course, in actual use the client will be the one to decide the
> policy on newly created files.  However, as far as I understand
> RFC7204-3.4.1, what matters is basically the set intersection of the
> policies supported on the client and server.  The client will only set a
> context it knows about, and the server will only allow that context to be
> set if it knows about it.

RFC 7862 is the more up to date reference.  We implement what it calls "limited server mode": https://tools.ietf.org/html/rfc7862#section-9.5.2

I'm not actually sure whether the server rejects unknown contexts; the hooks that knfsd calls probably do permit SELinux to do that.

> Maybe the RHEL7 client, but in my case that would be a Fedora thing and I'd
> hope that they'd leave it on.  It's more about server policy and how the OS
> there expects to manage it in the face of evolving clients.  But in any case
> I assume (or at least I sure hope) that the selinux folks have thought long
> and hard about how they intend to handle this case.

I don't think this was really ever meant to be on by default.  The main motivation (as I remember it) was actually sVirt.  Opinions from SELinux people would be helpful; cc'ing dwalsh, hopefully he knows who.

Comment 8 J. Bruce Fields 2016-12-22 17:28:29 UTC
But I think we need to a switch to turn server-side labeled NFS support on and off regardless.  I wonder if it should be per-export or server wide.  I think per-export is doable if it would be useful.

Comment 9 JianHong Yin 2016-12-23 02:42:52 UTC
(In reply to J. Bruce Fields from comment #8)
> But I think we need to a switch to turn server-side labeled NFS support on
> and off regardless.  I wonder if it should be per-export or server wide.  I
> think per-export is doable if it would be useful.

Hi J. Bruce
  how about change default "SeLinux Policy of NFS"
  # setsebool -P use_nfs_home_dirs on
  *ref:
    https://linux.die.net/man/8/nfs_selinux
    http://serverfault.com/questions/764140/wrong-selinux-labels-on-nfs-homes/764225

Comment 10 Jason Tibbitts 2016-12-23 07:14:54 UTC
That boolean has no relevance whatsoever to the current issue.  It has always been set on all of my clients, and was what worked _before_ the change to export v4.2 by default.

To simplify, with that boolean set, nfs_t is assumed to be equivalent to any of the home directory types.  With NFS v4.1 or lower, the client sees all files from an NFS mount mount as having type nfs_t, regardless of the type they have on the server, and so that boolean is how you make NFS home directories work.

With v4.2, the client sees the actual types assigned to the files on the server.  That isn't generally going to be nfs_t, so the boolean has no effect.

Comment 11 JianHong Yin 2016-12-23 07:28:12 UTC
(In reply to Jason Tibbitts from comment #10)
> That boolean has no relevance whatsoever to the current issue.  It has
> always been set on all of my clients, and was what worked _before_ the
> change to export v4.2 by default.
> 
> To simplify, with that boolean set, nfs_t is assumed to be equivalent to any
> of the home directory types.  With NFS v4.1 or lower, the client sees all
> files from an NFS mount mount as having type nfs_t, regardless of the type
> they have on the server, and so that boolean is how you make NFS home
> directories work.
> 
> With v4.2, the client sees the actual types assigned to the files on the
> server.  That isn't generally going to be nfs_t, so the boolean has no
> effect.

Hi Jason
 Good to know that, thanks your explanation and patient.

Comment 12 Daniel Walsh 2016-12-23 10:01:41 UTC
You could of course fix the labels of the homedir, by running restorecon -R ~/

Comment 13 Jason Tibbitts 2016-12-23 16:49:24 UTC
Dan,

As I wrote above, it's not quite that simple.

Who is supposed to run restorecon?  Where is it supposed to be run?  (Client or server, and if client, which one)?

Your use of '~' implies that you believe the user should do it.  But when the labels are wrong, the user can't even log in on a text VT (and of course they have no way to log in to the file server).  So they can't be the one to run restorecon, unless it happens very early in login.  The PAM stack is probably the only thing that happens early enough.  That's really not an operation you want to happen on every login.

Root can't do it on the client; it would only work if the filesystems were exported with sec=sys,no_root_squash.  I'm using kerberos.

So that leaves root running on the server.  Which as mentioned above is what I've done, by adding a bunch of fcontext equivalences. (Sadly patterns don't seem work there.)  But there are ongoing questions about whether anyone can expect restorecon run on an RHEL7 machine to label things such that Fedora (or any other distro which might be a client) will be happy.  The server has to know about any types before a client will be allowed to set them.

(It occurs to me that it would be interesting for the server to map any unknown types to nfs_t, so that the use_nfs_home_dirs boolean on the client could allow such unknown types to function.)

And in any case, none of this gets to the question of whether it was a good idea to turn this on in a point release without a release note, and whether that's something which needs to be undone.

Comment 14 Daniel Walsh 2016-12-24 11:24:41 UTC
NFS Support up to now, needs to be dominated by the Client.  Their is no support for the server (Last time I looked) to be able to verify the client context.  So we need to trust the client totally.

Can't differentiate the clients so whoever does it last wins.  With NFS label support it is up to the administrator of the systems to make sure they have equivalent policies on all systems.  So running a RHEL5 system and a RHEL7 system on the same himedir is not going to work.

Root could su USER, kinit user and then run restorecon.

I agree this should not be turned on in a point release, it should have been done in a major release.

Handling labeling between RHEL7 and Fedora could be difficult if labeling in the homedir has changed, and as you said, the labels on the client need to be understood on the server.  If the server is only an NFS server, you could disable SELinux on the server, and basically make it a dumb server.  But if you are attempting to share between to workstations, then it is up to you to maintain the labels, as you have.

No easy situation.  

By the way can't the client ask for no labeling, which would go back to the nfs_t?

Comment 15 J. Bruce Fields 2016-12-24 23:04:33 UTC
(In reply to Daniel Walsh from comment #14)
> NFS Support up to now, needs to be dominated by the Client.  Their is no
> support for the server (Last time I looked) to be able to verify the client
> context.  So we need to trust the client totally.
> 
> Can't differentiate the clients so whoever does it last wins.  With NFS
> label support it is up to the administrator of the systems to make sure they
> have equivalent policies on all systems.  So running a RHEL5 system and a
> RHEL7 system on the same himedir is not going to work.
> 
> Root could su USER, kinit user and then run restorecon.
> 
> I agree this should not be turned on in a point release, it should have been
> done in a major release.

I don't think we'll want this on by default even in a future major release--I don't see any way to make the transition transparent, and it's just too common to serve the same export to heterogeneous clients.

How about requiring a "labeled" export option before turning on labeled nfs support on an export?

The problem there will be preexisting labeled nfs users, who will then see it turned off on future upgrade.  Are there few enough of those that we could get away with this?

If not, then we can revert to 4.2 off by default, introduce "labeled/nolabeled" export options with "labeled" as the default, then switch that default in a future major version, after some warning.  Ugh.

Comment 16 JianHong Yin 2016-12-26 03:14:31 UTC
(In reply to J. Bruce Fields from comment #15)
> (In reply to Daniel Walsh from comment #14)
> > NFS Support up to now, needs to be dominated by the Client.  Their is no
> > support for the server (Last time I looked) to be able to verify the client
> > context.  So we need to trust the client totally.
> > 
> > Can't differentiate the clients so whoever does it last wins.  With NFS
> > label support it is up to the administrator of the systems to make sure they
> > have equivalent policies on all systems.  So running a RHEL5 system and a
> > RHEL7 system on the same himedir is not going to work.
> > 
> > Root could su USER, kinit user and then run restorecon.
> > 
> > I agree this should not be turned on in a point release, it should have been
> > done in a major release.
> 
> I don't think we'll want this on by default even in a future major
> release--I don't see any way to make the transition transparent, and it's
> just too common to serve the same export to heterogeneous clients.
> 
> How about requiring a "labeled" export option before turning on labeled nfs
> support on an export?
> 
> The problem there will be preexisting labeled nfs users, who will then see
> it turned off on future upgrade.  Are there few enough of those that we
> could get away with this?
make sense.
 and we might should broadcast this question in all redhat products teams
 to see if there's any product or solution that use labeled nfs feature.

> 
> If not, then we can revert to 4.2 off by default, introduce
> "labeled/nolabeled" export options with "labeled" as the default, then
> switch that default in a future major version, after some warning.  Ugh.

Comment 17 Daniel Walsh 2016-12-26 11:55:20 UTC
I am fine with this suggestion.  I see the major advantage to NFS labeling would be around MCS Situations.   Where you want to use NFS for storing content created by VM's or containers.  Having all containers/VMs able to write to nfs_t provides no isolation on breakout.  Having labeling support would prevent container1 from attacking container2s content stored on an NFS share.

Other situations would be for single layer shares which can be handled currently with mount context="" options.

NFS Homedirs can work, but the increase of labels and the chance that the saem homedir would be used on different versions of SELinux Policy is a problem as was stated above.

Comment 18 J. Bruce Fields 2017-01-03 20:01:52 UTC
Created attachment 1236980 [details]
nfsd: opt in to labeled nfs per export

I believe the attached is all we need for the (upstream) kernel, but it will also need an nfs-utils patch and some testing.  Changelog:

    nfsd: opt in to labeled nfs per export
    
    Currently turning on NFSv4.2 results in 4.2 clients suddenly seeing the
    individual file labels as they're set on the server.  This is not what
    they've previously seen, and not appropriate in may cases.  (In
    particular, if clients have heterogenous security policies then one
    client's labels may not even make sense to another.)  Labeled NFS should
    be opted in only in those cases when the administrator knows it makes
    sense.
    
    It's helpful to be able to turn 4.2 on by default, and otherwise the
    protocol upgrade seems free of regressions.  So, default labeled NFS to
    off and provide an export flag to reenable it.
    
    Users wanting labeled NFS support on an export will henceforth need to:
    
            - make sure 4.2 support is enabled on client and server (as
              before), and
            - upgrade the server nfs-utils to a version supporting the new
              "security_label" export flag.
            - set that "security_label" flag on the export.
    
    This is commit may be seen as a regression to anyone currently depending
    on security labels.  We believe those cases are currently rare.

Comment 20 J. Bruce Fields 2017-01-04 16:05:35 UTC
Created attachment 1237226 [details]
nfs-utils "security_label" patch

Corresponding nfs-utils patch.  Also untested--I probably won't get to that till next Monday, test results welcome if anyone else beats me to it.

Comment 21 Jason Tibbitts 2017-01-04 19:02:51 UTC
For nfs-utils, I might suggest additionally an option to globally enable the new flag so that the current RHEL7.3 behavior can be achieved with a single addition to the RPCNFSDARGS line in /etc/sysconfig/nfs.

Comment 22 J. Bruce Fields 2017-01-04 20:56:49 UTC
(In reply to Jason Tibbitts from comment #21)
> For nfs-utils, I might suggest additionally an option to globally enable the
> new flag so that the current RHEL7.3 behavior can be achieved with a single
> addition to the RPCNFSDARGS line in /etc/sysconfig/nfs.

So, that would mean they only have to touch one line in /etc/sysconfig/nfs instead of modifying potentially a lot of lines in /etc/exports.  They'll still have to update nfs-utils, though.  I'm not sure if that's such a big improvement.  And it's another configuration toggle to support long after this.

So, I'd rather not, but maybe it's an idea to keep in mind if this turns out to be a big pain for someone.

Comment 25 Yongcheng Yang 2017-03-27 08:31:01 UTC
(In reply to J. Bruce Fields from comment #18)

> I believe the attached is all we need for the (upstream) kernel, but it will
> also need an nfs-utils patch and some testing.

Please track Bug 1435899 as the relevant user-space (nfs-utils) update.

Comment 26 Rafael Aquini 2017-04-27 03:56:34 UTC
Patch(es) committed on kernel repository and an interim kernel build is undergoing testing

Comment 28 Rafael Aquini 2017-05-01 13:17:05 UTC
Patch(es) available on kernel-3.10.0-658.el7

Comment 41 JianHong Yin 2017-05-31 02:11:42 UTC
new export option works

[root@ibm-z-03 ~]# exportfs -s
/nfsshare  *(rw,sync,wdelay,hide,no_subtree_check,sec=sys,secure,no_root_squash,no_all_squash)
/nfsshare_labelled  *(rw,sync,wdelay,hide,no_subtree_check,security_label,sec=sys,secure,no_root_squash,no_all_squash)


filed new bugs bz1454617 bz1449877 to tracker new introduced issue

Comment 46 errata-xmlrpc 2017-08-02 05:00:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:1842


Note You need to log in before you can comment on or make changes to this bug.