If possible, please leave this ticket public so that others might be able to find it and any solutions I might add to it.

Seems this because the default mount version of nfs is NFSv4.2 in Fedora
  and the "SELinux Labeled NFS support" is enabled in NFSv4.2

see: http://serverfault.com/questions/764140/wrong-selinux-labels-on-nfs-homes/764225

This bug is still private, unfortunately, but for the record:

On each file server I added a big load of selinux fcontext equivalences between each of /export/* and /home, then relabeled the lot.  It appears that the /home file contexts are essentially the same between RHEL7 and current Fedora, so this _should_ be OK and it does appear to be working.

One area of concern is that updating the NFS server resulted in surprising behavior. Updating from EL 7.2 to 7.3 enabled options that resulted in service outages. There were no warnings about this in the 7.3 Release Notes (thanks for checking on that, Jason!).

Disabling NFSv4.2 on the server after the upgrade (to mitigate the SELinux issue) could also have consequences for clients that have already mounted the server with NFSv4.2. Another option that was considered was remounting the server's exports with the context option, but that is also a disruptive remedy.

Would it be possible to leave NFSv4.2 disabled by default in EL 7.3? Or, disable NFSv4.2 in update scenarios, but enable it only for fresh installs? At the very least, a documentation update (what to expect, how to alleviate the issue) should be considered.

Has there been testing done to see how NFSv4.2-capable clients behave across updates from EL 7.2 to 7.3, and then reverts from 7.3 to 7.2 ?

I can't see why this should be private; fixed.

(In reply to Jason Tibbitts from comment #3)
> On each file server I added a big load of selinux fcontext equivalences
> between each of /export/* and /home, then relabeled the lot.  It appears
> that the /home file contexts are essentially the same between RHEL7 and
> current Fedora, so this _should_ be OK and it does appear to be working.

The server's just a passive container storing the contexts, so I'd think the more reliable approach would be to relabel from a client.

Doesn't this mean we're also imposing a new requirement that all the clients accessing a given filesystem have SELinux policies that are compatible in some way?

Anyway, maybe the correct solution is to keep labeled NFS off by default (and should it be off on the client too?).  I'd rather do that specifically for labeled NFS, as the other 4.2 features should be safe on upgrade, but if we need a quick solution then maybe reverting the default-to-4.2 change is right.

(In reply to Chuck Lever from comment #4)
> Has there been testing done to see how NFSv4.2-capable clients behave across
> updates from EL 7.2 to 7.3, and then reverts from 7.3 to 7.2 ?

Not that I know of.

If a client mounts a 7.3 server with 4.2 and then the server turns off 4.1 support--I doubt any client handles that gracefully.  Then again, I doubt distribution downgrades are something any distribution fully supports anyway, as this probably isn't the only difficulty.

(In reply to J. Bruce Fields from comment #5)
> I can't see why this should be private; fixed.

It got moved to the kernel components, and all tickets there are private by default (and can't be made public by the reporter).

> The server's just a passive container storing the contexts, so I'd think the
> more reliable approach would be to relabel from a client.

I don't know; that opens up all kinds of complexity.  Root can't in general do it, so the users would have to do it themselves or it would have to happen as part of the login process.  And running a relabel over the network can't be particularly fast.

Now, of course, in actual use the client will be the one to decide the policy on newly created files.  However, as far as I understand RFC7204-3.4.1, what matters is basically the set intersection of the policies supported on the client and server.  The client will only set a context it knows about, and the server will only allow that context to be set if it knows about it.

> Doesn't this mean we're also imposing a new requirement that all the clients
> accessing a given filesystem have SELinux policies that are compatible in
> some way?

Yes.  As I mentioned previously, it appears that current RHEL7 and Fedora will label things identically, but that's certainly not guaranteed to always be the case.  I don't know if there's any kind of commitment from the RHEL selinux folks about at least supporting newly added upstream policy labels.  I would also hope that the selinux folks have considered the possibility that the client policy might get too far ahead, and either degrade reasonably or provide mechanisms for supporting file servers with old policies.

It all gets a bit complicated for my limited comprehension, anyway.

> Anyway, maybe the correct solution is to keep labeled NFS off by default
> (and should it be off on the client too?).
 
Maybe the RHEL7 client, but in my case that would be a Fedora thing and I'd hope that they'd leave it on.  It's more about server policy and how the OS there expects to manage it in the face of evolving clients.  But in any case I assume (or at least I sure hope) that the selinux folks have thought long and hard about how they intend to handle this case.

(In reply to Jason Tibbitts from comment #6)
> I don't know; that opens up all kinds of complexity.  Root can't in general
> do it, so the users would have to do it themselves or it would have to
> happen as part of the login process.  And running a relabel over the network
> can't be particularly fast.
> 
> Now, of course, in actual use the client will be the one to decide the
> policy on newly created files.  However, as far as I understand
> RFC7204-3.4.1, what matters is basically the set intersection of the
> policies supported on the client and server.  The client will only set a
> context it knows about, and the server will only allow that context to be
> set if it knows about it.

RFC 7862 is the more up to date reference.  We implement what it calls "limited server mode": https://tools.ietf.org/html/rfc7862#section-9.5.2

I'm not actually sure whether the server rejects unknown contexts; the hooks that knfsd calls probably do permit SELinux to do that.

> Maybe the RHEL7 client, but in my case that would be a Fedora thing and I'd
> hope that they'd leave it on.  It's more about server policy and how the OS
> there expects to manage it in the face of evolving clients.  But in any case
> I assume (or at least I sure hope) that the selinux folks have thought long
> and hard about how they intend to handle this case.

I don't think this was really ever meant to be on by default.  The main motivation (as I remember it) was actually sVirt.  Opinions from SELinux people would be helpful; cc'ing dwalsh, hopefully he knows who.

But I think we need to a switch to turn server-side labeled NFS support on and off regardless.  I wonder if it should be per-export or server wide.  I think per-export is doable if it would be useful.

(In reply to J. Bruce Fields from comment #8)
> But I think we need to a switch to turn server-side labeled NFS support on
> and off regardless.  I wonder if it should be per-export or server wide.  I
> think per-export is doable if it would be useful.

Hi J. Bruce
  how about change default "SeLinux Policy of NFS"
  # setsebool -P use_nfs_home_dirs on
  *ref:
    https://linux.die.net/man/8/nfs_selinux
    http://serverfault.com/questions/764140/wrong-selinux-labels-on-nfs-homes/764225

That boolean has no relevance whatsoever to the current issue.  It has always been set on all of my clients, and was what worked _before_ the change to export v4.2 by default.

To simplify, with that boolean set, nfs_t is assumed to be equivalent to any of the home directory types.  With NFS v4.1 or lower, the client sees all files from an NFS mount mount as having type nfs_t, regardless of the type they have on the server, and so that boolean is how you make NFS home directories work.

With v4.2, the client sees the actual types assigned to the files on the server.  That isn't generally going to be nfs_t, so the boolean has no effect.

(In reply to Jason Tibbitts from comment #10)
> That boolean has no relevance whatsoever to the current issue.  It has
> always been set on all of my clients, and was what worked _before_ the
> change to export v4.2 by default.
> 
> To simplify, with that boolean set, nfs_t is assumed to be equivalent to any
> of the home directory types.  With NFS v4.1 or lower, the client sees all
> files from an NFS mount mount as having type nfs_t, regardless of the type
> they have on the server, and so that boolean is how you make NFS home
> directories work.
> 
> With v4.2, the client sees the actual types assigned to the files on the
> server.  That isn't generally going to be nfs_t, so the boolean has no
> effect.

Hi Jason
 Good to know that, thanks your explanation and patient.

You could of course fix the labels of the homedir, by running restorecon -R ~/

Dan,

As I wrote above, it's not quite that simple.

Who is supposed to run restorecon?  Where is it supposed to be run?  (Client or server, and if client, which one)?

Your use of '~' implies that you believe the user should do it.  But when the labels are wrong, the user can't even log in on a text VT (and of course they have no way to log in to the file server).  So they can't be the one to run restorecon, unless it happens very early in login.  The PAM stack is probably the only thing that happens early enough.  That's really not an operation you want to happen on every login.

Root can't do it on the client; it would only work if the filesystems were exported with sec=sys,no_root_squash.  I'm using kerberos.

So that leaves root running on the server.  Which as mentioned above is what I've done, by adding a bunch of fcontext equivalences. (Sadly patterns don't seem work there.)  But there are ongoing questions about whether anyone can expect restorecon run on an RHEL7 machine to label things such that Fedora (or any other distro which might be a client) will be happy.  The server has to know about any types before a client will be allowed to set them.

(It occurs to me that it would be interesting for the server to map any unknown types to nfs_t, so that the use_nfs_home_dirs boolean on the client could allow such unknown types to function.)

And in any case, none of this gets to the question of whether it was a good idea to turn this on in a point release without a release note, and whether that's something which needs to be undone.

NFS Support up to now, needs to be dominated by the Client.  Their is no support for the server (Last time I looked) to be able to verify the client context.  So we need to trust the client totally.

Can't differentiate the clients so whoever does it last wins.  With NFS label support it is up to the administrator of the systems to make sure they have equivalent policies on all systems.  So running a RHEL5 system and a RHEL7 system on the same himedir is not going to work.

Root could su USER, kinit user and then run restorecon.

I agree this should not be turned on in a point release, it should have been done in a major release.

Handling labeling between RHEL7 and Fedora could be difficult if labeling in the homedir has changed, and as you said, the labels on the client need to be understood on the server.  If the server is only an NFS server, you could disable SELinux on the server, and basically make it a dumb server.  But if you are attempting to share between to workstations, then it is up to you to maintain the labels, as you have.

No easy situation.  

By the way can't the client ask for no labeling, which would go back to the nfs_t?

(In reply to Daniel Walsh from comment #14)
> NFS Support up to now, needs to be dominated by the Client.  Their is no
> support for the server (Last time I looked) to be able to verify the client
> context.  So we need to trust the client totally.
> 
> Can't differentiate the clients so whoever does it last wins.  With NFS
> label support it is up to the administrator of the systems to make sure they
> have equivalent policies on all systems.  So running a RHEL5 system and a
> RHEL7 system on the same himedir is not going to work.
> 
> Root could su USER, kinit user and then run restorecon.
> 
> I agree this should not be turned on in a point release, it should have been
> done in a major release.

I don't think we'll want this on by default even in a future major release--I don't see any way to make the transition transparent, and it's just too common to serve the same export to heterogeneous clients.

How about requiring a "labeled" export option before turning on labeled nfs support on an export?

The problem there will be preexisting labeled nfs users, who will then see it turned off on future upgrade.  Are there few enough of those that we could get away with this?

If not, then we can revert to 4.2 off by default, introduce "labeled/nolabeled" export options with "labeled" as the default, then switch that default in a future major version, after some warning.  Ugh.

(In reply to J. Bruce Fields from comment #15)
> (In reply to Daniel Walsh from comment #14)
> > NFS Support up to now, needs to be dominated by the Client.  Their is no
> > support for the server (Last time I looked) to be able to verify the client
> > context.  So we need to trust the client totally.
> > 
> > Can't differentiate the clients so whoever does it last wins.  With NFS
> > label support it is up to the administrator of the systems to make sure they
> > have equivalent policies on all systems.  So running a RHEL5 system and a
> > RHEL7 system on the same himedir is not going to work.
> > 
> > Root could su USER, kinit user and then run restorecon.
> > 
> > I agree this should not be turned on in a point release, it should have been
> > done in a major release.
> 
> I don't think we'll want this on by default even in a future major
> release--I don't see any way to make the transition transparent, and it's
> just too common to serve the same export to heterogeneous clients.
> 
> How about requiring a "labeled" export option before turning on labeled nfs
> support on an export?
> 
> The problem there will be preexisting labeled nfs users, who will then see
> it turned off on future upgrade.  Are there few enough of those that we
> could get away with this?
make sense.
 and we might should broadcast this question in all redhat products teams
 to see if there's any product or solution that use labeled nfs feature.

> 
> If not, then we can revert to 4.2 off by default, introduce
> "labeled/nolabeled" export options with "labeled" as the default, then
> switch that default in a future major version, after some warning.  Ugh.

I am fine with this suggestion.  I see the major advantage to NFS labeling would be around MCS Situations.   Where you want to use NFS for storing content created by VM's or containers.  Having all containers/VMs able to write to nfs_t provides no isolation on breakout.  Having labeling support would prevent container1 from attacking container2s content stored on an NFS share.

Other situations would be for single layer shares which can be handled currently with mount context="" options.

NFS Homedirs can work, but the increase of labels and the chance that the saem homedir would be used on different versions of SELinux Policy is a problem as was stated above.

Created attachment 1236980 [details]
nfsd: opt in to labeled nfs per export

I believe the attached is all we need for the (upstream) kernel, but it will also need an nfs-utils patch and some testing.  Changelog:

    nfsd: opt in to labeled nfs per export
    
    Currently turning on NFSv4.2 results in 4.2 clients suddenly seeing the
    individual file labels as they're set on the server.  This is not what
    they've previously seen, and not appropriate in may cases.  (In
    particular, if clients have heterogenous security policies then one
    client's labels may not even make sense to another.)  Labeled NFS should
    be opted in only in those cases when the administrator knows it makes
    sense.
    
    It's helpful to be able to turn 4.2 on by default, and otherwise the
    protocol upgrade seems free of regressions.  So, default labeled NFS to
    off and provide an export flag to reenable it.
    
    Users wanting labeled NFS support on an export will henceforth need to:
    
            - make sure 4.2 support is enabled on client and server (as
              before), and
            - upgrade the server nfs-utils to a version supporting the new
              "security_label" export flag.
            - set that "security_label" flag on the export.
    
    This is commit may be seen as a regression to anyone currently depending
    on security labels.  We believe those cases are currently rare.

Created attachment 1237226 [details]
nfs-utils "security_label" patch

Corresponding nfs-utils patch.  Also untested--I probably won't get to that till next Monday, test results welcome if anyone else beats me to it.

For nfs-utils, I might suggest additionally an option to globally enable the new flag so that the current RHEL7.3 behavior can be achieved with a single addition to the RPCNFSDARGS line in /etc/sysconfig/nfs.

(In reply to Jason Tibbitts from comment #21)
> For nfs-utils, I might suggest additionally an option to globally enable the
> new flag so that the current RHEL7.3 behavior can be achieved with a single
> addition to the RPCNFSDARGS line in /etc/sysconfig/nfs.

So, that would mean they only have to touch one line in /etc/sysconfig/nfs instead of modifying potentially a lot of lines in /etc/exports.  They'll still have to update nfs-utils, though.  I'm not sure if that's such a big improvement.  And it's another configuration toggle to support long after this.

So, I'd rather not, but maybe it's an idea to keep in mind if this turns out to be a big pain for someone.

(In reply to J. Bruce Fields from comment #18)

> I believe the attached is all we need for the (upstream) kernel, but it will
> also need an nfs-utils patch and some testing.

Please track Bug 1435899 as the relevant user-space (nfs-utils) update.

Patch(es) committed on kernel repository and an interim kernel build is undergoing testing

Patch(es) available on kernel-3.10.0-658.el7

new export option works

[root@ibm-z-03 ~]# exportfs -s
/nfsshare  *(rw,sync,wdelay,hide,no_subtree_check,sec=sys,secure,no_root_squash,no_all_squash)
/nfsshare_labelled  *(rw,sync,wdelay,hide,no_subtree_check,security_label,sec=sys,secure,no_root_squash,no_all_squash)


filed new bugs bz1454617 bz1449877 to tracker new introduced issue

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:1842