Bug 670822

+++ This bug was initially created as a clone of Bug #667654 +++

Description of problem:

I can mount two CIFS shares that are on the same server, each for a different user who has valid krb5 credentials. However only the user who's share is
mounted first can access his data. I experimented with '-o multiuser' and
/proc/fs/cifs/MultiuserMount but that changed nothing. cifs.upcall uses
the '-l' flag in request-key.conf.

What I do is to login as two users 'walteste' and 'suuchan' and get a TGT
for each of them. Then I do

# mount.cifs -o user=walteste,uid=walteste,sec=krb5 '//nas-nethz-users.d.ethz.ch/share-w-$' /mnt/1
# mount.cifs -o user=suuchan,uid=suuchan,sec=krb5 '//nas-nethz-users.d.ethz.ch/share-s-$' /mnt/2

as 'root' to mount the two shares. The one I mount first can be accessed by
the corresponding user (user sees files). The share mounted last is empty
(user sees no files).

What I notice is that with cifsFYI set to 3, I see that the first mount 
invokes cifs.upcall and it finds the ticket. The second mount does not
trigger cifs.upcall so I assume the kernel thinks it already has the right
key even though it has not. The reproducer below is for a case where the
second user does not even have a ticket so the issue may be in the kernel.


Version-Release number of selected component (if applicable):

cifs-utils-4.7-4.el6.i686
kernel-2.6.32-71.7.1.el6.i686

How reproducible:

Always


Steps to Reproduce:
1. Login with user a and get a TGT
2. mount a CIFS share from a server using 'sec=krb5,uid=a,user=a'
3. mount a CIFS share from a server using 'sec=krb5,uid=b,user=b'
  

Actual results:

Second share is mounted even though user b does not have a ticket.


Expected results:

Second mount should fail.

--- Additional comment from jlayton on 2011-01-06 08:45:03 EST ---

The new multiuser mount code should be in 6.1...

The cifs code tries to aggressively "share" different server objects between mounts. For krb5 SMB sessions, it does this by checking to see whether the credential uid (i.e. the real uid of the user doing the mount) is the same as that of an existing session. Also, the username field is unused for krb5 mounts (the server only cares about the krb5 ticket).

Because you're performing both mounts as the same user (root) it's matching the same session for both mounts. The idea here is that "borrowing" the credential cache of another user as cifs used to do is a bad thing. What we really need to know when doing a mount is whether the (real) user doing the mount has the proper credentials.

What I could use here is some clarification of your use case. How exactly are you planning to use this in practice? Doing a kinit as one user and performing the mount as another (root) seems a bit cumbersome...

--- Additional comment from walteste.ch on 2011-01-06 09:30:07 EST ---

Thanks for the clarifications. I saw you mentioning that the the multiuser
support will be in 6.1 elsewhere but didn't consider it relevant for this case.

What I actually want to do is use autofs to mount the shares. This seems to work for one user but we need this to work for multiple users on the same system.
Alternatively I was considering to use pam-script and mount the shares during
login but then again I would mount them as root to avoid the SUID bit for
mount.cifs.

--- Additional comment from jlayton on 2011-01-06 10:47:52 EST ---

Resetting component to kernel since that's where the problem is.

--- Additional comment from jlayton on 2011-01-06 11:06:08 EST ---

Another option to consider is pam_mount which is geared toward this sort of thing, but it looks like it hasn't made it into EPEL yet:

    https://admin.fedoraproject.org/pkgdb/acls/name/pam_mount

It might be worth opening a bug against it to see if the maintainer of it would build it for epel6.

Note that I gave a talk on cifs and krb5 at the RH summit this year, so you may want to look over that as you design this solution:

    https://access.redhat.com/knowledge/videos/player?id=013

autofs is rather problematic for this sort of thing. I assume that you were going to do something like have autofs pass in '-o uid=$UID'? If so, then how will you ensure that the "right" user is the one that walks into the mountpoint and triggers the mount?

We can probably solve this by adding a cred_uid= mount option that's only settable when the real uid is root. I'm not sure I'm entirely comfortable with such a solution however, and that would need to pass review upstream first.

--- Additional comment from walteste.ch on 2011-01-06 12:57:22 EST ---

I'll give pam_mount a try and watch the video tomorrow. Thanks for the input.

Autofs would have worked nicely for us because if we also use DFS we could
use an auto.home map like this:

  * -fstype=cifs,user=&,uid=&,gid=users,sec=krb5 ://d/dfs/users/all/&

The mounts are for home directories so dir == uid == user. The only thing
that could go wrong is that 'cifs.upcall -l' would not find a ticket if
someone tries to mount the home of another user who is not logged in and
only if autofs does some negative caching.

This could probably be addressed with a program map:

  #!/bin/sh
  user="$1";
  uid=`id -ru -- "$user"` || exit 1
  # <-- insert check here if there is a krb5cc_(uid)_* owned by (uid)
  initial=`echo "$user" | cut -c1`
  echo "-fstype=cifsuid=$user,gid=users,sec=krb5 \
  ://nas-nethz-users.d.ethz.ch/share-$initial-\$/$user"

Anyway, if you think that the multiuser support will solve all of this
I'll go for that as soon as 6.1 is there. Is there already a test kernel with
this feature around? 

The solution with a 'cred_uid' option would at least be consistent with the
'-l' flag for cifs.upcall because that flag makes only 50% sense if the kerner
does not call it. A 'legacy' option that would force the kernel to honour the 'user' option would also do.

BTW, I have seen FC13 systems that mount multiple CIFS shares as root during
login. Does that already have the multiuser code?

--- Additional comment from jlayton on 2011-01-06 13:14:43 EST ---

Multiuser mounts may solve this for you. You should be able to just mount this once:

   //d/dfs/users/all

...and all accesses to things under that mount would be done using the user's own krb5 creds. I have the code backported to RHEL6, so if you're willing to test it out that would be great. The best way would be for you to open a support case and get it attached to this bug, and then I'll have the support folks make the kernel available to you.

--- Additional comment from walteste.ch on 2011-01-06 14:19:43 EST ---

OK, case 00399988 opened. 

This sounds nice although that initial mount would probably require some
'machine' credentials (hidden password or keytab). I can live with that but
with our AD its fabulously easy to lock that account out. :(

--- Additional comment from jlayton on 2011-01-06 14:49:48 EST ---

Thanks for opening the case.

A keytab is probably your best bet, I need to verify however that cifs.upcall will access the keytab when the credcache is expired or not present however. The situation is a little different from NFS where root is generally mapped to the machine creds. Machine creds in AD have no capability for filesystem access, so that sort of rules them out for this purpose.

One thing I've been struggling with (even upstream) is trying to determine a "best-practice" for the credentials that root should use for mounting and accessing the share. I've generally just created a user that equates to "root". Note that that user needn't have any special permissions. It can even be mapped to a generic "nobody" user or something.

--- Additional comment from walteste.ch on 2011-01-07 06:15:49 EST ---

I did some testing on a FC13 system. This already has the multiuser code.
If I do the tests in the report it all works, *without* the multiuser option
and *without* the -l switch for cifs.upcall:

  #  mount.cifs -o user=walteste,uid=walteste,sec=krb5 \
  '//nas-nethz-users.d.ethz.ch/share-w-$' /mnt/1
  #  mount.cifs -o user=suuchan,uid=suuchan,sec=krb5 \
  '//nas-nethz-users.d.ethz.ch/share-s-$' /mnt/2

  * I can mount both shares as root and each of the users can access his files.

  * If one of the ticket is missing I get 'mount error(126): Required key
    not available' for the respective mount, even if one share is already
    mounted.

  * I can leave the '-l' option for cifs.upcall away and it will use the right
    ticket for each mount:

    cifs.upcall: find_krb5_cc: considering /tmp/krb5cc_501
    cifs.upcall: find_krb5_cc: FILE:/tmp/krb5cc_501 is valid ccache
    ...
    cifs.upcall: find_krb5_cc: considering /tmp/krb5cc_501
    cifs.upcall: find_krb5_cc: /tmp/krb5cc_501 is owned by 501, not 500

    According to https://bugzilla.redhat.com/show_bug.cgi?id=667382 that
    should have failed and use /tmp/krb5cc_0.

   * If root also has a ticket with rights to browse DFS, then automounting
     both shares with 

     * -fstype=cifs,user=&,uid=&,gid=users,sec=krb5 ://d/dfs/users/all/&

     also works (and autofs indeed caches failed mounts due to missing
     credentials for a short while).

It seems that the patches for the multiuser support fix the issues of this
bug as a side effect. One reason more for me to want this as soon as possible
in RHEL6. :)

--- Additional comment from jlayton on 2011-01-07 08:21:58 EST ---

(In reply to comment #13)
> I did some testing on a FC13 system. This already has the multiuser code.
> If I do the tests in the report it all works, *without* the multiuser option
> and *without* the -l switch for cifs.upcall:
> 

No, f13 does not have the multiuser code. It just now went upstream in 2.6.37. That kernel probably still matches smb sessions by username which is incorrect for krb5. That bug however is probably papering over the problem that you're hitting here.

--- Additional comment from walteste.ch on 2011-01-07 09:04:54 EST ---

OK, 'man mount.cifs' describes the 'multiuser' option so I assumed its implemented, especially because the stock rhel6 cifs-utils-4.4-5
does not have that section.

The FC13 kernel is 2.6.34.7-66, on RHEL6 its 2.6.32-71.7.1. If I understand you right they *should* both behave the same regarding the way the credential is
selected because neither has the new code. In 2.6.33 or 2.6.34 something must
have been added that changes the behavior in favor of what I want to do but that
is actually a bug.

Without knowing the internals of mount.cifs, the kernel and the problems of how
to correctly determine to mount as who, it is still my belief that when the
'user' option is specified with mount.cifs then the credentials of that user
should be used and not the one that were cached for the user 'root'. In this
respect I think the behavior if FC13 is correct and the one for RHEL6 is not.

What I do not understand yet is how CIFS mounts will behave in the future
if the 'multiuser' option is not used, e.g., for FC15 or RHEL7. Will it
behave like FC13 or RHEL 6.0?

Where I want to lead this is that even if the 'multiuser' functionality is
there you still want a more or less consistent behavior if you leave it off
(unless you make multiuser mandatory). If I can choose I would rather see
the FC13 behavior in the future and it would make sense to back-port it to
RHEL6.

Anyway, I'll give pam_mount a try but if the problem is in the kernel
then it shouldn't succeed where mount.cifs fails.

--- Additional comment from jlayton on 2011-01-07 10:08:20 EST ---

(In reply to comment #15)

Keeping the manpage and kernel in sync is pretty much impossible. It would be nice if cifs code in the kernel would reject options that it doesn't recognize but the option parsing code is a real mess.

It's not specifically the multiuser code that changed that behavior. It was some earlier patches. RHEL6 has those patches, and F13 does not.

The problem is this: we want to avoid establishing more than one SMB session per user, as that's a waste of resources on the server and client and causes a number of other problems that can lead to the client holding fewer oplocks. So, we need some mechanism to say that we want to use *this* session to talk to the server (and hence *this* set of credentials) and not that session.

Matching by username is problematic since we can't reliably count on the username being set to anything meaningful. The username is mostly ignored in the case of a krb5 mount. The only exception is the case of keytabs, but we don't even really *have* to use it for that. In the case of a session established from kernel space (the multiuser mount case) it definitely won't be set to anything meaningful.

What we really want to do is match existing sessions based on credential cache, but we are not able to easily get at that info in kernel space (we'd have to upcall for it). The best we can reasonably do is match by the uid of the credential cache owner.

Later revisions will act more like RHEL6.0 as it stands now, but that's not to say that we can't accommodate this use-case. Matching krb5 sessions based on username is wrong, even if it happened to work for you in this situation. The simplest fix is to add a "creduid" mount option, but I'd like to consider the problem a little more before I try to push anything upstream.

--- Additional comment from walteste.ch on 2011-01-07 15:36:16 EST ---

Thanks for the detailed background information, I understand now how this came
to happen.

I am all in favour of the 'creduid' option. Basically its just what I was
asking for. The only difference is that you propose to put it in a separate
option while I was assuming that the 'user' option will change namespace and semantic with the authentication scheme. For krb5 that can only be the uname
(or uid) of the local user having the tickets to use. That only root can
choose the user and for all others the UID of the caller must match the
'creduid' was implicit to me.

My only other point is that for root and root only the choice to cache the credentials for the caller is a bad idea. Of course that works for all users
who can only mount shares using their own credentials anyway. But you would
expect root to be able to mount on behalf of other local users.

Since mounting shares as root with sec=krb5 does not really work now in RHEL6
as it stands, how about a quick fix where you disable the aggressive caching
and just open a new session every time when root is the caller? That would leave
an admin at least the choice to use the feature as long as his servers
support the load. Of course I do not know what would break if you add an
'if (uid != 0) { }' arond the cache lookup code but it's an idea. It will also
not be a solution for people toying with SELinux and roles but there are not
too many of those. :)

--- Additional comment from jlayton on 2011-01-07 16:30:51 EST ---

> But you would expect root to be able to mount on behalf of other local users.

No, I don't expect that at all. The bottom line is that allowing anyone (even root) to "borrow" another user's credcache is just wrong. It's true that the krb5 implementation in cifs used to do that, but that was a bad design (and my fault too).

The root of the problem here however is *really* that cifs has had a built in design limitation of a single set of credentials per mount. That limitation has caused the implementation of a number of hacks to deal with it. The multiuser mount code changes that, but we've had to "shoehorn" that in to some degree since the code was not originally designed for it.

That said, people will still want to use the old single-credential per mount model for some time, so we need to be able to accommodate that use-case.

> Since mounting shares as root with sec=krb5 does not really work now in RHEL6
> as it stands, how about a quick fix where you disable the aggressive caching
> and just open a new session every time when root is the caller?

I'm steadfastly against putting in hacks that are not upstream. Down that path lies madness...

We'll fix this in RHEL6 eventually, but that fix needs to pass the muster upstream first.

The creduid option is probably a reasonable fix, but as I said, I need to consider the problem a little more. There may be a better approach that we just haven't thought of yet.

--- Additional comment from walteste.ch on 2011-01-07 18:21:27 EST ---

We should probably leave the discussion here, this could go on for a while. 

My goal (for which I have a quite tight deadline) is to get multiple CIFS 
home directories mounted on RHEL 6.0 workstations. For the moment the
simplest seems to use pam_script, make mount.cifs SUID and use 'su' to mount
the directory as the user. I'll opt for that because it will definitely work. 

If 'multiuser' support will make it into 6.1 I will be happy and rather
leave this issue unresolved and leave you more time to port Minshall+French
symlink support back into RHEL6. :)

--- Additional comment from jlayton on 2011-01-07 20:01:33 EST ---

Understood. It's technically a regression (at least upstream) so I am on the hook to fix this there anyway, but that may not happen within your timeframe. Once support hooks up the a.r.c to this BZ, I'll ask them to pass along a test kernel to you with all of the patches that I have queued up for 6.1 so far.

Also, mfsymlink support is already being proposed for 6.1 (see bug 651878). The test kernel should have that as well.

--- Additional comment from jlayton on 2011-01-10 09:02:37 EST ---

Created attachment 472594 [details]
patch -- add creduid= mount option

Here's the patch I'm considering.

I'll add this to my RHEL6 patch stack for testing as well.

--- Additional comment from jlayton on 2011-01-11 10:37:46 EST ---

Created attachment 472835 [details]
patch -- add cruid= mount option

That patch won't work -- mount.cifs allows for "abbreviations" and "creduid=" collides with "credentials=". So, I'll plan to propose this patch instead which does work. The mount option will be called 'cruid='.

--- Additional comment from jlayton on 2011-01-12 11:16:41 EST ---

Stefan, do you have a support case open against this (specific) problem? I'd like to pass on a kernel for you to test if you do...

--- Additional comment from walteste.ch on 2011-01-13 03:12:59 EST ---

I have opened case 00402003 where I explicitly ask for the --cruid option.
I didn;t expect this to be implemented so fast. A test kernel is warmly welcome.

--- Additional comment from jlayton on 2011-01-13 11:29:28 EST ---

I've asked the support folks to pass the kernel on to you. For now, you'll need to use a numeric uid in the cruid= option. Once cifs-utils has been updated you'll be able to use usernames in that field too. mount.cifs will do the conversion before passing them off to the kernel.

The kernel will also have multiuser and mfsymlink support if you want to play with those too.

--- Additional comment from walteste.ch on 2011-01-18 09:47:11 EST ---

Thanks a lot, I got the kernel. Together with cifs-utils-4.7-4 the mfsymlink and cruid options work just fine. We're now deploying some test systems to a wider
audience to see if things are robust but so far nothing was reported.