The new multiuser mount code should be in 6.1...

The cifs code tries to aggressively "share" different server objects between mounts. For krb5 SMB sessions, it does this by checking to see whether the credential uid (i.e. the real uid of the user doing the mount) is the same as that of an existing session. Also, the username field is unused for krb5 mounts (the server only cares about the krb5 ticket).

Because you're performing both mounts as the same user (root) it's matching the same session for both mounts. The idea here is that "borrowing" the credential cache of another user as cifs used to do is a bad thing. What we really need to know when doing a mount is whether the (real) user doing the mount has the proper credentials.

What I could use here is some clarification of your use case. How exactly are you planning to use this in practice? Doing a kinit as one user and performing the mount as another (root) seems a bit cumbersome...

Thanks for the clarifications. I saw you mentioning that the the multiuser
support will be in 6.1 elsewhere but didn't consider it relevant for this case.

What I actually want to do is use autofs to mount the shares. This seems to work for one user but we need this to work for multiple users on the same system.
Alternatively I was considering to use pam-script and mount the shares during
login but then again I would mount them as root to avoid the SUID bit for
mount.cifs.

Resetting component to kernel since that's where the problem is.

Another option to consider is pam_mount which is geared toward this sort of thing, but it looks like it hasn't made it into EPEL yet:

    https://admin.fedoraproject.org/pkgdb/acls/name/pam_mount

It might be worth opening a bug against it to see if the maintainer of it would build it for epel6.

Note that I gave a talk on cifs and krb5 at the RH summit this year, so you may want to look over that as you design this solution:

    https://access.redhat.com/knowledge/videos/player?id=013

autofs is rather problematic for this sort of thing. I assume that you were going to do something like have autofs pass in '-o uid=$UID'? If so, then how will you ensure that the "right" user is the one that walks into the mountpoint and triggers the mount?

We can probably solve this by adding a cred_uid= mount option that's only settable when the real uid is root. I'm not sure I'm entirely comfortable with such a solution however, and that would need to pass review upstream first.

I'll give pam_mount a try and watch the video tomorrow. Thanks for the input.

Autofs would have worked nicely for us because if we also use DFS we could
use an auto.home map like this:

  * -fstype=cifs,user=&,uid=&,gid=users,sec=krb5 ://d/dfs/users/all/&

The mounts are for home directories so dir == uid == user. The only thing
that could go wrong is that 'cifs.upcall -l' would not find a ticket if
someone tries to mount the home of another user who is not logged in and
only if autofs does some negative caching.

This could probably be addressed with a program map:

  #!/bin/sh
  user="$1";
  uid=`id -ru -- "$user"` || exit 1
  # <-- insert check here if there is a krb5cc_(uid)_* owned by (uid)
  initial=`echo "$user" | cut -c1`
  echo "-fstype=cifsuid=$user,gid=users,sec=krb5 \
  ://nas-nethz-users.d.ethz.ch/share-$initial-\$/$user"

Anyway, if you think that the multiuser support will solve all of this
I'll go for that as soon as 6.1 is there. Is there already a test kernel with
this feature around? 

The solution with a 'cred_uid' option would at least be consistent with the
'-l' flag for cifs.upcall because that flag makes only 50% sense if the kerner
does not call it. A 'legacy' option that would force the kernel to honour the 'user' option would also do.

BTW, I have seen FC13 systems that mount multiple CIFS shares as root during
login. Does that already have the multiuser code?

Multiuser mounts may solve this for you. You should be able to just mount this once:

   //d/dfs/users/all

...and all accesses to things under that mount would be done using the user's own krb5 creds. I have the code backported to RHEL6, so if you're willing to test it out that would be great. The best way would be for you to open a support case and get it attached to this bug, and then I'll have the support folks make the kernel available to you.

OK, case 00399988 opened. 

This sounds nice although that initial mount would probably require some
'machine' credentials (hidden password or keytab). I can live with that but
with our AD its fabulously easy to lock that account out. :(

Thanks for opening the case.

A keytab is probably your best bet, I need to verify however that cifs.upcall will access the keytab when the credcache is expired or not present however. The situation is a little different from NFS where root is generally mapped to the machine creds. Machine creds in AD have no capability for filesystem access, so that sort of rules them out for this purpose.

One thing I've been struggling with (even upstream) is trying to determine a "best-practice" for the credentials that root should use for mounting and accessing the share. I've generally just created a user that equates to "root". Note that that user needn't have any special permissions. It can even be mapped to a generic "nobody" user or something.

I did some testing on a FC13 system. This already has the multiuser code.
If I do the tests in the report it all works, *without* the multiuser option
and *without* the -l switch for cifs.upcall:

  #  mount.cifs -o user=walteste,uid=walteste,sec=krb5 \
  '//nas-nethz-users.d.ethz.ch/share-w-$' /mnt/1
  #  mount.cifs -o user=suuchan,uid=suuchan,sec=krb5 \
  '//nas-nethz-users.d.ethz.ch/share-s-$' /mnt/2

  * I can mount both shares as root and each of the users can access his files.

  * If one of the ticket is missing I get 'mount error(126): Required key
    not available' for the respective mount, even if one share is already
    mounted.

  * I can leave the '-l' option for cifs.upcall away and it will use the right
    ticket for each mount:

    cifs.upcall: find_krb5_cc: considering /tmp/krb5cc_501
    cifs.upcall: find_krb5_cc: FILE:/tmp/krb5cc_501 is valid ccache
    ...
    cifs.upcall: find_krb5_cc: considering /tmp/krb5cc_501
    cifs.upcall: find_krb5_cc: /tmp/krb5cc_501 is owned by 501, not 500

    According to https://bugzilla.redhat.com/show_bug.cgi?id=667382 that
    should have failed and use /tmp/krb5cc_0.

   * If root also has a ticket with rights to browse DFS, then automounting
     both shares with 

     * -fstype=cifs,user=&,uid=&,gid=users,sec=krb5 ://d/dfs/users/all/&

     also works (and autofs indeed caches failed mounts due to missing
     credentials for a short while).

It seems that the patches for the multiuser support fix the issues of this
bug as a side effect. One reason more for me to want this as soon as possible
in RHEL6. :)

(In reply to comment #13)
> I did some testing on a FC13 system. This already has the multiuser code.
> If I do the tests in the report it all works, *without* the multiuser option
> and *without* the -l switch for cifs.upcall:
> 

No, f13 does not have the multiuser code. It just now went upstream in 2.6.37. That kernel probably still matches smb sessions by username which is incorrect for krb5. That bug however is probably papering over the problem that you're hitting here.

OK, 'man mount.cifs' describes the 'multiuser' option so I assumed its implemented, especially because the stock rhel6 cifs-utils-4.4-5
does not have that section.

The FC13 kernel is 2.6.34.7-66, on RHEL6 its 2.6.32-71.7.1. If I understand you right they *should* both behave the same regarding the way the credential is
selected because neither has the new code. In 2.6.33 or 2.6.34 something must
have been added that changes the behavior in favor of what I want to do but that
is actually a bug.

Without knowing the internals of mount.cifs, the kernel and the problems of how
to correctly determine to mount as who, it is still my belief that when the
'user' option is specified with mount.cifs then the credentials of that user
should be used and not the one that were cached for the user 'root'. In this
respect I think the behavior if FC13 is correct and the one for RHEL6 is not.

What I do not understand yet is how CIFS mounts will behave in the future
if the 'multiuser' option is not used, e.g., for FC15 or RHEL7. Will it
behave like FC13 or RHEL 6.0?

Where I want to lead this is that even if the 'multiuser' functionality is
there you still want a more or less consistent behavior if you leave it off
(unless you make multiuser mandatory). If I can choose I would rather see
the FC13 behavior in the future and it would make sense to back-port it to
RHEL6.

Anyway, I'll give pam_mount a try but if the problem is in the kernel
then it shouldn't succeed where mount.cifs fails.

(In reply to comment #15)

Keeping the manpage and kernel in sync is pretty much impossible. It would be nice if cifs code in the kernel would reject options that it doesn't recognize but the option parsing code is a real mess.

It's not specifically the multiuser code that changed that behavior. It was some earlier patches. RHEL6 has those patches, and F13 does not.

The problem is this: we want to avoid establishing more than one SMB session per user, as that's a waste of resources on the server and client and causes a number of other problems that can lead to the client holding fewer oplocks. So, we need some mechanism to say that we want to use *this* session to talk to the server (and hence *this* set of credentials) and not that session.

Matching by username is problematic since we can't reliably count on the username being set to anything meaningful. The username is mostly ignored in the case of a krb5 mount. The only exception is the case of keytabs, but we don't even really *have* to use it for that. In the case of a session established from kernel space (the multiuser mount case) it definitely won't be set to anything meaningful.

What we really want to do is match existing sessions based on credential cache, but we are not able to easily get at that info in kernel space (we'd have to upcall for it). The best we can reasonably do is match by the uid of the credential cache owner.

Later revisions will act more like RHEL6.0 as it stands now, but that's not to say that we can't accommodate this use-case. Matching krb5 sessions based on username is wrong, even if it happened to work for you in this situation. The simplest fix is to add a "creduid" mount option, but I'd like to consider the problem a little more before I try to push anything upstream.

Thanks for the detailed background information, I understand now how this came
to happen.

I am all in favour of the 'creduid' option. Basically its just what I was
asking for. The only difference is that you propose to put it in a separate
option while I was assuming that the 'user' option will change namespace and semantic with the authentication scheme. For krb5 that can only be the uname
(or uid) of the local user having the tickets to use. That only root can
choose the user and for all others the UID of the caller must match the
'creduid' was implicit to me.

My only other point is that for root and root only the choice to cache the credentials for the caller is a bad idea. Of course that works for all users
who can only mount shares using their own credentials anyway. But you would
expect root to be able to mount on behalf of other local users.

Since mounting shares as root with sec=krb5 does not really work now in RHEL6
as it stands, how about a quick fix where you disable the aggressive caching
and just open a new session every time when root is the caller? That would leave
an admin at least the choice to use the feature as long as his servers
support the load. Of course I do not know what would break if you add an
'if (uid != 0) { }' arond the cache lookup code but it's an idea. It will also
not be a solution for people toying with SELinux and roles but there are not
too many of those. :)

> But you would expect root to be able to mount on behalf of other local users.

No, I don't expect that at all. The bottom line is that allowing anyone (even root) to "borrow" another user's credcache is just wrong. It's true that the krb5 implementation in cifs used to do that, but that was a bad design (and my fault too).

The root of the problem here however is *really* that cifs has had a built in design limitation of a single set of credentials per mount. That limitation has caused the implementation of a number of hacks to deal with it. The multiuser mount code changes that, but we've had to "shoehorn" that in to some degree since the code was not originally designed for it.

That said, people will still want to use the old single-credential per mount model for some time, so we need to be able to accommodate that use-case.

> Since mounting shares as root with sec=krb5 does not really work now in RHEL6
> as it stands, how about a quick fix where you disable the aggressive caching
> and just open a new session every time when root is the caller?

I'm steadfastly against putting in hacks that are not upstream. Down that path lies madness...

We'll fix this in RHEL6 eventually, but that fix needs to pass the muster upstream first.

The creduid option is probably a reasonable fix, but as I said, I need to consider the problem a little more. There may be a better approach that we just haven't thought of yet.

We should probably leave the discussion here, this could go on for a while. 

My goal (for which I have a quite tight deadline) is to get multiple CIFS 
home directories mounted on RHEL 6.0 workstations. For the moment the
simplest seems to use pam_script, make mount.cifs SUID and use 'su' to mount
the directory as the user. I'll opt for that because it will definitely work. 

If 'multiuser' support will make it into 6.1 I will be happy and rather
leave this issue unresolved and leave you more time to port Minshall+French
symlink support back into RHEL6. :)

Understood. It's technically a regression (at least upstream) so I am on the hook to fix this there anyway, but that may not happen within your timeframe. Once support hooks up the a.r.c to this BZ, I'll ask them to pass along a test kernel to you with all of the patches that I have queued up for 6.1 so far.

Also, mfsymlink support is already being proposed for 6.1 (see bug 651878). The test kernel should have that as well.

Created attachment 472594 [details]
patch -- add creduid= mount option

Here's the patch I'm considering.

I'll add this to my RHEL6 patch stack for testing as well.

Created attachment 472835 [details]
patch -- add cruid= mount option

That patch won't work -- mount.cifs allows for "abbreviations" and "creduid=" collides with "credentials=". So, I'll plan to propose this patch instead which does work. The mount option will be called 'cruid='.

Stefan, do you have a support case open against this (specific) problem? I'd like to pass on a kernel for you to test if you do...

I have opened case 00402003 where I explicitly ask for the --cruid option.
I didn;t expect this to be implemented so fast. A test kernel is warmly welcome.

I've asked the support folks to pass the kernel on to you. For now, you'll need to use a numeric uid in the cruid= option. Once cifs-utils has been updated you'll be able to use usernames in that field too. mount.cifs will do the conversion before passing them off to the kernel.

The kernel will also have multiuser and mfsymlink support if you want to play with those too.

Thanks a lot, I got the kernel. Together with cifs-utils-4.7-4 the mfsymlink and cruid options work just fine. We're now deploying some test systems to a wider
audience to see if things are robust but so far nothing was reported.

Patch(es) available on kernel-2.6.32-112.el6

Either the -l option or the cruid option causes strange SELinux behavior on mounted /home directories.

GNOME session completely stucks and packagekit/dbus are acting silly. No denies are logged (with default dontaudit settings), but setting SELinux to permissive brings back the normal behaviour.

Tested on F14 with 2.6.35.6-48 with enabled mfsymlinks and cruid.

Greets
Marcus

Hmm, I don't know what that would be. These options are fairly straightforward and don't really have much to do with the actual permissions on the filesystem itself.

Are you sure that it's really a problem with '-l' or cruid=? If you mount up the filesystem without using those, does this problem go away?

Dear Jeff,

yes, mounting the share without the -l or cruid option does not cause the mentioned problems (using a kernel that sill works without -l, of course).

Disabling SELinux also helps, but is not the preferred solution in our scenario.

Greets
Marcus

So does the mount actually not happen in that case? Again, neither of those mechanisms should have any real affect on the client's operation once the mount has been performed.

Do you see any differences in the security context reported by the kernel on files in one case vs. the other?

With the latest Kernel, mounting is only possible with either -l or cruid set. Previous kernels allowed mounting without these options (which was obviuously a bug). With the latest kernels we have to disable SELinux in order to workaround these problems.

In general sebool samba_enable_home_dirs=1 has to be set.

In both cases context of the files is cifs_t.

Kind Regards
Marcus

(In reply to comment #33)

> Tested on F14 with 2.6.35.6-48 with enabled mfsymlinks and cruid.
> 

Given that this bug is against RHEL6 and the problem is evidently with f14, I think it would be best for you to open a new bug for this. Once you do that, I think we'll need to figure out what the problem actually *is* and why disabling SELinux helps. I'll plan to discuss this with some of our SELinux folks tomorrow..

Please cc me on the bug when you do this and I'll plan to grab it.

Dear Jeff,

as you mentioned this really does not seem to be related to SELinux (not sure why it behaved that way). The cause for the problem was the latest sssd update. I am going to talk to Stephen about that. Sorry for the noise.

Kind Regards
Marcus

Ok, that makes a bit more sense. Thanks for following up.

Hi Jeff,

We've been working with your test kernel for a while and identified an issue.
I choose to update case 00402003 because it seems tricky to me and I don't
know how support wants to deal with this. Here is what I wrote:

We are using the test kernel extensively in our test environment and so far
have identified one issue that does not relate to CIFS but to kerberos and
NFS4 and may be a side effect of the changes for CIFS.

When we boot a client with the latest release kernel (2.6.32-71.18.1.el6) we
can successfully mount NFS4 shares with sec=krb5 (server also runs
2.6.32-71.18.1.el6):

  mount -o sec=krb5 server:/export /mnt -t nfs4

When we boot the client unchanged into kernel-2.6.32-95.el6.jtltest.013 then
the mount fails. There is no info in the logs on the server or client and
running rpc.gssd and rpc.svcgssd with -f and -vvvv shows no warnings or errors.
The only hint we see so far is the error packet:

Remote Procedure Call, Type:Reply XID:0x2f9885a8
    Fragment header: Last fragment, 20 bytes
        1... .... .... .... .... .... .... .... = Last Fragment: Yes
        .000 0000 0000 0000 0000 0000 0001 0100 = Fragment Length: 20
    XID: 0x2f9885a8 (798524840)
    Message Type: Reply (1)
    [Program: NFS (100003)]
    [Program Version: 4]
    [Procedure: COMP (1)]
    Reply State: denied (1)
    [This is a reply to a request in frame 116]
    [Time from request: 0.000564000 seconds]
    Reject State: AUTH_ERROR (1)
    Auth State: GSS credential problem (13)

I assume something related to the kerberos handling changed and I am wondering
if the 6.1 release kernel will have the same problem.

Interesting. I doubt this is related to any of the CIFS changes. The RPC krb5 code is quite separate from the parts in CIFS. It's more likely that something in sunrpc has regressed. Could you open a new bug for this and cc me on it?

(As a side note, we're going to be discussing unifying some the RPC and CIFS GSSAPI code at the upcoming LSF summit, but for now they are not really related)

reproduced and verified by testing the related cifs-utils bug:
https://bugzilla.redhat.com/show_bug.cgi?id=669377#c7
https://bugzilla.redhat.com/show_bug.cgi?id=669377#c10

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0542.html