Started looking over the code that's actually in mainline at this point (I also
have sent a note to Steve French asking if he already has some out-of-tree code
written for this).

I've tinkered with the rawhide CIFS code to allow me to set SecurityFlags to
0x9009. That corresponds to:

CIFSSEC_MUST_SIGN | CIFSSEC_MUST_KRB5

With that, the Negotiate Protocol request has this:

.... 1... .... .... = Extended Security Negotiation: Extended security
negotiation is supported

...the server response looks correct. The mount attempt falls down after the
NegProt response is received and no session setup is sent, but that's pretty
much expected at this point.

There already seems to be an expectation that CIFS is going to use netlink
sockets for upcalls (there's some skeletal code already and it seems to point in
that direction). In looking at it, I'm thinking that's a reasonable thing here.
There's a bit more overhead than with a rpc_pipefs type interface, but it
doesn't require any special setup (like actually mounting rpc_pipefs).

The largest security blobs I've seen are ~1k, which is well within a page, but
I'm not clear on what the max payload size of a netlink packet. It would be much
preferable to make sure that the request can fit in a single packet since
netlink is connectionless.

Jeff,
in large domains the PAC can be *very* big, in the 60Kb order. A single ticket,
which contains the PAC and is passed to the server, may be that big. As you have
to pass the ticket down you need to ensure you can handle it.

Ahh that's a good point. Netlink isn't considered reliable when the machine is
under memory pressure, etc...but there are similar issues with using /proc-style
interfaces. Any time you're sending large quantities of data between kernel and
userspace, things get tricky (large being anything close to or more than a
page). It may be good to see if we can strip off the PAC before sending to
userspace as well (not sure how difficult that would be).

Given that upcalls aren't likely to be a frequent occurrence (only during
negprot/session_setup), I'm thinking that this is still probably the best
mechanism to start with.

CONFIG_CIFS_UPCALL is dependent upon CONFIG_CONNECTOR. Looking at the connector
API (see Documentation/connector in the kernel sources), it allows for a seq and
ack number. If we have to break up the security blob into multiple pieces to
send to userspace then we can take advantage of that to make sure that we get
all of the pieces. If we don't then we can send a NACK of some sort to the
kernel and have it resend the entire blob.

If you're looking at the userspace piece, then I'd suggest writing it in a way
that abstracts the GSSAPI stuff from the actual send/receive mechanisms. That
way if this proves not to be workable then changing that part out shouldn't be
too difficult.

(In reply to comment #4)
It may be good to see if we can strip off the PAC before sending to
> userspace as well (not sure how difficult that would be).

A) It would be impossible, it's part of the ticket, and you don't have the
encryption material to sign it :-)
B) It is the other way around, the PAC is IN the user ticket and *need* to be
passed to the server which will use it to set up your authorization credentials.
So it is userspace -> kernel, but if you keep processing in userspace, it is
actually simply unknown to the kernel, it is just that blobs that flow in and
out will be big.

> CONFIG_CIFS_UPCALL is dependent upon CONFIG_CONNECTOR. Looking at the connector
> API (see Documentation/connector in the kernel sources), it allows for a seq and
> ack number. If we have to break up the security blob into multiple pieces to
> send to userspace then we can take advantage of that to make sure that we get
> all of the pieces. If we don't then we can send a NACK of some sort to the
> kernel and have it resend the entire blob.

Yes the problem is that this interface, as you see, oblige you to implement a
lot of logic to handle just kernel-userspace communication. It is a bit like
implementing TCP again ... a pain :)

> If you're looking at the userspace piece, then I'd suggest writing it in a way
> that abstracts the GSSAPI stuff from the actual send/receive mechanisms. That
> way if this proves not to be workable then changing that part out shouldn't be
> too difficult.

GSSAPI libraries already do that as they have no clue what is the transport.

> It would be impossible, it's part of the ticket, and you don't have the
> encryption material to sign it :-)

Yep, I just realized that. So we have to ship the whole thing to the kernel. If
it's that large, doing that won't be trivial, but there's not much we can do
about that regardless of the mechanism we use...

Created attachment 215841 [details]
patch 1 -- rudimentary patch for doing upcalls using keyctl

This patch is a start at adding upcall capability using the keyctl facility.
It's not as "wrapperized" as we probably want, but that turns out to be harder
to do than expected since we have to do some setup work during module
initialization (setting up key types, etc).

I've experimented with it by adding a line like this to /etc/request-key.conf:

#OP	TYPE	DESCRIPTION	CALLOUT INFO	PROGRAM ARG1 ARG2 ARG3 ...
#====== ======= =============== =============== ===============================

create	cifs.spnego	*		*		|/bin/cat

With this, the security blob just gets piped into stdin of /bin/cat and the
stdout ends up being the key payload. With this, I see the same payload go in
and out:

 fs/cifs/cifsfs.c: Devname: //dhcp231-40.rdu.redhat.com/testuser flags: 64 
 fs/cifs/connect.c: CIFS VFS: in cifs_mount as Xid: 0 with uid: 0
 fs/cifs/connect.c: Username: testuser
 fs/cifs/connect.c: UNC: \\dhcp231-40.rdu.redhat.com\testuser ip: 10.11.231.40
 fs/cifs/connect.c: Socket created
 fs/cifs/connect.c: sndbuf 16384 rcvbuf 87380 rcvtimeo 0x7fffffffffffffff
 fs/cifs/connect.c: Demultiplex PID: 7122
 fs/cifs/connect.c: Existing smb sess not found
 fs/cifs/cifssmb.c: secFlags 0x8009
 fs/cifs/transport.c: For smb_command 114
 fs/cifs/transport.c: Sending smb of length 78
 fs/cifs/connect.c: rfc1002 length 0xb7
 fs/cifs/cifssmb.c: Dialect: 2
SPNEGO blob:: dump of 94 bytes of data at 0xffff81000771c150
 06065c60 0501062b 52a00205 30a05030 ` \ . . + . . . . . � R 0 P � 0
 09062e30 8248862a 020112f7 2a090602 0 . . . * . H . � . . . . . . *
 f7864886 02020112 862a0a06 12f78648 . H . � . . . . . . * . H . 
 03020201 062b0a06 82010401 0a020237 . . . . . . + . . . . . 7 . . .
 1a301ca3 161b18a0 326e6977 746a336b � . 0 . � . . . w i n 2 k 3 j t
 4540246c 504d4158 432e454c a56b4d4f l $ @ E X A M P L E . C O M

This matches the SPNEGO blob on the wire. keyctl show also shows:

760709127 --alswrv	0     0   \_ cifs.spnego: 00000000:10.11.231.40

This currently hangs the key off of cifsSesInfo. Presumably once we do the
session setup, we can do a key_put on this and it should get freed.

This makes some assumptions so far:

1) no blobs bigger than 64k. If we need bigger than that, we might should
consider vmallocs for them (might be reasonable anyway)

2) the SPNEGO blobs don't have embedded NULLs. So far, none I've seen do, but
that might not be a safe assumption. If it's not safe, then we'll probably need
to do some sort of binary->string conversion for the blob before handing off to
userspace (base64 or something).

I'm thinking we might also want to add a second keytype for session keys so
they can be generated in an upcall as well. We could hang the mac_signing_key
off of a struct key. When upcalls aren't needed they can just generate the key
as they do now, and do an add_key with all of the data.

Note that I've also seen some lockdep warnings with keyctl. dhowells has
responded that it's a false alarm -- looking at that now...

To be clear, the data that's sent from kernel to userspace can't have embedded
NULLs. That data gets passed to the request-key program as a string, and is null
terminated. The output from the userspace program *can* have embedded NULLs so
we just need a program that takes the description and the SPNEGO blob and
printf's a new SPNEGO blob suitable for the session setup to stdout. Raw data is
fine there (and would in fact be preferred).

Can't we use the connector api to pass the blob up - using the keyctl to tell
user space the read from connector the blob from kernel etc. so that there is no
need to do conversion of the embedded nuls

We could, but connector has some things that make it sub-optimal. It's based on
netlink which is connectionless. If queues are full, etc, then packets can get
dropped there, so we need to implement sequencing and acks...

At the end of the day, the data gets moved via copy_to_user/copy_from_user.
Having to deal with sockets for that seems like a pain. I'd prefer to keep this
as simple as possible and have to deal with only one conduit from kernel to
userspace. keyctl is pretty darned simple...

Also, the conversion here could be something as simple as walking through each
byte of the blob and doing:

     sprintf(deststring, "%2.2x", blob[i])

...the userspace program would just have to do the reverse. That does basically
double the size of the blob though, so it would be nice if we had a more
space-efficient conversion scheme. I think though, that vmalloc is probably
suitable for this. It's not as big a deal to allocate a huge temp buffer if we
use that...

Created attachment 217871 [details]
patch 2 -- convert blob to hex string before sending

I'm going to take from Steve's comment that the assumption about embedded NULLs
isn't reliable. This patch converts the blob into a hex string before sending
it off to userspace.

Tested with a simple userspace perl program that just does:

while (read(STDIN, $hexbyte, 2)) {
	printf("%c", hex($hexbyte));
}

The blob I get back looks correct:

SPNEGO blob:: dump of 94 bytes of data at 0xffff8100128640a8
 06065c60 0501062b 52ff0205 30ff5030 ` \ . . + . . . . . � R 0 P � 0
 09062e30 ff48ff2a 020112ff 2a090602 0 . . . * � H � � . . . . . . *
 ffff48ff 02020112 ff2a0a06 12ffff48 � H � � . . . . . . * � H � �
.
 03020201 062b0a06 ff010401 0a020237 . . . . . . + . . . . � 7 . . .
 1a301cff 161b18ff 326e6977 746a336b � . 0 . � . . . w i n 2 k 3 j t
 4540246c 504d4158 432e454c a56b4d4f l $ @ E X A M P L E . C O M

I guess I also need to fix up the caller of cifs_spnego_convert to set secType.
For now, I suppose we can assume that it's kerberos in this situation, but we
probably need to handle it more intelligently.

Looking at CIFS_SessSetup now to see what changes need to be made there...

Created attachment 218261 [details]
patch 3 -- fix session setup request packet for SPNEGO auth

This patch supercedes the last one. With this, I get an actual sane looking
Session Setup packet getting sent over the wire (according to wireshark). Since
I don't have a userspace program that does a correct SPNEGO conversion I've
just tested it by sending back the same blob that came in the negProt reply.
The server replies to the session setup request with
NT_STATUS_MORE_PROCESSING_REQUIRED.

I think this is all I can really do at this point until the callout program is
available. Some questions I have though:

1) I'm not sure what we want to do about the signing key. Are there any docs on
how that's calculated with Kerberos? Any thoughts on whether to calculate it in
kernel space, or whether we should do another upcall for it?

2) I'm not sure how best to handle NTLMSSP vs. Kerberos in CIFSSMBNegotiate.
How do we detect it? Or do we even care since it's just a SecurityBlob?

3) Is it possible to have a server that does kerberos but not unicode? Do I
need to break up ascii_ssetup_strings in a similar fashion how the unicode one
is done?

The patch needs to be broken up into smaller more easily digestable chunks too.
I plan to tackle that next week.

There are also some cleanup patches embedded in this one that should probably
go in before we start adding in the actual kerberos stuff.

Created attachment 218271 [details]
patch 4 -- add upcalls using keyctl for CIFS auth

Last patch had a missing delta, the number of iovecs passed to SendReceive2 was
off...

Looking at the cifsauth patch, I am wondering if we really need to save the
spnego key in the session struct - if we reconnect don't we need to regenerate it?

I'd imagine we would. I just added it to that for now to give me a way to easily
pass it to the session setup. I pretty much envisioned freeing the key once the
session setup packet went out.

I'm still working on a respin based on some changes to keyctl that I discussed
with David H. and I'll have a look at that too...

I would like to check in a few of the changes to fs/cifs/sess.c within this two
week merge window (not necessarily the completed upcall - if the keyctl or other
changes are still being debated).  I may take a look at this if Shirish makes
enough progress on the cifsacl->unix_mode code patches this week.

Sounds good. I have some preliminary patches that need to go in before the main
kerberos patches go in anyway. I'll try to have a set together early this week
for review.

Created attachment 228611 [details]
patch 5 -- add upcalls using keyctl for CIFS auth

Revised patch that contains a new method for encoding the upcall data. This
should be much more space efficient than hex encoding. With Dave Howells' patch
to allow for larger keysizes from userspace, this scheme is only limited by
ARG_MAX. That's currently set to 128k (at least in Fedora). I think we can
reasonably deal with SPNEGO blobs of ~100k with this scheme (which should be
plenty -- Simo seemed to think that there was a 64k limit to keys anyway).

I've also sent a patchset upstream of small cleanup patches, plus a few of the
sess.c changes that Steve wanted. I'm not comfortable merging too much there
until we have something that actually work.

I've opened bug 334931 to track development of the userspace piece of this.
Development is focused there at the moment...

Created attachment 232741 [details]
patch -- update to use new keyctl binary blob patches

This is the latest iteration of the cifs spnego patch. Major changes are:

- takes advantage of David H's patches to allow passing of a binary blob to
request_key_with_auxdata(). This means we don't have to encode the blob when we
send it off to userspace.

- makes it so that we don't hang the key off of the cifsSesInfo struct. Instead
cifs_session_setup declares a pointer to a key and passes that around.

- some functions have changed to hopefully more descriptive names

Currently this relies on a patchset from David H that's not yet upstream. It's
probably too late to get those into 2.6.24 (some are very new), but hopefully
it'll get into -mm soon and can make 2.6.25.

Created attachment 236521 [details]
patch -- add upcalls using keyctl for CIFS auth

I think this patch is now pretty close to being in a working state. With this +
a hacked up version of Igor's userspace program, I was able to get a SUCCESS
response from the server on a session setup request. The tree connect then
proceeded to fail with an NT_STATUS_ACCESS_DENIED, but I'm not certain that I
have everything set up correctly permissions-wise on the server.

I need to do some code cleanup and break up the patchset into more easily
digestible chunks. I also want to see if I can get a working mount before we
pull any of this into upstream.

I do, however, have at least one preliminary patch that should probably go in
beforehand. I'll post upstream soon too...

Above patch is against Steve F's cifs-2.6 git tree. It also incorporates some of
David Howells' keyctl patches that are not yet upstream.

Success! With a hacked up version of Igor Mammedov's userspace request-key
callout, I was able to sucessfully mount a win2k3 server using kerberos auth.
There's still plenty of work to be done, but the kernel piece seems to be working.

Created attachment 237721 [details]
patch -- implement SPNEGO by offloading to userspace via keyctl

This patch is a cleaned up and revised version of the one posted yesterday. It
also works with a modified version of Igor's program. This also more carefully
handles the case when CONFIG_CIFS_UPCALL isn't set.

The patch also incorporates 3 patches from David Howells that aren't yet
upstream.

It should apply cleanly on top of Steve F's cifs-2.6 tree with
44093ca2fef3c52dc7d186116862d74f9a676e0f as the head.

I've posted a broken-out version of this patchset upstream to the
linux-cifs-client mailing list as an RFC. Awaiting comments.

Created attachment 248401 [details]
patch -- get SPNEGO blob for session setup via downcall

This patchset has been partially merged into the upstream cifs-2.6 tree. As of
now, it's considered experimental and depends on CONFIG_CIFS_UPCALL which
depends on CONFIG_CIFS_EXPERIMENTAL. The userspace request-key callout is still
being worked on, but the patch has been tested with a test callout program and
works correctly.

There's still work to be done, here's the list of to-do's that I posted with
the patchset:

---------------[snip]----------------

1) The userspace request-key program is not yet complete. Igor and Simo
Sorce have volunteered to write it. I've used Igor's userspace program
to test it, but it needs some work (primarily work on having it deal
with different UID's). I can envision this being part of the samba tree.
smbclient needs to roll up SPNEGO blobs too, so it seems like that code
could be shared. I'll defer that decision to those who are writing
the program, though...

2) Multi-stage SPNEGO conversations are not yet implemented. It
shouldn't be too tough to add them, but as of today, I'm not clear on
what they actually look like on the wire. Particularly, I don't know
what the SMB status field will look like. So for now, if multi-stage
SPNEGO is needed, the mount will probably fail. If someone is able
to trigger this and can provide captures or info about it, then we
should be able to get that added.

3) multi-user mounts are not currently implemented. I've been chatting
with Steve about the best scheme for handling them, so for now, things
are pretty much still single-user.

---------------[snip]----------------

The main thing for our customers I think is #3. Kerberos auth is much more
useful if it allows multiple user credentials to be used. Once this part gets
settled in upstream I'll start working on that piece. The design for how to
handle multisession mounts is still being debated.

One other note...

With the current upstream code, the keyctl API limits the size of key payload to
32k. David Howells has a patch to increase that to 1M (using a vmalloc if
kmalloc fails), but it probably won't make it upstream until 2.6.25.

That means that for now, this patch will only work when the SPENGO blob from
userspace is < 32k. Once that patch is merged we should be able to deal with any
reasonable size blob.

Changing dependencies since we're more or less waiting on the userspace piece at
this point...

Steve has committed the last of the current patchset upstream. They should make
the 1.52 release. With that (and once the cifs.spnego.c samba patch is
committed), we should be able to support the trivial case of Kerberos mounts
where all linux users share the same session to the server (and hence, the same
creds).

To make krb5 mounts really useful though, I think we need to allow for cifs to
spawn new sessions as needed. There are a couple of different approaches (pardon
the hand-waviness, since I don't have this worked out well):

1) allow for multiple sessions per mount and have the code select the
appropriate session when a user goes to access it. This would probably mean
keeping a reference count for the session. On unmount, we'd decrement the
refcount on any session attached to the mount and drop any that hit 0. It'll
probably be a bit more complex than this, but can be done within the context of
cifs code.

2) spawn a new per-user mount when a new user accesses the share. This looks a
bit more complex:

For instance, suppose a new user with proper krb5 creds touches a mountpoint. We
spawn a new per-process namespace mount with the MNT_SHRINKABLE flag set. Where
do we attach this mount? It's possible for processes to get their own namespaces
on a clone() call, but it's pretty rare and I don't think we want CIFS to try to
do that after the fact.

If we just try to use MNT_SHRINKABLE like it's being used for existing
filesystems (AFS and NFS), then we have to decide where to put the new mount. I
suppose we could overlay the existing mount, but that seems sort of messy and
it's not clear to me what we gain from that model.

After looking at it, I think just handling a list of sessions sounds like the
easiest approach, but maybe I'm misunderstanding what Steve intends here...

Comments? Thoughts?

I've started looking over the multiuser piece today. When there is more than one
krb5 mount to the same server using different creds, there are currently
problems. The first krb5 mount works fine. For instance, consider these 2 mounts
in /etc/fstab:

//win2k3.example.com/export /mnt/cifs       cifs     user=root,sec=krb5i,noauto 0 0
//win2k3.example.com/testuser /mnt/testuser       cifs    
user,user=testuser,uid=50000,gid=50000,sec=krb5i,noauto 0 0

If both root and testuser have a valid TGT, both mounts should work. In practice
though, the second one always fails. I think it's because it tries to use info
from the first session incorrectly.

Investigating that now. Once we have some of these kinks worked out with the
existing code, we can start working on real multiuser mounts with kerberos.

Testing this at home. First mount succeeded. Second one fails with this cifsFYI
info:

 fs/cifs/cifsfs.c: Devname: //win2k3.poochiereds.net/testuser flags: 64 
 fs/cifs/connect.c: CIFS VFS: in cifs_mount as Xid: 222 with uid: 0
 fs/cifs/connect.c: Username: testuser
 fs/cifs/connect.c: UNC: \\win2k3.poochiereds.net\testuser ip: 192.168.1.8
 fs/cifs/connect.c: Existing tcp session with server found
 fs/cifs/connect.c: Existing smb sess not found
 fs/cifs/connect.c: Security Mode: 0xf Capabilities: 0x8001f3fd TimeAdjust: 0
 fs/cifs/sess.c: sess setup type 6
 fs/cifs/cifs_spnego.c: key description =
ver=0x1;host=win2k3.poochiereds.net;ip4=192.168.1.8;sec=krb5;uid=0xc350
 fs/cifs/transport.c: For smb_command 115
 fs/cifs/transport.c: Sending smb:  total_len 1396
 fs/cifs/connect.c: rfc1002 length 0x27
 CIFS VFS: Unexpected SMB signature
Status code returned 0xc0000022 NT_STATUS_ACCESS_DENIED
 fs/cifs/netmisc.c: Mapping smb error code 5 to POSIX err -13
 fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
 fs/cifs/sess.c: ssetup rc from sendrecv2 is -13
 fs/cifs/sess.c: ssetup freeing small buf d5e47e00
 CIFS VFS: Send error in SessSetup = -13
 fs/cifs/connect.c: No session or bad tcon
 fs/cifs/connect.c: CIFS VFS: leaving cifs_mount (xid = 222) rc = -13
 CIFS VFS: cifs_mount failed w/return code = -13

...I'm thinking that sharing the TCP session is the issue, but I'm still
investigating why.

As Simo pointed out, the problem seems to be with signatures. When I disabled
mandatory signing on the server and mounted with sec=krb5 both mounts worked
correctly. Still looking at possible resolutions...

When I mount the same share twice (once as root and once as testuser) and turn
on /proc/fs/cifs/MultiuserMount, things seem to work correctly. When I log in as
"testuser" and touch a file on root's mountpoint, then the testuser session is
actually used.

I did a temporary patch to disable sharing of tcp session if the username
doesn't match and signing was enabled on the tcp session. That worked, so I
think we'll probably need something like that, though we need to handle all
possible cases and that wouldn't.

The way signing flags are handled is a bit of a mess at the moment, so some
cleanup there first might be in order. The rule should be that we only share TCP
sessions with another mount if the server is the same and:

1) signing is disabled on both mounts

...or...

2) the user is the same on both mounts

...any other situation should cause a new TCP session to be spawned.

The last issue turns out to have been a bug in the SPNEGO patch. New SPNEGO SMB
sessions were overwriting the key, when they should have allowed the first
session after a Negotiate Protocol exchange to have precedence. I've pushed a
patch upstream to fix that and a minor alignment issue with the string section
of the session setup.

At this point, I'm going to go ahead and close this BZ as complete. I may open
others as a point of discussion as to how we want to handle multiuser mounts.

Since upcalls have now been enabled in rawhide kernels, closing this with a
resolution of RAWHIDE.