Debug log from server would be helpful. Also specifying what exact configuration you are using. Does it work with different ccache types?

Jakub,

Thanks for the debug tip.  It does not appear to be limited to KCM but also KEYRING is affected.

KCM does not setup the ticket cache (or set krb5ccname) prior to the pam session stack execution
KEYRING (default_ccache_name = KEYRING:persistent:%{uid}) does not setup the ticket cache (or set krb5ccname) prior to the pam session stack execution
FILE does setup the ticket cache, define krb5ccache, and has the valid forwarded tickets populated prior to the pam session stack execution

as a comparison to RHEL7, both KEYRING and FILE have valid caches and krb5ccache var defined for the pam session stack execution.

Also worth noting, seems the pam stack is executed in this order.
-pam.d/sshd:account
-pam.d/systemd-user:account
-pam.d/systemd-user:session
-pam.d/sshd:session

With systemd-user changes in RHEL8, I’m uncertain yet as to if systemd-user should also need a valid ccache.  More testing will be needed to see if this is an issue as well.

Note we're running the default sshd config with GSSAPIAuthentication set to yes.

I will send you debug logs privately.

Hi Jakub,

pam_afs_session looks for the existence of KRB5CCNAME as a signal that the sssd/pam stack successfully obtained a TGT. If authentication was performed using something other than krb5 and no ticket forwarding occurred, then pam_afs_session does not attempt to use the krb5 ccache because the contents might be stale or from an attacker.  When forwarding tickets, the tickets must be put into the ccache and KRB5CCNAME set before pam_afs_session is called for "account" and for "session".

Jeffrey Altman
AuriStor, Inc.

(In reply to Jeffrey Altman from comment #3)
> Hi Jakub,
> 
> pam_afs_session looks for the existence of KRB5CCNAME as a signal that the
> sssd/pam stack successfully obtained a TGT. If authentication was performed
> using something other than krb5 and no ticket forwarding occurred, then
> pam_afs_session does not attempt to use the krb5 ccache because the contents
> might be stale or from an attacker.  When forwarding tickets, the tickets
> must be put into the ccache and KRB5CCNAME set before pam_afs_session is
> called for "account" and for "session".

Thank you for clarification of your use case. 

In this case, authentication is not done using sssd/pam stack, but using gssapi-with-mic  mechanism inside openssh as visible from the debug logs:

> Accepted gssapi-with-mic for <NAME> from <IP> port 55006 ssh2: <NAME>@MITLL.AD.LOCAL

This is done completely outside of the pam:auth step and as you can see in the debug log, pam is invoked only for account and session steps. This authentication is done inside of sshd.

From your initial testing:

> auth   optional     pam_exec.so debug /tmp/debug.sh

This will never be invoked by openssh. Please, try the same with session or account step.

Looking into the traces, the FILE ccache goes through different code path as it is a "legacy" behavior of not setting anything in krb5.conf (or using `KerberosUniqueCCache`).

The KRB5CCNAME environment variables are no longer set by default, because it was troublesome in some use cases (#1199363) and kerberos applications should be able to find ticket themselves even without this environment variables. If that is not the case for pam (probably because of running as different than target user?), it is probably a good idea to keep using FILE cache.

If we will agree that setting KRB5CCANME environment variable is something that is useful in combination with new ccache types, we can see what can be done to support `KerberosUniqueCCache` with other ccache types, but last time I was touching this, kerberos developers were discouraging me from going in this direction (can not find the reference now).

Additionally, I do not think pam_afs_session is something that is part of RHEL (I do not see that even in EPEL8), so I am really not sure what is the level of support we should give to this setup (even though I agree that if it is something that worked before, we should not cut that).

Let me know if you will have some update. I am adding kerberos developers if they will have something to say about this.

> The KRB5CCNAME environment variables are no longer set by default, because it was troublesome in some use cases (#1199363) and kerberos applications should be able to find ticket themselves even without this environment variables. If that is not the case for pam (probably because of running as different than target user?), it is probably a good idea to keep using FILE cache.

Although the KRB5CCNAME environment variables might not be required to be set in the process environment, it is required to be set in the pam environment (struct pam_args.pamh) so that pam session modules know that the Kerberos credentials are valid.  This is true even if the value that is set in the pam environment is obtained from the krb5 library default.

pam_afs_session is a module distributed by AuriStor in conjunction with the AuriStor File System.   AuriStor is a Red Hat Enterprise Linux OEM partner.

> The KRB5CCNAME environment variables are no longer set by default, because it was troublesome in some use cases (#1199363) and kerberos applications should be able to find ticket themselves even without this environment variables. If that is not the case for pam (probably because of running as different than target user?), it is probably a good idea to keep using FILE cache.

There might be some confusion regarding the advice that was provided.   The KRB5CCNAME setting that is referred to here is the value passed to pam session modules via the "struct pam_args.pamh" environment block; not the KRB5CCNAME environment value set in the result process environment block.  The pam KRB5CCNAME envvar is required to be set so that pam session modules know that the Kerberos credentials are valid.  This is true even if the value that is set in the pam environment is obtained from the krb5 library default.  Without this signal, the pam session modules might use a result from krb5_cc_default() that refers to ccache containing some other user's tickets depending on the local krb5.conf settings.

>Additionally, I do not think pam_afs_session is something that is part of RHEL (I do not see that even in EPEL8), so I am really not sure what is the level of support we should give to this setup (even though I agree that if it is something that worked before, we should not cut that).

pam_afs_session is a module distributed by AuriStor in conjunction with the AuriStor File System.   An alternative version of pam_afs_session is shipped by third parties for use with OpenAFS.   AuriStor is a Red Hat Enterprise Linux OEM partner.

The setting of KRB5CCNAME as a signal to pam session modules that the krb5 ccache contents are valid is independent of any specific pam session module that might obtain service tickets for any purpose.

pam_afs_session isn't something Red Hat maintains, but like all software our customers use, we should avoid breaking it unduly :)  (The SSSD project recommends its use for those wanting to add AFS capabilities.)

I don't recall suggesting that KRB5CCNAME be unset in all cases; perhaps that's due to the conversation around https://bugzilla.redhat.com/show_bug.cgi?id=1199363#c20 ?

Since I'm unfortunately not well-versed with pam_afs_session, do I understand correctly from #c6 that pam_afs_session won't use Kerberos credentials unless KRB5CCNAME is set in the pam environment?  The "implementation notes" section of https://github.com/auristor/pam-afs-session specifies what happens when it is set, but I'm not sure what happens if it isn't - does it just fail?  (Is that what we're seeing here?)

It makes sense to me that we should set KRB5CCNAME in the PAM env if we're going to otherwise.  The case I'm unclear on is when we're using the default ccache from ssh.

Hi Robbie,

If KRB5CCNAME is unset, then pam_afs_session does not know that it can trust the contents of the ccache and it refuses to execute the configured "aklog" process.

There is an "always_aklog" option that I believe Karl has tried with mixed results.

Hi Robbie and Jeff,

The pam_afs_session will get a token during the session stack if always_aklog is enabled, so there are creds in the cache.  Uust no KRB5CCNAME set in time.  Again, this only happens with GSSAPI and ssh, where the auth stack is never executed.  Password auth with sssd in the pam stack does set a KRB5CCNAME which then allows pam_afs_session to acquire tokens via aklog without issue.

Given there's obviously a ticket cache in existence in both cases, would be good to make the behavior for all pam modules the same.

Does setting `KerberosUniqueCCache yes` in sshd_config works for you?

It does help, but still not quite right.

with KerberosUniqueCCache yes:

$ klist
Ticket cache: FILE:/tmp/krb5cc_966406121_ZjhOQwHmi6
Default principal: user@REALM

should be using the KCM cache.  with KerberosUniqueCCache no:

$ klist
Ticket cache: KCM:966406121:67253
Default principal: user@REALM

Sorry, I somehow missed your comment.

This is expected as documented, the KerberosUniqueCCache always uses file cache at this moment.

If we would like to have this supporting also different credential cache types, we would need to change this to feature request with proper justifications.

Hi Jakub,

Thanks for the update and guidance.  I suppose now looking at the documentation, KerberosUniqueCCache does not solve our problem.

Per the man-page:
     KerberosUniqueCCache
             Specifies whether to store the acquired tickets in the per-session credential cache under /tmp/ or whether to use per-user credential cache as configured in /etc/krb5.conf.  The default value no can lead to over‐writing previous tickets by subseqent connections to the same user account.

(btw, "subseqent" is misspelled)

We are looking for ssh to populate the ticket cache as defined in /etc/krb5.conf, in modern RHEL8 terms, that's the KCM ticket cache.

In my view, I still classify this as a bug, because the cache is populated for all other types of logins except for ticket forwarding.  Additionally, this works in RHEL 7 (with keyring and file where the KRB5CCNAME is defined with a valid cache with forwarded tickets) and seems to have been broken along the way somewhere.

The short history of recent changes to credential caches support can be summarized like this:

The RHEL7 was blindly setting KRB5CCNAME all the time, which was breaking applications running in the ssh session (bug #1199363). As the KRB5CCNAME was just pointing to the location equal to what was set in krb5.conf, it was suggested to me by kerberos developers [1] that we should not set the variable if it is pointing to the default location as applications should be able to handle that. As a fallback we kept the old behavior with the compatibility option `KerberosUniqueCCache`. Further discussion was in bug #1566494, where we introduced usage of krb5_cc_new_unique() and fixed many bugs how the ccache was created and cleaned up. I think I was suggesting whether creating per-session ccaches in different place than in FILE (as was supported originally) is something that would make sense [2], but if I read the discussion right again, there was no definite answer so we did not continue this path.

I think we agree that it would be a use case we should support so I will have a look what we can do. 


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1199363#c20
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1566494#c16

I spent today some time talking with Robbie over IRC throwing some ideas how to approach this issue. The default_ccache_name configuration option in krb5.conf can not be reasonably used for creating per-session caches as it is identifying default location, which is predictable per user. The only possible option to reuse this would be to throw a ticket into this cache and store its location into KRB5CCNAME environment variable (and when configured, to remove it on logout). This will have advantage that it will just work, but might have disadvantage that it does not guarantee any session separation -- all of the tickets will still be in the one credential cache and any session can just unset the KRB5CCNAME and "klist -A" tickets of other sessions. Not sure if this is a problem or not.

One of the attempts to create per-session credential cache was using default_ccache_name = KEYRING:session:ssh, but it did looks like it does not work, probably because of how ssh handles kerberos in ssh (I assume because of UID juggling). This might require some more debugging.

This means, that the way that makes most sense for me is to either use the credential cache type from default_ccache_name (or have some different configuration option for selecting ccache type inside of sshd) and use some sane default locations for per-session ccaches. we already have FILE:/tmp/krb5cc_%d_XXXXXXXXXX for FILE:, we can create very similar path for DIR:, for keyring, the KEYRING:session would make the most sense if it would work, but I do not think we have any sensible for KCM as it is also instantiated per-user.

Karl, Simo, Robbie, please, correct me if I miss something, if something need some clarification or whether you have some strong opinion which direction we should take with this bug.

From my perspective, one important-but-unanswered question here is whether we want this to work for all ccache types, or whether just the one (non-FILE) is sufficient.  Unfortunately the collection types all behave differently in ways that we haven't really abstracted over.  Simo and I were discussing this a bit with upstream, and while we could probably provide a nice interface, we still need to decide what the semantics should be - that is, what "correct" means - first.

From what Simo wrote me on IRC last week, it looks like the KEYRING:session probably does not work because it is created before the new (for user) keyring session is created through PAM (during session stage). Checking the code, the store_credentials is called before the pam_session() is called. But the the pam_session is the place where we already need the environment variable for pam_afs_session so it looks like a chicken-egg problem.

KCM session credential caches are not implemented at all so I will get in touch with the SSSD developers whether this is something on the roadmap.

In any case, it would be nice to see the whole pam stack. We assume that pam_afs_session comes after pam_session setting up keying. If not, the we can not use session keyring at all.

One possible solution might be creation of MEMORY: ccache inside of sshd and creation of a new pam module that would be inserted after pam_session, which would copy the MEMORY ccache to the new KEYRING ccache of the newly created session. It would still be quite fragile, but sounds like the only option for keyring session caches.

The last possibility might be some integration with pam_sss, which is already part of all pam stacks, but again, it might be problem that it does not come early enough for pam_afs_session.

>it looks like the KEYRING:session probably does not work because it is created before the new (for user) keyring session is created through PAM (during session stage)

The Kerberos credential cache must be created during PAM SETCRED not SESSION.  From pam_setcred(3)

> The pam_setcred function is used to establish, maintain and delete the credentials of a user. It should be called to set the credentials after a user has been authenticated and before a session is opened for the user (with pam_open_session(3)). 

Creating a credential cache in SESSION in problematic because there is no guarantee of the ordering of session modules relative to each other.

Hi Jeff,
if that is a requirement then KEYRING:session will simply not be an option, as the session KEYRING for a session does not exist until pam_session creates it.

> the session KEYRING for a session does not exist until pam_session creates it.

That sounds like a problem with how pam_setcred has been implemented, not pam_session. 

Note that a setcred module is permitted to be called from pam_sm_open_session as well as pam_sm_setcred.

A setcred module is permitted to create a session context if one doesn't already exist.

Just for the record, this is the code in openssh, which we talk about, first, the credentials are created (L2278), then pam_setcred (L2284) and then pam_session (L2284) are called:

https://github.com/openssh/openssh-portable/blob/acfe2ac5fe033e227ad3a56624fbbe4af8b5da04/sshd.c#L2278

So this would suggest that one possible option might be to try to create session earlier from ssh if it would be at least a bit possible.

Just to weigh in on the unique credential caches / session - a separate credential cache for each session with the same uid is best for us.

For which cache type to use, my vote is of course KCM because of the KCM support for credential management that doesn't need to happen outside of the cache manager.  Although I do agree that all should be considered for compatibility.

(In reply to Jakub Jelen from comment #21)
> Just for the record, this is the code in openssh, which we talk about,
> first, the credentials are created (L2278), then pam_setcred (L2284) and
> then pam_session (L2284) are called:
> 
> https://github.com/openssh/openssh-portable/blob/
> acfe2ac5fe033e227ad3a56624fbbe4af8b5da04/sshd.c#L2278
> 
> So this would suggest that one possible option might be to try to create
> session earlier from ssh if it would be at least a bit possible.

Could you provide me a pointer to the repository containing the implementation of pam_sm_open_session and pam_sm_setcred?

Thanks.

(In reply to Karl Grindley from comment #22)
> Just to weigh in on the unique credential caches / session - a separate
> credential cache for each session with the same uid is best for us.
> 
> For which cache type to use, my vote is of course KCM because of the KCM
> support for credential management that doesn't need to happen outside of the
> cache manager.  Although I do agree that all should be considered for
> compatibility.

Karl,

Could you please answer the following questions about your use case?

 1. Do you ever anticipate the Kerberos client principal to be different across the multiple sessions for a specific uid?

 2. Is the reason that separate credential caches are desired because 
  
   a. after the last session terminates you wish the Kerberos tickets to be destroyed, and
 
   b. you do not want a session termination prior to the final session termination to destroy the cache contents of any other session?

 3. Is the desire for separate credential caches per session because you want each session to be forced to authenticate every "TGT lifetime"?

 4. Are there any other possible reasons for desiring independent credential caches per uid session?

Thanks.

(In reply to Jeffrey Altman from comment #23)
> (In reply to Jakub Jelen from comment #21)
> > Just for the record, this is the code in openssh, which we talk about,
> > first, the credentials are created (L2278), then pam_setcred (L2284) and
> > then pam_session (L2284) are called:
> > 
> > https://github.com/openssh/openssh-portable/blob/
> > acfe2ac5fe033e227ad3a56624fbbe4af8b5da04/sshd.c#L2278
> > 
> > So this would suggest that one possible option might be to try to create
> > session earlier from ssh if it would be at least a bit possible.
> 
> Could you provide me a pointer to the repository containing the
> implementation of pam_sm_open_session and pam_sm_setcred?

Do you mean the implementation of these functions in pam_afs_session module? It lives here:

https://git.eyrie.org/?p=afs/pam-afs-session.git;a=blob;f=public.c;h=35f64065e33eb9c8974dae45352623772d6b376c;hb=HEAD

(In reply to Jakub Jelen from comment #25)
> (In reply to Jeffrey Altman from comment #23)
> > Could you provide me a pointer to the repository containing the
> > implementation of pam_sm_open_session and pam_sm_setcred?
> 
> Do you mean the implementation of these functions in pam_afs_session module?

No.  I'm asking about the functions I named.  OpenSSH calls pam_open_session which calls pam_sm_open_session which in turn calls each of the pam session modules and should be calling the pam setcred modules.

Likewise OpenSSH calls pam_setcred which calls pam_sm_setcred which in turn calls each of the pam setcred modules.

I want to see the implementation of the pam_sm_xxxx functions.

> It lives here:
> 
> https://git.eyrie.org/?p=afs/pam-afs-session.git;a=blob;f=public.c;
> h=35f64065e33eb9c8974dae45352623772d6b376c;hb=HEAD

That repository was abandoned five years ago.  See the list of orphaned software:

  https://www.eyrie.org/~eagle/software/pam-afs-session/
  
AuriStor ships a pam_afs_session as part of the AuriStorFS distribution which is the one that Karl is using.

Since you are referencing Russ Allbery's work.   His pam_krb5 is actively maintained and is the one that ships in Debian/Ubuntu.  It is was also at one point actively supported cross-platform on Solaris, Apple, AIX, etc.

  https://www.eyrie.org/~eagle/software/pam-krb5/

Its history is independent of the sourceforge based pam_krb5 that Red Hat ships.   In my opinion, Russ' pam_krb5 is more complete.  Note that Russ' pam_krb5 has quite a bit of logic specific to addressing the problems caused by OpenSSH performing the "auth" stage in a separate process from "setcred" and "session".     Russ's pam_krb5_setcred is designed to be called from both pam_sm_setcred and pam_sm_open_session. 

I would like to see the sources to RHEL's pam_sm_setcred and pam_sm_open_session.   I want to confirm whether or not pam_sm_open_session is calling the "setcred" modules.

> I would like to see the sources to RHEL's pam_sm_setcred and pam_sm_open_session.   I want to confirm whether or not pam_sm_open_session is calling the "setcred" modules.

It is still not clear what you mean by this. The pam_sm_open_session is a function implemented by the various PAM modules - there is no "single source". Do you mean the source of pam_krb5 or pam_sss (which is actually the module that Fedora/RHEL uses for kerberos authentication)?

If you are talking here about the source of the Linux-PAM library then no, pam_open_session/pam_close_session definitely does not call setcred modules (i.e. configured in the auth PAM stack). It calls only the session modules (i.e. configured in the session PAM stack).

(In reply to Tomas Mraz from comment #27)
> > I would like to see the sources to RHEL's pam_sm_setcred and pam_sm_open_session.   I want to confirm whether or not pam_sm_open_session is calling the "setcred" modules.
> 
> It is still not clear what you mean by this. The pam_sm_open_session is a
> function implemented by the various PAM modules - there is no "single
> source". Do you mean the source of pam_krb5 or pam_sss (which is actually
> the module that Fedora/RHEL uses for kerberos authentication)?
>
> If you are talking here about the source of the Linux-PAM library then no,
> pam_open_session/pam_close_session definitely does not call setcred modules
> (i.e. configured in the auth PAM stack). It calls only the session modules
> (i.e. configured in the session PAM stack).

Please point me at the source for pam_krb5 and pam_sss implementations of pam_sm_setcred and pam_sm_open_session as well as the pam library.

I suspect the answer to the problem at hand is that the pam_sm_setcred that is storing krb5 credentials needs to do so into a MEMORY ccache and then pam_sm_open_session needs to copy the credentials from the MEMORY ccache to the permanent session ccache whatever type that is.

The pam environment variable KRB5CCNAME must be set by the krb5 pam_sm_setcred to the MEMORY ccache.   When the krb5 pam_sm_open_session executes and creates the permanent ccache it can update the pam KRB5CCNAME envvar to point at the permanent ccache.   As long as pam_sm_open_session functions use the pam envvar KRB5CCNAME they will have a valid krb5 ccache to use regardless of the ordering.

The PAM library is at https://github.com/linux-pam/linux-pam

The pam_sss is at https://github.com/SSSD/sssd/blob/master/src/sss_client/pam_sss.c

I am sorry for misleading you - there is actually no pam_krb5 AFAIK in RHEL-8.

> there is actually no pam_krb5 AFAIK in RHEL-8.

Correct - the intent is to use pam_sss (which implements most of its functionality) and pam_afs_session (which should handle the AFS use case).  The pam_krb5 package was marked deprecated around RHEL 7.5 and we haven't shipped it as part of RHEL 8.  (It's also gone from Fedora 32+.)  See https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.5_release_notes/chap-red_hat_enterprise_linux-7.5_release_notes-deprecated_functionality_in_rhel7

Ping.

Just curious the progress of the bug.

I'm requesting this be reopened.

we've found a significant amount of usability surrounding krb ticket caches, KCM and multiple caches being generated by new sessions and inherited by old sessions.  Often this leads to old sessions loosing network service access.

KRB5CCNAME should be locked to the cred cache generated on login and unique to that session.