Can you re-check with a different credential cache type, e.g. FILE:, to make sure it is not related to the recent KEYRING related issues?

No luck.

[root@rhel7-1 pam.d]# export KRB5CCNAME=/tmp/krb5cc_$(id -u)

[root@rhel7-1 pam.d]# kdestroy

[root@rhel7-1 pam.d]# export KRB5CCNAME=/tmp/krb5cc_$(id -u)

[root@rhel7-1 pam.d]# kinit aduser.TEST
Password for aduser.TEST: 

[root@rhel7-1 pam.d]# ssh -K -l aduser.TEST rhel7-1.testrelm.com
aduser.TEST.com's password: 

[root@rhel7-1 pam.d]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: aduser.TEST

Valid starting       Expires              Service principal
11/26/2013 13:57:52  11/26/2013 23:57:52  krbtgt/AD2.EXAMPLE.TEST.TEST
	renew until 11/27/2013 13:57:50
11/26/2013 13:58:08  11/26/2013 23:57:52  krbtgt/TESTRELM.COM.TEST
	renew until 11/27/2013 13:57:50
11/26/2013 13:58:08  11/26/2013 23:57:52  host/rhel7-1.testrelm.com
	renew until 11/27/2013 13:57:50

Created attachment 829460 [details]
tcpdump during ssh attempt

Created attachment 829466 [details]
var log files

I just realized that I hadn't setup the .k5login for the user.

So, before I close this as not a bug, I've got a question:

Could something have changed to start enforcing this?  

This had previously worked without a .k5login in automation test script:

Password for aduser1: 
:: [   PASS   ] :: Running 'echo Secret123|kinit aduser1' (Expected 0, got 0)

Could not chdir to home directory /home/aduser1: No such file or directory
login successful
:: [   PASS   ] :: Running 'timeout 20s             ssh -K -l aduser1 ipaqa64vmi.spoore11250932.test 'echo login successful'' (Expected 0, got 0)

thanks,
Scott

Sumit or Alexander, can you please answer to Scott?

Using .k5login bypasses pam_sss authentication. We really need to solve the issue on ssshd or sssd side with GSSAPI not being accepted somehow.

Alexander, does the Scott's scenario work in you environment? If not, maybe Scott could prepare an env with reproduction that you or Sumit could look at.

Scott, you always have to either set auth_to_local in krb5.conf or create a .k5login file. I guess if it worked before without .k5login auth_to_local was configured.

Alexander, .5login does not bypass the PAM authentication step, it's the GSSAPI authentication itself that bypasses it. .k5login is only needed to see if the Kerberos principal is allowed to access the system as the corresponding local user. The sssd side of making this more reliable is already tracked by  https://fedorahosted.org/sssd/ticket/1835 , we are just waiting for for MIT krb5-1.12 be be released :-)

Martin, if both .k5login file and auth_to_local entries are missing the failure is expected.

Right, adding needsinfo for Scott to confirm that he has auth_to_local set correctly.

Yes, I have auth_to_local set like this in krb5.conf:

  auth_to_local = RULE:[1:$1@$0](^.*@AD2.EXAMPLE.TEST$)s/@AD2.EXAMPLE.TEST/@ad2.example.test/
  auth_to_local = DEFAULT

And that's what automation was doing.  At the moment, I'm having an issue (unrelated) with my test env where I first saw this.  I'm trying to setup another for further testing.

Thanks,
Scott

Maybe my other issue isn't unrelated.  I think I was confusing things.

Last night, I re-ran ipa-adtrust-install on the server that was already set up.  Selected yes for everything and was NOT prompted for NETBIOS name.  So looked good for the test I was running. 

Now, I think I'm seeing the same thing even with .k5login for a user so not sure what's happening.

Sumit filed bug today for OpenSSH making wrong selection of the primary ccache in the ccache collection: bug #1035254. I get the same behavior in F19, it looks like F20 and RHEL7 are affected too.

Yes, Sumit confirmed that this is indeed bug #1035254.  

I'm switching this over to openssh.  A note that Sumit pointed out for a workaround:

Run "kdestroy -A"

And, because auth_to_local is translating to lower case, use -l aduser.test in lower case to match that. 

FYI, results of testing showing bug #1035254:

[root@rhel7-1 log]# klist -l
Principal name                 Cache name
--------------                 ----------
aduser2.TEST       KEYRING:persistent:0:krb_ccache_P9dvtt8
admin             KEYRING:persistent:0:krb_ccache_IX4v52E
aduser.TEST        KEYRING:persistent:0:0 (Expired)

[root@rhel7-1 log]# klist -A
Ticket cache: KEYRING:persistent:0:krb_ccache_P9dvtt8
Default principal: aduser2.TEST

Valid starting       Expires              Service principal
11/27/2013 09:04:03  11/27/2013 19:04:03  krbtgt/AD2.EXAMPLE.TEST.TEST
	renew until 11/28/2013 09:04:01

klist: No credentials cache found while retrieving principal name

Ticket cache: KEYRING:persistent:0:krb_ccache_IX4v52E
Default principal: admin

Valid starting       Expires              Service principal
11/26/2013 20:37:32  11/27/2013 20:35:06  host/rhel7-1.testrelm.com
11/26/2013 20:35:06  11/27/2013 20:35:06  krbtgt/TESTRELM.COM

Ticket cache: KEYRING:persistent:0:0
Default principal: aduser.TEST

Valid starting       Expires              Service principal
11/26/2013 22:22:35  11/27/2013 08:22:35  krbtgt/AD2.EXAMPLE.TEST.TEST
	renew until 11/27/2013 22:22:33

[root@rhel7-1 log]# kdestroy -A

[root@rhel7-1 log]# klist -A

[root@rhel7-1 log]# kinit aduser.TEST
Password for aduser.TEST: 

[root@rhel7-1 log]# klist -A
Ticket cache: KEYRING:persistent:0:krb_ccache_P9dvtt8
Default principal: aduser.TEST

Valid starting       Expires              Service principal
11/27/2013 09:24:32  11/27/2013 19:24:24  host/rhel7-1.testrelm.com
	renew until 11/28/2013 09:24:22
11/27/2013 09:24:32  11/27/2013 19:24:24  krbtgt/TESTRELM.COM.TEST
	renew until 11/28/2013 09:24:22
11/27/2013 09:24:24  11/27/2013 19:24:24  krbtgt/AD2.EXAMPLE.TEST.TEST
	renew until 11/28/2013 09:24:22

[root@rhel7-1 ~]# ssh -l aduser.test $(hostname)
Last login: Wed Nov 27 09:26:09 2013 from rhel7-1.testrelm.com
-sh-4.2$  exit
logout

[root@rhel7-1 ~]# ssh -K -l aduser.test $(hostname)
Last login: Wed Nov 27 09:27:37 2013 from rhel7-1.testrelm.com
-sh-4.2$

I've done some dicussion with Simo and he pointed out that this is expected behavior for Kerberos credentials cache collections as described at http://k5wiki.kerberos.org/wiki/Projects/Client_principal_selection

What we need to do is to convert this bug into documentation bug for Kerberos. At the very least, this change of the behavior due to introduction of the ccache collections should go to release notes (Fedora 20 and RHEL7).

I don't think this is an openssh bug, this is expected behavior when a user has 2 TGTs.
The one matching the REALM of the target server is chosen. The GSSAPI libraries have no way to know that the 2 realms have a trust, moreover the library has no way to read the user's mind and guess that he want's to use the "farthest" identity even if it knows the 2 realms have a trust relationship.

Please read here how principal selection operates with collection caches:
 http://k5wiki.kerberos.org/wiki/Projects/Client_principal_selection

I think this is just a documentation bug and should be moved back to the ipa component.

Regarding QE test, it need to be modified to clean up credentials before testing trust configurations to ensure this behaviour isn't triggered.

Ok. Changing to doc component for IPA to let Deon drive the documentation change.

Document verified