Bug 1689908

For convenience, the reported unhandled python exception is this:

ext_cred_store.pyx:182:gssapi.raw.ext_cred_store.acquire_cred_from:gssapi.raw.misc.GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (39756032): Principal in credential cache does not match desired name

Traceback (most recent call last):
  File "/usr/libexec/ipa/oddjob/com.redhat.idm.trust-fetch-domains", line 129, in <module>
    cred = kinit_keytab(principal, keytab_name, ccache_name)
  File "/usr/lib/python3.6/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab
    cred = gssapi.Credentials(name=name, store=store, usage='initiate')
  File "/usr/local/lib64/python3.6/site-packages/gssapi/creds.py", line 64, in __new__
    store=store)
  File "/usr/local/lib64/python3.6/site-packages/gssapi/creds.py", line 148, in acquire
    usage)
  File "gssapi/raw/ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from
gssapi.raw.misc.GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (39756032): Principal in credential cache does not match desired name

Local variables in innermost frame:
__name__: 'gssapi.raw.ext_cred_store'
__doc__: 'Credential Store Extension'
__package__: 'gssapi.raw'
__loader__: <_frozen_importlib_external.ExtensionFileLoader object at 0x7f4d8eb82320>
__spec__: ModuleSpec(name='gssapi.raw.ext_cred_store', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f4d8eb82320>, origin='/usr/local/lib64/python3.6/site-packages/gssapi/raw/ext_cred_store.cpython-36m-x86_64-linux-gnu.so')
__builtins__: <module 'builtins' (built-in)>
GSSAPI: 'BASE'
namedtuple: <function namedtuple at 0x7f4da78291e0>
AddCredResult: <class 'gssapi.raw.named_tuples.AddCredResult'>
AcquireCredResult: <class 'gssapi.raw.named_tuples.AcquireCredResult'>
StoreCredResult: <class 'gssapi.raw.named_tuples.StoreCredResult'>
GSSError: <class 'gssapi.raw.misc.GSSError'>
acquire_cred_from: <built-in function acquire_cred_from>
add_cred_from: <built-in function add_cred_from>
store_cred_into: <built-in function store_cred_into>
__test__: {}
__file__: '/usr/local/lib64/python3.6/site-packages/gssapi/raw/ext_cred_store.cpython-36m-x86_64-linux-gnu.so'

The reason it happened is because we have the following code in the oddjob helper:
---------------------
keytab_name = '/etc/samba/samba.keytab'

principal = str('cifs/' + api.env.host)

oneway_ccache_name = '/var/run/ipa/krb5cc_oddjob_trusts_fetch'
ccache_name = '/var/run/ipa/krb5cc_oddjob_trusts'

# Standard sequence:
# - check if ccache exists
#   - if not, initialize it from Samba's keytab
# - check if ccache contains valid TGT
#   - if not, initialize it from Samba's keytab
# - refer the correct ccache object for further use
#
have_ccache = False
try:
    cred = kinit_keytab(principal, keytab_name, ccache_name)
    if cred.lifetime > 0:
        have_ccache = True
except gssapi.exceptions.ExpiredCredentialsError:
    pass
if not have_ccache:
    # delete stale ccache and try again
    if os.path.exists(oneway_ccache_name):
        os.unlink(ccache_name)
    cred = kinit_keytab(principal, keytab_name, ccache_name)
---------------------

if kinit_keytab() raises any other exception, we fail to delete stale ccache because we never reach that code path.

On the test system we had
[root@ad-test ~]# klist -k -t /etc/samba/samba.keytab 
Keytab name: FILE:/etc/samba/samba.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 07.03.2019 04.01.14 cifs/ad-test.testrelm.test
   1 07.03.2019 04.01.14 cifs/ad-test.testrelm.test
[root@ad-test ~]# ll /var/run/ipa/krb5cc_oddjob_trusts
-rw-------. 1 root root 1845 26. 2. 01:01 /var/run/ipa/krb5cc_oddjob_trusts
[root@ad-test ~]# klist -c /var/run/ipa/krb5cc_oddjob_trusts
Ticket cache: FILE:/var/run/ipa/krb5cc_oddjob_trusts
Default principal: cifs/ad-test.trustcli16.test

Valid starting       Expires              Service principal
26.02.2019 01.01.11  27.02.2019 01.01.11  krbtgt/TRUSTCLI16.TEST


So there was stale ccache from old installation which used completely different Kerberos realm, thus the issue reported by gssapi code.

A solution is to treat all gssapi errors during kinit as a sign that we should clean up the stale ccache.

Upstream ticket:
https://pagure.io/freeipa/issue/8333

Once the upstream community implements this feature it will be pulled into a corresponding Red Hat Enterprise Linux release following the corresponding schedules.
From now on this issue will be tracked in the community issue tracker only.
Closing this BZ.