Description of problem
======================

I wrote script automating kerberos keytab setup which uses ipa-getkeytab
tool to do the job. The problem is that after exporting several keytabs, this
script breaks authorization for ipa admin tools (so that it's no longer
possible to use eg. ipa user-show).

Version-Release number of selected component (if applicable)
============================================================

# rpm -qa | egrep '(^ipa|krb|sssd)'
krb5-libs-1.10.3-33.el6.x86_64
sssd-ad-1.11.6-30.el6_6.3.x86_64
sssd-proxy-1.11.6-30.el6_6.3.x86_64
sssd-1.11.6-30.el6_6.3.x86_64
python-krbV-1.0.90-3.el6.x86_64
sssd-client-1.11.6-30.el6_6.3.x86_64
krb5-server-1.10.3-33.el6.x86_64
sssd-common-1.11.6-30.el6_6.3.x86_64
sssd-common-pac-1.11.6-30.el6_6.3.x86_64
sssd-ipa-1.11.6-30.el6_6.3.x86_64
sssd-krb5-1.11.6-30.el6_6.3.x86_64
python-sssdconfig-1.11.6-30.el6_6.3.noarch
ipa-client-3.0.0-42.el6.x86_64
ipa-server-3.0.0-42.el6.x86_64
ipa-pki-common-theme-9.0.3-7.el6.noarch
ipa-pki-ca-theme-9.0.3-7.el6.noarch
pam_krb5-2.3.11-9.el6.x86_64
krb5-workstation-1.10.3-33.el6.x86_64
sssd-krb5-common-1.11.6-30.el6_6.3.x86_64
sssd-ldap-1.11.6-30.el6_6.3.x86_64
ipa-python-3.0.0-42.el6.x86_64
ipa-admintools-3.0.0-42.el6.x86_64
ipa-server-selinux-3.0.0-42.el6.x86_64

How reproducible
================

100% (for my cluster configuration - see below)

Steps to Reproduce
==================

The setup of the cluster is completelly automated and so I would be able to
provide access to broken cluster on request. Quick description including the
way we use ipa tools to make it clear how we use it follows:

1)    setup cluster with 3 nodes with IPA
1.1)  master is configured as ipa server
      code snipplet used to install ipa-server:

~~~
rlRun "ipa-server-install \
  --hostname=${GH_IPASERVER} \
  --ip-address=${ip} \
  --realm=${GH_IPA_REALM} \
  --domain=${GH_IPA_DOMAIN} \
  --ds-password=${GH_IPA_PASSWD} \
  --admin-password=${GH_IPA_PASSWD} \
  --unattended \
  --no-ntp"
~~~

1.2) two nodes are configured as ipa clients
     code snipplet:

~~~
rlRun "ipa-client-install \
  --server=${GH_IPASERVER} \
  --realm=${GH_IPA_REALM} \
  --domain=${GH_IPA_DOMAIN} \
  --principal=${GH_IPA_ADMIN} \
  --password=${GH_IPA_PASSWD} \
  --unattended \
  --mkhomedir"
~~~

1.3) create some users via ipa add-user (not sure if needed)
     code snipplet:

~~~
rlRun "ipa user-add ${name} --first ${name} --last ${name} \
  --homedir ${home} --shell ${shell} --noprivate \
  --uid ${uid} --gidnumber ${gid}"
~~~

2) run the keytab script on master machine (admin ticket is requred):

~~~
# kinit admin
# ipa-hadoop-service-setup.sh $HOSTNAME host-principal-keytab-list.csv
~~~

Where:

 * ipa-hadoop-service-setup.sh - script attached to this BZ, creates keytabs
 * $HOSTNAME should contain fqdn of master machine
 * host-principal-keytab-list.csv - contains descript of keytabs, attached as
   well

Note: To catch the moment when the problem happens, I added the following code
after every operation with keytabs:

~~~
ipa user-show ambari-qa
if [[ $? -ne 0 ]]; then
  echo "fail: IPA is broken!" >&2
  exit 22
fi
~~~

Actual results
==============

During the key export phase, IPA auth for cli admin tolls stops working
completelly.

~~~
# exporting keytab for hbase.COM on dhcp-lab-193.englab.brq.redhat.com
Keytab successfully retrieved and stored in: tmp.keytab
  User login: ambari-qa
  First name: ambari-qa
  Last name: ambari-qa
  Home directory: /home/ambari-qa
  Login shell: /bin/bash
  Email address: ambari-qa.redhat.com
  UID: 20507
  GID: 20400
  Account disabled: False
  Password: False
  Member of groups: ipausers
  Kerberos keys available: True
  User login: ambari-qa
  First name: ambari-qa
  Last name: ambari-qa
  Home directory: /home/ambari-qa
  Login shell: /bin/bash
  Email address: ambari-qa.redhat.com
  UID: 20507
  GID: 20400
  Account disabled: False
  Password: False
  Member of groups: ipausers
  Kerberos keys available: True
tmp.keytab                                                                           100%  290     0.3KB/s   00:00
  User login: ambari-qa
  First name: ambari-qa
  Last name: ambari-qa
  Home directory: /home/ambari-qa
  Login shell: /bin/bash
  Email address: ambari-qa.redhat.com
  UID: 20507
  GID: 20400
  Account disabled: False
  Password: False
  Member of groups: ipausers
  Kerberos keys available: True
# exporting keytab for HTTP/dhcp-lab-193.englab.brq.redhat.com.COM on dhcp-lab-193.englab.brq.redhat.com
Keytab successfully retrieved and stored in: tmp.keytab
ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (KDC returned error string: 2ND_TKT_SERVER)
fail: IPA is broken!
~~~

In this case, keytab was successfully exported for
hbase.COM, but the next export for
HTTP/dhcp-lab-193.englab.brq.redhat.com.COM fails (which is
13th keytab being created there).

Then I can try to reauthenticate, but this doesn't help:

~~~
# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: admin.COM

Valid starting     Expires            Service principal
03/06/15 19:32:02  03/07/15 19:31:56  krbtgt/RHSHADOOPQA.REDHAT.COM.COM
03/06/15 19:33:22  03/07/15 19:31:56  HTTP/dhcp-lab-193.englab.brq.redhat.com.COM
03/06/15 19:37:39  03/07/15 19:31:56  ldap/dhcp-lab-193.englab.brq.redhat.com.COM
03/06/15 19:37:42  03/07/15 19:31:56  host/dhcp-lab-171.englab.brq.redhat.com.COM
03/06/15 19:38:55  03/07/15 19:31:56  host/dhcp-lab-193.englab.brq.redhat.com.COM
# kdestroy
# klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_0)
# kinit admin
Password for admin.COM:
# ipa user-show ambari-qa
ipa: ERROR: cannot connect to u'https://dhcp-lab-193.englab.brq.redhat.com/ipa/xml': Authorization Required
# keyctl list @s
1 key in keyring:
192158270: --alswrv     0    -1 keyring: _uid.0
#
~~~

Note that keyctl doesn't show auth cookie as expected for some reason.

Expected results
================

Authorization is not broken after running given script so that I'm still able
to use ipa admin tools as admin user.

Eg. this would still work as shown below:

~~~
# kinit admin
# ipa user-show ambari-qa
  User login: ambari-qa
  First name: ambari-qa
  Last name: ambari-qa
  Home directory: /home/ambari-qa
  Login shell: /bin/bash
  Email address: ambari-qa.redhat.com
  UID: 20507
  GID: 20400
  Account disabled: False
  Password: False
  Member of groups: ipausers
  Kerberos keys available: False
# keyctl list @s
2 keys in keyring:
192158270: --alswrv     0    -1 keyring: _uid.0
816027917: --alswrv     0     0 user: ipa_session_cookie:admin.COM
~~~

Additional info
===============

While it is possible that my keytab script is not optimal (comments welcome),
it should not break authorization for ipa tools.