RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1404910 - IPA server upgrade fails with Kerberos database internal error.
Summary: IPA server upgrade fails with Kerberos database internal error.
Keywords:
Status: CLOSED DUPLICATE of bug 1402810
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.3
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: ---
Assignee: IPA Maintainers
QA Contact: Kaleem
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-12-15 04:40 UTC by Nikhil Dehadrai
Modified: 2016-12-16 15:00 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-16 15:00:22 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Nikhil Dehadrai 2016-12-15 04:40:09 UTC 
Description of problem:
IPA upgrade fails with Kerberos database internal error.

Version-Release number of selected component (if applicable):
ipa-server-4.4.0-14.el7_3.3.x86_64

How reproducible:
always

Steps to Reproduce:
1. Setup IPA server with 7.3GA version.
2. Configure latest repo for RHEL-7.3 async ipa security errata.
3. Initiate upgrade process using command 'yum install -y 'ipa*' sssd'

Actual results:
1. After step 3, eventhough ipa-server package is updated but the ipa-server upgrade fails with following message at the console:

  Cleanup    : libini_config-1.2.0-25.el7.x86_64                        146/146
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
OSError: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/dogtag.keytab'
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
  Verifying  : sssd-common-pac-1.14.0-43.el7_3.11.x86_64                  1/146
  Verifying  : tomcat-lib-7.0.69-10.el7.noarch                            2/146
  Verifying  : pki-ca-10.3.3-15.el7_3.noarch                              3/146
  Verifying  : policycoreutils-python-2.5-11.el7_3.x86_64                 4/146

2. ipaupgrade.log returns "Kerberos database internal error".
2016-12-14T11:03:58Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-TESTRELM-TEST.socket from SchemaCache
2016-12-14T11:03:58Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-TESTRELM-TEST.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0xd0b3320>
2016-12-14T11:03:59Z INFO Retrieving keytab
2016-12-14T11:03:59Z DEBUG Starting external process
2016-12-14T11:03:59Z DEBUG args=kadmin.local -q ktadd -k /etc/pki/pki-tomcat/dogtag.keytab dogtag/auto-hv-01-guest10.testrelm.test -x ipa-setup-override-restrictions
2016-12-14T11:03:59Z DEBUG Process finished, return code=0
2016-12-14T11:03:59Z DEBUG stdout=Authenticating as principal admin/admin with password.
 
2016-12-14T11:03:59Z DEBUG stderr=kadmin.local: Kerberos database internal error while changing dogtag/auto-hv-01-guest10.testrelm.test's key
 
2016-12-14T11:03:59Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2016-12-14T11:03:59Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run
    server.upgrade()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1863, in upgrade
    upgrade_configuration()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1796, in upgrade_configuration
    ca.setup_lightweight_ca_key_retrieval()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1404, in setup_lightweight_ca_key_retrieval
    self.__setup_lightweight_ca_key_retrieval_kerberos()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1435, in __setup_lightweight_ca_key_retrieval_kerberos
    os.chmod(keytab, 0o600)
 
2016-12-14T11:03:59Z DEBUG The ipa-server-upgrade command failed, exception: OSError: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/dogtag.keytab'
2016-12-14T11:03:59Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:
OSError: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/dogtag.keytab'
2016-12-14T11:03:59Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information


Expected results:
No error message should be observed and the upgrade process should be successful.

Additional info:
Similar behavior is observed during 7.2.z > 7.3.1 upgrade.
Comment 3 Kaleem 2016-12-15 08:59:15 UTC 
Testblocker keyword removed as IPA usable after upgarde.
Comment 9 Martin Babinsky 2016-12-15 12:31:55 UTC 
The error occurs when you try to use kadmin.local to re-set Kerberos key for a service located in the 'cn=services,cn=accounts,$SUFFIX' subtree. Re-setting the key using ipa-getkeytab works, as well as re-setting a key using kadmin.local on a service located in 'cn=$REALM,cn=kerberos,$SUFFIX' subtree.

I have worked out the following minimal reproducer:

1.) install IPA server using ipa-server-4.4.0-14.el7_3.1.x86_64 RPMs
2.) add a service principal to 'cn=kerberos' subtree using kadmin.local:

"""
[root@master1 ~]# kadmin.local -q 'addprinc -randkey svc/master1.ipa.test' -x ipa-setup-override-restrictions
Authenticating as principal admin/admin with password.
WARNING: no policy specified for svc/master1.ipa.test; defaulting to no policy
Principal "svc/master1.ipa.test" created.
"""

3.) Create the service keytab using kadmin.local:

"""
[root@master1 ~]# kadmin.local -q 'ktadd -k svc.keytab svc/master1.ipa.test' -x ipa-setup-override-restrictions
Authenticating as principal admin/admin with password.
Entry for principal svc/master1.ipa.test with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:svc.keytab.
<SNIP>
"""

4a.) Verify that re-setting the Kerberos key works using kadmin by running the command from 3.) again

"""
# kadmin.local -q 'ktadd -k svc.keytab svc/master1.ipa.test' -x ipa-setup-override-restrictions
Authenticating as principal admin/admin with password.
Entry for principal svc/master1.ipa.test with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:svc.keytab.
<SNIP>
"""
Notice the kvno=3 in this case so the key was re-created successfully.

4b.) verify that ipa-getkeytab also works:

"""
# ipa-getkeytab -p svc/`hostname` -k svc.keytab 
Keytab successfully retrieved and stored in: svc.keytab
"""

5.)
Now delete the service and create the same service principal using IPA API which places it in 'cn=services,cn=accounts' subtree

"""
[root@master1 ~]# ipa service-add svc/`hostname`
---------------------------------------------
Added service "svc/master1.ipa.test"
---------------------------------------------
  Principal name: svc/master1.ipa.test
  Principal alias: svc/master1.ipa.test
  Managed by: master1.ipa.test
"""

6a.) Repeat step 3 or 4a (fetching service key via kadmin.local) and notice that database internal error is thrown:

"""
[root@master1 ~]#  kadmin.local -q 'ktadd -k svc.keytab svc/master1.ipa.test' -x ipa-setup-override-restrictions
Authenticating as principal admin/admin with password.
kadmin.local: Kerberos database internal error while changing svc/master1.ipa.test's key
"""

6b.) repeat step 4b (request keytab via ipa-getkeytab) and see it succeed
"""
[root@master1 ~]# ipa-getkeytab -p svc/`hostname` -k svc.keytab 
Keytab successfully retrieved and stored in: svc.keytab
"""

Expected result:

Steps 3, 4a, 4b, 6a, and 6b all should succeed

Actual result:

Step 6a fails with the error seen in the report.
Comment 10 Martin Babinsky 2016-12-15 14:15:11 UTC 
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/6561
Comment 11 Alexander Bokovoy 2016-12-15 15:40:37 UTC 
Pull request https://github.com/freeipa/freeipa/pull/345 fixes the problem.
Comment 12 Martin Babinsky 2016-12-15 16:35:35 UTC 
Fixed upstream
ipa-4-3:
https://fedorahosted.org/freeipa/changeset/b9b919e127c453eda02ea142d7cd80c16aa5ca31
ipa-4-4:
https://fedorahosted.org/freeipa/changeset/84f6df6349b5c412467746777e905d9e4f8792ca
master:
https://fedorahosted.org/freeipa/changeset/73f33569c8893610e246b2f44a7aeaec872b37e6
Comment 13 Petr Vobornik 2016-12-16 15:00:22 UTC 
fixed in 
 4.4.0-14.1.1
 4.4.0-14.4

caused by bug in patch for bug 1402810

*** This bug has been marked as a duplicate of bug 1402810 ***
Note You need to log in before you can comment on or make changes to this bug.