Bug 1375656

Summary:	ipa-replica-install with 4.3.2-2.fc24 fails with CCacheError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (39756044): Credential cache is empty
Product:	Red Hat Enterprise Linux 7	Reporter:	Petr Vobornik <pvoborni>
Component:	ipa	Assignee:	IPA Maintainers <ipa-maint>
Status:	CLOSED WORKSFORME	QA Contact:	Kaleem <ksiddiqu>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.3	CC:	abokovoy, extras-qa, ipa-maint, jcholast, jhrozek, jpazdziora, mbabinsk, mkosek, pvoborni, rcritten, ssorce
Target Milestone:	rc	Keywords:	Regression
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	Cause: When `ipa-replica-install` is run with `--setup-ca` and `setup-dns` options without existing ccache present(without ipa client installed first). Consequence: Replica installation on domain level 1 without client installed first will fail. Workaround (if any): Run `ipa-client-install` first, kinit as privileged user (or use OTP) and run `ipa-replica-install`. Result: Replica is installed on domain level 1.	Story Points:	---
Clone Of:	1373883	Environment:
Last Closed:	2016-09-13 16:37:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:	1373883
Bug Blocks:

Description Petr Vobornik 2016-09-13 15:33:01 UTC

+++ This bug was initially created as a clone of Bug #1373883 +++

Description of problem:

ipa-replica-install with 4.3.2-2.fc24 fails with

  [22/24]: Restart HTTP server to pick up changes
  [23/24]: enabling CA instance
  [24/24]: Updating DNS CA records
  [error] CCacheError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (39756044): Credential cache is empty
ipa.ipapython.install.cli.install_tool(Replica): ERROR    Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (39756044): Credential cache is empty
ipa.ipapython.install.cli.install_tool(Replica): ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

Version-Release number of selected component (if applicable):

freeipa-server-4.3.2-2.fc24

How reproducible:

Deterministic for ipa-replica-install runs that reach this far.

Steps to Reproduce:
1. Have 4.3.2-2.fc24 master.
2. Have another Fedora 24 machine with freeipa-server-4.3.2-2.fc24 bits installed.
3. Run /usr/sbin/ipa-replica-install -U --setup-ca --setup-dns --forwarder=10.11.12.13 --ip-address=10.11.12.50 -P admin -w TheVery5ecretPa55word

Actual results:

  [14/24]: importing CA chain to RA certificate database
  [15/24]: fixing RA database permissions
  [16/24]: setting up signing cert profile
  [17/24]: setting audit signing renewal to 2 years
  [18/24]: configure certificate renewals
  [19/24]: configure Server-Cert certificate renewal
  [20/24]: Configure HTTP to proxy connections
  [21/24]: updating IPA configuration
  [22/24]: Restart HTTP server to pick up changes
  [23/24]: enabling CA instance
  [24/24]: Updating DNS CA records
  [error] CCacheError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (39756044): Credential cache is empty
ipa.ipapython.install.cli.install_tool(Replica): ERROR    Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (39756044): Credential cache is empty
ipa.ipapython.install.cli.install_tool(Replica): ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

The log ends with

2016-09-07T09:59:32Z DEBUG   duration: 1 seconds
2016-09-07T09:59:32Z DEBUG   [23/24]: enabling CA instance
2016-09-07T09:59:32Z DEBUG Starting external process
2016-09-07T09:59:32Z DEBUG args=/bin/systemctl disable pki-tomcatd.target
2016-09-07T09:59:32Z DEBUG Process finished, return code=0
2016-09-07T09:59:32Z DEBUG stdout=
2016-09-07T09:59:32Z DEBUG stderr=Removed symlink /etc/systemd/system/multi-user.target.wants/pki-tomcatd.target.

2016-09-07T09:59:32Z DEBUG   duration: 0 seconds
2016-09-07T09:59:32Z DEBUG   [24/24]: Updating DNS CA records
2016-09-07T09:59:32Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-TEST.socket from SchemaCache
2016-09-07T09:59:32Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-TEST.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f888caeffc8>
2016-09-07T09:59:32Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2016-09-07T09:59:32Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
2016-09-07T09:59:32Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 447, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 437, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1299, in __update_ca_records
    bind.add_ipa_ca_dns_records(api.env.host, api.env.domain)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/bindinstance.py", line 1086, in add_ipa_ca_dns_records
    self.api.Backend.ldap2.connect(autobind=True)
  File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 66, in connect
    conn = self.create_connection(*args, **kw)
  File "/usr/lib/python2.7/site-packages/ipaserver/plugins/ldap2.py", line 199, in create_connection
    principal = krb_utils.get_principal(ccache_name=ccache)
  File "/usr/lib/python2.7/site-packages/ipalib/krb_utils.py", line 185, in get_principal
    raise errors.CCacheError(message=unicode(e))
CCacheError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (39756044): Credential cache is empty

2016-09-07T09:59:32Z DEBUG   [error] CCacheError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (39756044): Credential cache is empty
2016-09-07T09:59:32Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 318, in run
    cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 310, in run
    self.execute()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 332, in execute
    for nothing in self._executor():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 372, in __runner
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 394, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 362, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 359, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 586, in _configure
    next(executor)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 372, in __runner
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 449, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 394, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 446, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 394, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 362, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 359, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install
    for nothing in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1687, in main
    promote(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 377, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1484, in promote
    ca_cert_bundle=ca_data)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1371, in configure_replica
    self.start_creation(runtime=210)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 447, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 437, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1299, in __update_ca_records
    bind.add_ipa_ca_dns_records(api.env.host, api.env.domain)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/bindinstance.py", line 1086, in add_ipa_ca_dns_records
    self.api.Backend.ldap2.connect(autobind=True)
  File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 66, in connect
    conn = self.create_connection(*args, **kw)
  File "/usr/lib/python2.7/site-packages/ipaserver/plugins/ldap2.py", line 199, in create_connection
    principal = krb_utils.get_principal(ccache_name=ccache)
  File "/usr/lib/python2.7/site-packages/ipalib/krb_utils.py", line 185, in get_principal
    raise errors.CCacheError(message=unicode(e))

2016-09-07T09:59:32Z DEBUG The ipa-replica-install command failed, exception: CCacheError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (39756044): Credential cache is empty
2016-09-07T09:59:32Z ERROR Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (39756044): Credential cache is empty
2016-09-07T09:59:32Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

Expected results:

No error, replica set up.

Additional info:

--- Additional comment from Jan Pazdziora on 2016-09-07 13:19:34 CEST ---

Reproducer beaker jobs:

https://beaker.engineering.redhat.com/jobs/1490407
https://beaker.engineering.redhat.com/jobs/1490408
https://beaker.engineering.redhat.com/jobs/1490410

--- Additional comment from Martin Babinsky on 2016-09-12 14:43:57 CEST ---

I was able to reproduce this bug locally. It happens when `ipa-replica-install` is run with `--setup-ca` and `setup-dns` options without existing ccache present.

A workaround is to first run `ipa-client-install`, kinit as privileged user (or use OTP) and run `ipa-replica-install`.

--- Additional comment from Martin Babinsky on 2016-09-12 14:49:14 CEST ---

Upstream ticket:
https://fedorahosted.org/freeipa/ticket/6299

Comment 3 Martin Babinsky 2016-09-13 16:37:42 UTC

The issue is actually not present in RHEL 7.3 builds due to refactoring of DNS record reation/update done during implementation of DNS Locations feature.

Verified to work using ipa-server-4.4.0-10.el7.x86_64