Bug 845691 - ipa-client-install Failed to obtain host TGT
ipa-client-install Failed to obtain host TGT
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: ipa (Show other bugs)
6.3
Unspecified Unspecified
high Severity unspecified
: rc
: ---
Assigned To: Rob Crittenden
Namita Soman
:
Depends On:
Blocks: 976372
  Show dependency treegraph
 
Reported: 2012-08-03 18:07 EDT by Scott Poore
Modified: 2013-06-20 13:18 EDT (History)
4 users (show)

See Also:
Fixed In Version: ipa-3.0.0-1.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 976372 (view as bug list)
Environment:
Last Closed: 2013-02-21 04:17:02 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
IPA Replica dirsrv access log file (1.09 MB, application/octet-stream)
2012-08-03 18:07 EDT, Scott Poore
no flags Details

  None (edit)
Description Scott Poore 2012-08-03 18:07:33 EDT
Created attachment 602195 [details]
IPA Replica dirsrv access log file

Description of problem:

Occassionally ipa-client-install fails KRB pre-auth and rolls back like this:

# ipa-client-install  -U --domain=testrelm.com --realm=TESTRELM.COM -p admin -w PASSWORD --server=qe-blade-13.testrelm.com

Failed to obtain host TGT.
Discovery was successful!
Hostname: qe-blade-04.testrelm.com
Realm: TESTRELM.COM
DNS Domain: testrelm.com
IPA Server: qe-blade-13.testrelm.com
BaseDN: dc=testrelm,dc=com


Synchronizing time with KDC...

Enrolled in IPA realm TESTRELM.COM
Created /etc/ipa/default.conf
New SSSD config will be created.
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm TESTRELM.COM
Installation failed. Rolling back changes.

This is primarily being seen when trying to install clients (in automated test) when the server topology is a branched tree.  

Topology:

            M          
           / \       
          R1  R2          
         /    /\       
        R3   R4-R5       

C1 - C6 are balances across M, R1 - R5 where:
C1 -> M
C2 -> R1
...
C6 -> R5

In this particular case, it was C4 -> R3 but, I've also seen C1 -> M fail the same way.


Version-Release number of selected component (if applicable):
RHEL6.3 server
ipa-client-2.2.0-16.el6.x86_64
certmonger-0.56-1.el6.x86_64
krb5-workstation-1.9-33.el6.x86_64

How reproducible:
Unknown but is occurring noticably on automated tests.

Steps to Reproduce:
1.  <May need to build a similar server topology>
2.  ipa-client-install  -U --domain=$DOMAIN --realm=$RELM -p admin -w $ADMINPW --server=$SERVER
  
Actual results:
error as above.

Expected results:
no error and client now joined to IPA domain.


Additional info:

/var/log/ipaclient-install.log:
...
2012-08-03T15:45:59Z DEBUG args=/usr/bin/kinit -k -t /etc/krb5.keytab host/qe-blade-04.testrelm.com@TESTRELM.COM
2012-08-03T15:45:59Z DEBUG stdout=
2012-08-03T15:45:59Z DEBUG stderr=kinit: Preauthentication failed while getting initial credentials

2012-08-03T15:45:59Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
2012-08-03T15:45:59Z DEBUG args=/usr/bin/certutil -L -d /etc/pki/nssdb -n IPA CA
2012-08-03T15:45:59Z DEBUG stdout=Certificate:

 ...cert info here....let me know if needed.

2012-08-03T15:45:59Z DEBUG stderr=
2012-08-03T15:45:59Z DEBUG args=/usr/bin/certutil -D -d /etc/pki/nssdb -n IPA CA
2012-08-03T15:45:59Z DEBUG stdout=
2012-08-03T15:45:59Z DEBUG stderr=
2012-08-03T15:45:59Z DEBUG args=/sbin/service messagebus start 
2012-08-03T15:45:59Z DEBUG stdout=Starting system message bus: 

2012-08-03T15:45:59Z DEBUG stderr=
2012-08-03T15:45:59Z DEBUG args=/sbin/service certmonger start 
2012-08-03T15:45:59Z DEBUG stdout=
2012-08-03T15:45:59Z DEBUG stderr=
2012-08-03T15:45:59Z DEBUG args=/usr/bin/certutil -L -d /etc/pki/nssdb -n IPA Machine Certificate - qe-blade-04.testrelm.com
2012-08-03T15:45:59Z DEBUG stdout=
2012-08-03T15:45:59Z DEBUG stderr=certutil: Could not find cert: IPA Machine Certificate - qe-blade-04.testrelm.com
: File not found
...

Now it shows log entries for undoing config.

IPA SERVER httpd/access_log:

[root@qe-blade-13 log]# grep <IPADDRESS> httpd/access_log |cut -f2- -d' '
- - [03/Aug/2012:11:45:54 -0400] "GET /ipa/config/ca.crt HTTP/1.0" 200 1321 "-" "Wget/1.12 (linux-gnu)"
- - [03/Aug/2012:11:45:54 -0400] "GET /ipa/config/ca.crt HTTP/1.0" 200 1321 "-" "Wget/1.12 (linux-gnu)"
- - [03/Aug/2012:11:45:55 -0400] "POST /ipa/xml HTTP/1.1" 401 1856
- admin@TESTRELM.COM [03/Aug/2012:11:45:55 -0400] "POST /ipa/xml HTTP/1.1" 200 1925
- - [03/Aug/2012:11:46:01 -0400] "POST /ipa/xml HTTP/1.1" 401 1856
- host/qe-blade-04.testrelm.com@TESTRELM.COM [03/Aug/2012:11:46:01 -0400] "POST /ipa/xml HTTP/1.1" 200 427

IPA SERVER httpd/error_log:

[Fri Aug 03 11:45:56 2012] [error] ipa: INFO: admin@TESTRELM.COM: join(u'qe-blade-04.testrelm.com', nshardwareplatform=u'x86_64', nsosversion=u'2.6.32-279.el6.x86_64'): SUCCESS
[Fri Aug 03 11:46:02 2012] [error] ipa: INFO: host/qe-blade-04.testrelm.com@TESTRELM.COM: host_disable(u'qe-blade-04.testrelm.com'): SUCCESS
Comment 2 Dmitri Pal 2012-08-07 09:53:42 EDT
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/2982
Comment 3 Rob Crittenden 2012-08-08 10:02:37 EDT
Can you post some of the KDC log for the master the client was trying to connect to? That may explain why the kinit failed.
Comment 4 Scott Poore 2012-08-09 09:44:27 EDT
Here are some entries from a different server with the same client failure:

Aug 08 22:44:13 ipaqavme.testrelm.com krb5kdc[19230](info): TGS_REQ (4 etypes {18 17 16 23}) 10.16.98.190: ISSUE: 
authtime 1344480252, etypes {rep=18 tkt=18 ses=18}, admin@TESTRELM.COM for ldap/ipaqavme.testrelm.com@TESTRELM.COM
Aug 08 22:44:19 ipaqavme.testrelm.com krb5kdc[19230](info): AS_REQ (4 etypes {18 17 16 23}) 10.16.76.42: ISSUE: au
thtime 1344480259, etypes {rep=18 tkt=18 ses=18}, host/qe-blade-11.testrelm.com@TESTRELM.COM for krbtgt/TESTRELM.C
OM@TESTRELM.COM
Aug 08 22:44:20 ipaqavme.testrelm.com krb5kdc[19230](info): TGS_REQ (4 etypes {18 17 16 23}) 10.16.98.190: ISSUE: authtime 1344480259, etypes {rep=18 tkt=18 ses=18}, host/qe-blade-11.testrelm.com@TESTRELM.COM for ldap/ipaqavme.testrelm.com@TESTRELM.COM
Comment 5 Scott Poore 2012-08-09 09:57:05 EDT
Oh, and the client that failed in that case was trying to use a different node in the tree for its --server line.  It's CLIENT2 so it would be connecting to the R1 replica.  Just in case any of that matters or helps.
Comment 6 Petr Viktorin 2012-08-31 10:07:10 EDT
Requested in the upstream ticket:  Can you take a network trace while this happens?
Comment 9 Scott Poore 2012-09-04 11:13:41 EDT
added tcpdump capture attachments but, had to mark them private.  Let me know if you cannot see them.
Comment 10 Rob Crittenden 2012-09-11 16:41:18 EDT
Fixed upstream.

It's due to the client using DNS to find a KDC that hasn't yet replicated info about the new client, and then asking it for a TGT. How to reproduce:

- Install a "master" and "replica"
- Change the Kerberos DNS entries to only point to the replica:
    for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do
        ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 $REPLICA_HOSTNAME"
    done
    ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 $MASTER_HOSTNAME"
    ipa dnsrecord-find $DOMAIN  # check
- Sever communication between the hosts, disabling replication:
    on master:
    # iptables -A INPUT -j DROP -p all --source $REPLICA_IP
- On client machine, put master as nameserver in /etc/resolv.conf & install client

master: 1f831393a24c0c3c7df50c4327395f1197f485e5

ipa-3-0: 25ddbfca5ef7456103970e4a915e2979876178fa
Comment 12 Scott Poore 2012-10-03 21:26:32 EDT
Is that reproducing the same problem?

When I use that method to reproduce the issue on ipa-server-2.2.0-16, I see this 

#### From command output:

[root@vm3 log]# ipa-client-install -U --domain=$DOMAIN --realm=$RELM -p $ADMINID -w $ADMINPW
Discovery was successful!
Hostname: vm3.testrelm.com
Realm: TESTRELM.COM
DNS Domain: testrelm.com
IPA Server: vm1.testrelm.com
BaseDN: dc=testrelm,dc=com


Synchronizing time with KDC...

Enrolled in IPA realm TESTRELM.COM
Created /etc/ipa/default.conf
Domain testrelm.com is already configured in existing SSSD config, creating a new one.
The old /etc/sssd/sssd.conf is backed up and will be restored during uninstall.
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm TESTRELM.COM
Failed to obtain host TGT.
Installation failed. Rolling back changes.

#### in ipaclient-install.log:

2012-10-04T01:01:30Z DEBUG args=/usr/bin/kinit -k -t /etc/krb5.keytab host/vm3.testrelm.com@TESTRELM.CO
M
2012-10-04T01:01:30Z DEBUG stdout=
2012-10-04T01:01:30Z DEBUG stderr=kinit: Client 'host/vm3.testrelm.com@TESTRELM.COM' not found in Kerberos database while getting initial credentials

##############################

However, looking back at one of the cases that prompted this ticket, I see this:

#### From command output:

:: [10:33:24] ::  RUN ipa-client-install
Failed to obtain host TGT.
Discovery was successful!
Hostname: qe-blade-09.testrelm.com
Realm: TESTRELM.COM
DNS Domain: testrelm.com
IPA Server: qe-blade-04.testrelm.com
BaseDN: dc=testrelm,dc=com


Synchronizing time with KDC...

Enrolled in IPA realm TESTRELM.COM
Created /etc/ipa/default.conf
New SSSD config will be created.
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm TESTRELM.COM
Installation failed. Rolling back changes.
:: [   FAIL   ] :: Running 'ipa-client-install  -U --domain=testrelm.com --realm=TESTRELM.COM -p admin -w Secret123 --server=qe-blade-04.testrelm.com' (Expected 0, got 1)

#### in ipaclient-install.log:

2012-09-02T14:33:29Z DEBUG args=/usr/bin/kinit -k -t /etc/krb5.keytab host/qe-blade-09.testrelm.com@TESTRELM.COM
2012-09-02T14:33:29Z DEBUG stdout=
2012-09-02T14:33:29Z DEBUG stderr=kinit: Preauthentication failed while getting initial credentials

##############################

So, questions:

1.) Does the locaion of the TGT error there matter that it differs?

2.) Do the different log messages for kinit mean different things?
Comment 13 Petr Viktorin 2012-10-04 10:38:17 EDT
1) I think the reordering is because in 2.2, the two messages are printed to different streams (stdout and stderr).

2) Yes, the log messages are suspicious. However, the net trace you provided confirms that the error came from the other server. There are many variables in play -- e.g. in what sequence the servers are contacted or how far the replication has gone -- so a different log message is possible.

Of course it's always possible that there is another problem with similar symptoms. It would probably be best if you left the net logging on.
Comment 17 Scott Poore 2012-10-04 11:17:19 EDT
Attached 3 tcpdumps but, had to mark them private.  Let me know if you cannot see them.
Comment 18 Petr Viktorin 2012-10-05 11:10:27 EDT
Okay, these dumps show a slightly different error. Again, though, the client is trying to reach a KDC on a different master than it's installing on, which wouldn't happen with the patch applied.
Comment 19 Scott Poore 2012-10-08 16:58:17 EDT
Verified.

Version::

ipa-client-3.0.0-2.el6.x86_64
krb5-workstation-1.10.3-6.el6.x86_64
sssd-1.9.2-0.20121005T1740Zgita87ffe4.el6.x86_64

Automated Test Results ::

Had to run these manually for now.

[root@vm3 install-client-cli]# ipaclientinstall_bugcheck_845691_fulltest

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: ipaclientinstall_bugcheck_845691_fulltest - ipa-client-install Failed to obtain host TGT
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [16:46:20] ::  Uninstall for next test
:: [16:46:20] ::  Uninstalling ipa client for next test
Unenrolling client from IPA server
Removing Kerberos service principals from /etc/krb5.keytab
Disabling client Kerberos and LDAP configurations
Redundant SSSD configuration file /etc/sssd/sssd.conf was moved to /etc/sssd/sssd.conf.deleted
Restoring client configuration files
nslcd daemon is not installed, skip configuration
Client uninstall complete.
:: [16:46:30] ::  sssd.conf for testing BZ 819982 does not exists
  Record name: _kerberos-master._tcp
  SRV record: 0 100 88 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos-master._tcp --srv-rec='0 100 88 vm2'"'
  Record name: _kerberos-master._udp
  SRV record: 0 100 88 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos-master._udp --srv-rec='0 100 88 vm2'"'
  Record name: _kerberos._tcp
  SRV record: 0 100 88 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos._tcp --srv-rec='0 100 88 vm2'"'
  Record name: _kerberos._udp
  SRV record: 0 100 88 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos._udp --srv-rec='0 100 88 vm2'"'
  Record name: _kpasswd._tcp
  SRV record: 0 100 464 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kpasswd._tcp --srv-rec='0 100 464 vm2'"'
  Record name: _kpasswd._udp
  SRV record: 0 100 464 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kpasswd._udp --srv-rec='0 100 464 vm2'"'
  Record name: _ldap._tcp
  SRV record: 0 100 389 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _ldap._tcp --srv-rec='0 100 389 vm1'"'
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "service iptables stop"'
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "iptables -A INPUT -j DROP -p all --source 192.168.122.102"'
:: [16:46:41] ::  EXECUTING: ipa-client-install --domain=testrelm.com --realm=TESTRELM.COM -p admin -w Secret123 --unattended --server=vm1.testrelm.com
:: [   PASS   ] :: Installing ipa client and configuring - with all params

:: [   PASS   ] :: BZ 845691 not found
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^above is the important line^^^^^^^^^^^^^^^^^^^^

  Record name: _kerberos-master._tcp
  SRV record: 0 100 88 vm2, 0 100 88 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos-master._tcp --srv-rec='0 100 88 vm1, 0 100 88 vm2'"'
  Record name: _kerberos-master._udp
  SRV record: 0 100 88 vm2, 0 100 88 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos-master._udp --srv-rec='0 100 88 vm1, 0 100 88 vm2'"'
  Record name: _kerberos._tcp
  SRV record: 0 100 88 vm2, 0 100 88 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos._tcp --srv-rec='0 100 88 vm1, 0 100 88 vm2'"'
  Record name: _kerberos._udp
  SRV record: 0 100 88 vm2, 0 100 88 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos._udp --srv-rec='0 100 88 vm1, 0 100 88 vm2'"'
  Record name: _kpasswd._tcp
  SRV record: 0 100 464 vm2, 0 100 464 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kpasswd._tcp --srv-rec='0 100 464 vm1, 0 100 464 vm2'"'
  Record name: _kpasswd._udp
  SRV record: 0 100 464 vm2, 0 100 464 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kpasswd._udp --srv-rec='0 100 464 vm1, 0 100 464 vm2'"'
  Record name: _ldap._tcp
  SRV record: 0 100 389 vm1, 0 100 389 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _ldap._tcp --srv-rec='0 100 389 vm1, 0 100 389 vm2'"'
iptables: Flushing firewall rules: [  OK  ]
iptables: Setting chains to policy ACCEPT: filter [  OK  ]
iptables: Unloading modules: [  OK  ]
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "service iptables stop"'


Manual Test Results ::

This is more of a followup to the automated test above.  

[root@vm3 install-client-cli]# grep kinit: /var/log/ipaclient-install.log 
[root@vm3 install-client-cli]# 

[root@vm3 install-client-cli]# cat /tmp/bz845691.testout 
Hostname: vm3.testrelm.com
Realm: TESTRELM.COM
DNS Domain: testrelm.com
IPA Server: vm1.testrelm.com
BaseDN: dc=testrelm,dc=com
Synchronizing time with KDC...
Enrolled in IPA realm TESTRELM.COM
Created /etc/ipa/default.conf
New SSSD config will be created
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm TESTRELM.COM
trying https://vm1.testrelm.com/ipa/xml
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_dsa_key.pub
Forwarding 'host_mod' to server u'http://vm1.testrelm.com/ipa/xml'
SSSD enabled
Configured /etc/openldap/ldap.conf
NTP enabled
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Client configuration complete.

[root@vm3 install-client-cli]# 
[root@vm3 install-client-cli]# CHK1=$(grep "kinit: Preauthentication failed while getting initial credentials" /var/log/ipaclient-install.log|wc -l) 
[root@vm3 install-client-cli]# if [ $CHK1 -gt 0 ]; then
>     rlLog "[FAIL1] BZ 845691 found...ipa-client-install Failed to obtain host TGT"
>     submit_log /var/log/ipaclient-install.log
> fi  
[root@vm3 install-client-cli]# 
[root@vm3 install-client-cli]# CHK2=$(grep "kinit: Client.*not found in Kerberos database while getting initial credentials" /var/log/ipaclient-install.log|wc -l) 
[root@vm3 install-client-cli]# if [ $CHK2 -gt 0 ]; then
>     rlLog "[FAIL2] BZ 845691 found...ipa-client-install Failed to obtain host TGT"
>     submit_log /var/log/ipaclient-install.log
> fi
Comment 22 errata-xmlrpc 2013-02-21 04:17:02 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0528.html
Comment 23 dhayes501 2013-05-08 15:21:09 EDT
I had this same issue: Failed to obtain host TGT.

My issue was I had a different name for the client in /etc/hosts

local ip: 192.168.1.20

/etc/sysconfig/network
hostname=ar7s8b

/etc/hosts
192.168.1.20 ar

when I ran ipa-client-install, it was recognizing my hostname as "ar" instead of "ar7s8b".

Probably an edge case, but I found this thread searching for the solution to my problem, so maybe my solution will help someone else...

Note You need to log in before you can comment on or make changes to this bug.