Bug 845691

Upstream ticket:
https://fedorahosted.org/freeipa/ticket/2982

Can you post some of the KDC log for the master the client was trying to connect to? That may explain why the kinit failed.

Here are some entries from a different server with the same client failure:

Aug 08 22:44:13 ipaqavme.testrelm.com krb5kdc[19230](info): TGS_REQ (4 etypes {18 17 16 23}) 10.16.98.190: ISSUE: 
authtime 1344480252, etypes {rep=18 tkt=18 ses=18}, admin for ldap/ipaqavme.testrelm.com
Aug 08 22:44:19 ipaqavme.testrelm.com krb5kdc[19230](info): AS_REQ (4 etypes {18 17 16 23}) 10.16.76.42: ISSUE: au
thtime 1344480259, etypes {rep=18 tkt=18 ses=18}, host/qe-blade-11.testrelm.com for krbtgt/TESTRELM.C
OM
Aug 08 22:44:20 ipaqavme.testrelm.com krb5kdc[19230](info): TGS_REQ (4 etypes {18 17 16 23}) 10.16.98.190: ISSUE: authtime 1344480259, etypes {rep=18 tkt=18 ses=18}, host/qe-blade-11.testrelm.com for ldap/ipaqavme.testrelm.com

Oh, and the client that failed in that case was trying to use a different node in the tree for its --server line.  It's CLIENT2 so it would be connecting to the R1 replica.  Just in case any of that matters or helps.

Requested in the upstream ticket:  Can you take a network trace while this happens?

added tcpdump capture attachments but, had to mark them private.  Let me know if you cannot see them.

Fixed upstream.

It's due to the client using DNS to find a KDC that hasn't yet replicated info about the new client, and then asking it for a TGT. How to reproduce:

- Install a "master" and "replica"
- Change the Kerberos DNS entries to only point to the replica:
    for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do
        ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 $REPLICA_HOSTNAME"
    done
    ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 $MASTER_HOSTNAME"
    ipa dnsrecord-find $DOMAIN  # check
- Sever communication between the hosts, disabling replication:
    on master:
    # iptables -A INPUT -j DROP -p all --source $REPLICA_IP
- On client machine, put master as nameserver in /etc/resolv.conf & install client

master: 1f831393a24c0c3c7df50c4327395f1197f485e5

ipa-3-0: 25ddbfca5ef7456103970e4a915e2979876178fa

Is that reproducing the same problem?

When I use that method to reproduce the issue on ipa-server-2.2.0-16, I see this 

#### From command output:

[root@vm3 log]# ipa-client-install -U --domain=$DOMAIN --realm=$RELM -p $ADMINID -w $ADMINPW
Discovery was successful!
Hostname: vm3.testrelm.com
Realm: TESTRELM.COM
DNS Domain: testrelm.com
IPA Server: vm1.testrelm.com
BaseDN: dc=testrelm,dc=com


Synchronizing time with KDC...

Enrolled in IPA realm TESTRELM.COM
Created /etc/ipa/default.conf
Domain testrelm.com is already configured in existing SSSD config, creating a new one.
The old /etc/sssd/sssd.conf is backed up and will be restored during uninstall.
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm TESTRELM.COM
Failed to obtain host TGT.
Installation failed. Rolling back changes.

#### in ipaclient-install.log:

2012-10-04T01:01:30Z DEBUG args=/usr/bin/kinit -k -t /etc/krb5.keytab host/vm3.testrelm.com
M
2012-10-04T01:01:30Z DEBUG stdout=
2012-10-04T01:01:30Z DEBUG stderr=kinit: Client 'host/vm3.testrelm.com' not found in Kerberos database while getting initial credentials

##############################

However, looking back at one of the cases that prompted this ticket, I see this:

#### From command output:

:: [10:33:24] ::  RUN ipa-client-install
Failed to obtain host TGT.
Discovery was successful!
Hostname: qe-blade-09.testrelm.com
Realm: TESTRELM.COM
DNS Domain: testrelm.com
IPA Server: qe-blade-04.testrelm.com
BaseDN: dc=testrelm,dc=com


Synchronizing time with KDC...

Enrolled in IPA realm TESTRELM.COM
Created /etc/ipa/default.conf
New SSSD config will be created.
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm TESTRELM.COM
Installation failed. Rolling back changes.
:: [   FAIL   ] :: Running 'ipa-client-install  -U --domain=testrelm.com --realm=TESTRELM.COM -p admin -w Secret123 --server=qe-blade-04.testrelm.com' (Expected 0, got 1)

#### in ipaclient-install.log:

2012-09-02T14:33:29Z DEBUG args=/usr/bin/kinit -k -t /etc/krb5.keytab host/qe-blade-09.testrelm.com
2012-09-02T14:33:29Z DEBUG stdout=
2012-09-02T14:33:29Z DEBUG stderr=kinit: Preauthentication failed while getting initial credentials

##############################

So, questions:

1.) Does the locaion of the TGT error there matter that it differs?

2.) Do the different log messages for kinit mean different things?

1) I think the reordering is because in 2.2, the two messages are printed to different streams (stdout and stderr).

2) Yes, the log messages are suspicious. However, the net trace you provided confirms that the error came from the other server. There are many variables in play -- e.g. in what sequence the servers are contacted or how far the replication has gone -- so a different log message is possible.

Of course it's always possible that there is another problem with similar symptoms. It would probably be best if you left the net logging on.

Attached 3 tcpdumps but, had to mark them private.  Let me know if you cannot see them.

Okay, these dumps show a slightly different error. Again, though, the client is trying to reach a KDC on a different master than it's installing on, which wouldn't happen with the patch applied.

Verified.

Version::

ipa-client-3.0.0-2.el6.x86_64
krb5-workstation-1.10.3-6.el6.x86_64
sssd-1.9.2-0.20121005T1740Zgita87ffe4.el6.x86_64

Automated Test Results ::

Had to run these manually for now.

[root@vm3 install-client-cli]# ipaclientinstall_bugcheck_845691_fulltest

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: ipaclientinstall_bugcheck_845691_fulltest - ipa-client-install Failed to obtain host TGT
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [16:46:20] ::  Uninstall for next test
:: [16:46:20] ::  Uninstalling ipa client for next test
Unenrolling client from IPA server
Removing Kerberos service principals from /etc/krb5.keytab
Disabling client Kerberos and LDAP configurations
Redundant SSSD configuration file /etc/sssd/sssd.conf was moved to /etc/sssd/sssd.conf.deleted
Restoring client configuration files
nslcd daemon is not installed, skip configuration
Client uninstall complete.
:: [16:46:30] ::  sssd.conf for testing BZ 819982 does not exists
  Record name: _kerberos-master._tcp
  SRV record: 0 100 88 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos-master._tcp --srv-rec='0 100 88 vm2'"'
  Record name: _kerberos-master._udp
  SRV record: 0 100 88 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos-master._udp --srv-rec='0 100 88 vm2'"'
  Record name: _kerberos._tcp
  SRV record: 0 100 88 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos._tcp --srv-rec='0 100 88 vm2'"'
  Record name: _kerberos._udp
  SRV record: 0 100 88 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos._udp --srv-rec='0 100 88 vm2'"'
  Record name: _kpasswd._tcp
  SRV record: 0 100 464 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kpasswd._tcp --srv-rec='0 100 464 vm2'"'
  Record name: _kpasswd._udp
  SRV record: 0 100 464 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kpasswd._udp --srv-rec='0 100 464 vm2'"'
  Record name: _ldap._tcp
  SRV record: 0 100 389 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _ldap._tcp --srv-rec='0 100 389 vm1'"'
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "service iptables stop"'
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "iptables -A INPUT -j DROP -p all --source 192.168.122.102"'
:: [16:46:41] ::  EXECUTING: ipa-client-install --domain=testrelm.com --realm=TESTRELM.COM -p admin -w Secret123 --unattended --server=vm1.testrelm.com
:: [   PASS   ] :: Installing ipa client and configuring - with all params

:: [   PASS   ] :: BZ 845691 not found
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^above is the important line^^^^^^^^^^^^^^^^^^^^

  Record name: _kerberos-master._tcp
  SRV record: 0 100 88 vm2, 0 100 88 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos-master._tcp --srv-rec='0 100 88 vm1, 0 100 88 vm2'"'
  Record name: _kerberos-master._udp
  SRV record: 0 100 88 vm2, 0 100 88 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos-master._udp --srv-rec='0 100 88 vm1, 0 100 88 vm2'"'
  Record name: _kerberos._tcp
  SRV record: 0 100 88 vm2, 0 100 88 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos._tcp --srv-rec='0 100 88 vm1, 0 100 88 vm2'"'
  Record name: _kerberos._udp
  SRV record: 0 100 88 vm2, 0 100 88 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kerberos._udp --srv-rec='0 100 88 vm1, 0 100 88 vm2'"'
  Record name: _kpasswd._tcp
  SRV record: 0 100 464 vm2, 0 100 464 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kpasswd._tcp --srv-rec='0 100 464 vm1, 0 100 464 vm2'"'
  Record name: _kpasswd._udp
  SRV record: 0 100 464 vm2, 0 100 464 vm1
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _kpasswd._udp --srv-rec='0 100 464 vm1, 0 100 464 vm2'"'
  Record name: _ldap._tcp
  SRV record: 0 100 389 vm1, 0 100 389 vm2
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "ipa dnsrecord-mod testrelm.com _ldap._tcp --srv-rec='0 100 389 vm1, 0 100 389 vm2'"'
iptables: Flushing firewall rules: [  OK  ]
iptables: Setting chains to policy ACCEPT: filter [  OK  ]
iptables: Unloading modules: [  OK  ]
:: [   PASS   ] :: Running 'ssh vm1.testrelm.com "service iptables stop"'


Manual Test Results ::

This is more of a followup to the automated test above.  

[root@vm3 install-client-cli]# grep kinit: /var/log/ipaclient-install.log 
[root@vm3 install-client-cli]# 

[root@vm3 install-client-cli]# cat /tmp/bz845691.testout 
Hostname: vm3.testrelm.com
Realm: TESTRELM.COM
DNS Domain: testrelm.com
IPA Server: vm1.testrelm.com
BaseDN: dc=testrelm,dc=com
Synchronizing time with KDC...
Enrolled in IPA realm TESTRELM.COM
Created /etc/ipa/default.conf
New SSSD config will be created
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm TESTRELM.COM
trying https://vm1.testrelm.com/ipa/xml
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_dsa_key.pub
Forwarding 'host_mod' to server u'http://vm1.testrelm.com/ipa/xml'
SSSD enabled
Configured /etc/openldap/ldap.conf
NTP enabled
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Client configuration complete.

[root@vm3 install-client-cli]# 
[root@vm3 install-client-cli]# CHK1=$(grep "kinit: Preauthentication failed while getting initial credentials" /var/log/ipaclient-install.log|wc -l) 
[root@vm3 install-client-cli]# if [ $CHK1 -gt 0 ]; then
>     rlLog "[FAIL1] BZ 845691 found...ipa-client-install Failed to obtain host TGT"
>     submit_log /var/log/ipaclient-install.log
> fi  
[root@vm3 install-client-cli]# 
[root@vm3 install-client-cli]# CHK2=$(grep "kinit: Client.*not found in Kerberos database while getting initial credentials" /var/log/ipaclient-install.log|wc -l) 
[root@vm3 install-client-cli]# if [ $CHK2 -gt 0 ]; then
>     rlLog "[FAIL2] BZ 845691 found...ipa-client-install Failed to obtain host TGT"
>     submit_log /var/log/ipaclient-install.log
> fi

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0528.html

I had this same issue: Failed to obtain host TGT.

My issue was I had a different name for the client in /etc/hosts

local ip: 192.168.1.20

/etc/sysconfig/network
hostname=ar7s8b

/etc/hosts
192.168.1.20 ar

when I ran ipa-client-install, it was recognizing my hostname as "ar" instead of "ar7s8b".

Probably an edge case, but I found this thread searching for the solution to my problem, so maybe my solution will help someone else...