Description of problem:

I'm still seeing issues with getent lookups of AD users after trust is setup even when time is in sync.

I've tried changing both the DNS domain and the REALM of the server to something entirely new/unique to our tests.  This was to confirm that something wasn't cached on AD side.  Yet, I'm still seeing this failure even after some waiting and restarts.


Version-Release number of selected component (if applicable):
  ipa-server-trust-ad.x86_64 0:3.3.1-3.el7                                      
  samba-client.x86_64 0:4.1.0-0.7.rc3.el7                                       
  samba-debuginfo.x86_64 0:4.1.0-0.7.rc3.el7                                    
  samba-winbind-clients.x86_64 0:4.1.0-0.7.rc3.el7                              

How reproducible:
always  in my test environment.

Steps to Reproduce:
1.  Setup AD Server
2.  Install IPA Server  
3.  yum -y install samba-client samba-winbind-clients \
                ipa-server-trust-ad samba-debuginfo
4.  ipa-adtrust-install --netbios-name=$NBNAME -a $ADMINPW -U
5.  ipa dnszone-add $ADdomain --name-server=$ADhost. \
                --admin-email="hostmaster@$ADdomain" --force \
                --forwarder=$ADip --forward-policy=only --ip-address=$ADip
6.  service named reload
7.  Add conditional forwarder on AD server pointing to IPA domain/server
8.  echo $ADMINPW | ipa trust-add $ADdomain --admin $ADadmin \
                --range-type=ipa-ad-trust --password
9.  vi /etc/krb5.conf  # add to [realms]
{{{
  auth_to_local = RULE:[1:$1@$0](^.*@ADTEST.QE$)s/@ADTEST.QE/@adtest.qe/
  auth_to_local = DEFAULT
}}}
10.  service krb5kdc restart
11.  service sssd stop
12.  rm -rf /var/lib/sss/{db,mc}/*
13.  service sssd start
14.  getent passwd 'ADTEST\Administrator'

Actual results:
returns nothing most of the time.  I have seen it work once when I logged in after the test ran.

Expected results:
Works and returns ADTEST\Administrator getent info:

Additional info:

I'm also seeing these sssd errors:

(Mon Sep 23 13:11:37 2013) [sssd[be[spoore220.test]]] [sasl_bind_send] (0x0100): Executing sasl bind mech: gssapi, user: host/ibm-x3650m4-01-vm-02.spoore220.test
(Mon Sep 23 13:11:38 2013) [sssd[be[spoore220.test]]] [sasl_bind_send] (0x0020): ldap_sasl_bind failed (-2)[Local error]
(Mon Sep 23 13:11:38 2013) [sssd[be[spoore220.test]]] [sasl_bind_send] (0x0080): Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (KDC policy rejects request)]
(Mon Sep 23 13:11:38 2013) [sssd[be[spoore220.test]]] [child_sig_handler] (0x1000): Waiting for child [24240].


More info from test case where I'm consistently seeing this issue:

0 100 389 ad12srv1.adtest.qe.
:: [   PASS   ] :: Running 'dig +short SRV _ldap._tcp.adtest.qe' (Expected 0, got 0)
--------------------------------------------------
Added Active Directory trust for realm "adtest.qe"
--------------------------------------------------
  Realm name: adtest.qe
  Domain NetBIOS name: ADTEST
  Domain Security Identifier: S-1-5-21-1910160501-511572375-3625658879
  SID blacklist incoming: S-1-0, S-1-1, S-1-2, S-1-3, S-1-5-1, S-1-5-2, S-1-5-3, S-1-5-4, S-1-5-5, S-1-5-6, S-1-5-7, S-1-5-8, S-1-5-9, S-1-5-10, S-1-5-11, S-1-5-12, S-1-5-13, S-1-5-14, S-1-5-15, S-1-5-16, S-1-5-17, S-1-5-18, S-1-5-19, S-1-5-20
  SID blacklist outgoing: S-1-0, S-1-1, S-1-2, S-1-3, S-1-5-1, S-1-5-2, S-1-5-3, S-1-5-4, S-1-5-5, S-1-5-6, S-1-5-7, S-1-5-8, S-1-5-9, S-1-5-10, S-1-5-11, S-1-5-12, S-1-5-13, S-1-5-14, S-1-5-15, S-1-5-16, S-1-5-17, S-1-5-18, S-1-5-19, S-1-5-20
  Trust direction: Two-way trust
  Trust type: Active Directory domain
  Trust status: Established and verified
:: [   PASS   ] :: Running 'echo Secret123 | ipa trust-add adtest.qe --admin Administrator                 --range-type=ipa-ad-trust --password' (Expected 0, got 0)
:: [ 22:11:40 ] ::  add auth_to_local mappings to /etc/krb5.conf
:: [   PASS   ] :: File '/etc/krb5.conf' should contain 'dns_lookup_kdc.*true' 
includedir /var/lib/sss/pubconf/krb5.include.d/

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = SPOORE220.TEST
 dns_lookup_realm = false
 dns_lookup_kdc = true
 rdns = false
 ticket_lifetime = 24h
 forwardable = yes

[realms]
 SPOORE220.TEST = {
  kdc = ibm-x3650m4-01-vm-02.spoore220.test:88
  master_kdc = ibm-x3650m4-01-vm-02.spoore220.test:88
  admin_server = ibm-x3650m4-01-vm-02.spoore220.test:749
  default_domain = spoore220.test
  pkinit_anchors = FILE:/etc/ipa/ca.crt
  auth_to_local = RULE:[1:$1@$0](^.*@ADTEST.QE$)s/@ADTEST.QE/@adtest.qe/
  auth_to_local = DEFAULT
}

[domain_realm]
 .spoore220.test = SPOORE220.TEST
 spoore220.test = SPOORE220.TEST

[dbmodules]
  SPOORE220.TEST = {
    db_library = ipadb.so
  }

:: [   PASS   ] :: Running 'cat /etc/krb5.conf' (Expected 0, got 0)
Redirecting to /bin/systemctl stop  krb5kdc.service
:: [   PASS   ] :: Running 'service krb5kdc stop' (Expected 0, got 0)
Redirecting to /bin/systemctl start  krb5kdc.service
:: [   PASS   ] :: Running 'service krb5kdc start' (Expected 0, got 0)
Redirecting to /bin/systemctl stop  sssd.service
:: [   PASS   ] :: Running 'service sssd stop' (Expected 0, got 0)
:: [   PASS   ] :: Running 'rm -rf /var/lib/sss/{db,mc}/*' (Expected 0, got 0)
Redirecting to /bin/systemctl start  sssd.service
:: [   PASS   ] :: Running 'service sssd start' (Expected 0, got 0)
:: [   PASS   ] :: Running 'wbinfo --online-status > /tmp/tmpout.ipa_trust_func_setup_check 2>&1' (Expected 0, got 0)
BUILTIN : online
SPOORE220 : online
:: [   PASS   ] :: Running 'cat /tmp/tmpout.ipa_trust_func_setup_check' (Expected 0, got 0)
:: [ 22:11:41 ] ::  Sleeping 300s to wait for winbind cache timeout

MARK-LWD-LOOP -- 2013-09-22 22:14:59 --
:: [   PASS   ] :: Running 'sleep 300' (Expected 0, got 0)
:: [   PASS   ] :: Running 'wbinfo --online-status > /tmp/tmpout.ipa_trust_func_setup_check 2>&1' (Expected 0, got 0)
BUILTIN : online
SPOORE220 : online
ADTEST : offline
:: [   PASS   ] :: Running 'cat /tmp/tmpout.ipa_trust_func_setup_check' (Expected 0, got 0)
:: [   PASS   ] :: Winbind found AD Netbios after cache expired: ADTEST 
:: [ 22:16:42 ] ::  getent...
:: [   FAIL   ] :: Running 'getent passwd 'ADTEST\Administrator'' (Expected 0, got 2)
:: [   FAIL   ] :: Running 'getent group 'ADTEST\Domain Users'' (Expected 0, got 2)
Redirecting to /bin/systemctl restart  winbind.service
:: [   PASS   ] :: Running 'service winbind restart' (Expected 0, got 0)
Redirecting to /bin/systemctl restart  smb.service
:: [   PASS   ] :: Running 'service smb restart' (Expected 0, got 0)
:: [   FAIL   ] :: Running 'getent passwd 'ADTEST\Administrator'' (Expected 0, got 2)
:: [   FAIL   ] :: Running 'getent group 'ADTEST\Domain Users'' (Expected 0, got 2)