Bug 1310141

Summary: trusted AD domain, AD users can only logon via Kerberos not using password
Product: Red Hat Enterprise Linux 7 Reporter: Silvio Wanka <Silvio.Wanka>
Component: sssdAssignee: SSSD Maintainers <sssd-maint>
Status: CLOSED NOTABUG QA Contact: Steeve Goveas <sgoveas>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.2CC: grajaiya, jhrozek, lslebodn, mkosek, mzidek, pbrezina, preichl, pvoborni, rcritten, sbose, Silvio.Wanka
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-26 11:59:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
debug_level=6 for the most sssd.conf sections
none
the requested new logs
none
sssd.conf from both servers and client
none
Re: Comment #23 from Sumit Bose none

Description Silvio Wanka 2016-02-19 15:04:40 UTC
I have add a trust to an AD domain. Now I try to logon as an user of this AD domain using ssh (PuTTY) to an ipa-client of the Unix domain. The ipa-client-install was done some weeks before the trust was created and the client is still 7.1 (ipa 4.1) and the server in meantime 7.2 (ipa 4.2).
If I try to logon from a Windows session there I'm logged on with the same user which I give as unix logon name then a kerberos ticket will be used an logon is successfully. But if try to logon with the same Ad user but the local Windows logon user is a different then I will of course asked for a passwort but I get allways "Access denied".

In the sssd logs are nothing but /var/log/secure you can see the first sucessfully logon via /var/log/secure and then "Failed password" for the definitly rigth password. An kinit on the same server for the same user (except realm in uppercase) works with the same password!

Feb 19 14:25:35 splunk01 sshd[12964]: error: AuthorizedKeysCommand /usr/bin/sss_ssh_authorizedkeys returned status 1
Feb 19 14:25:37 splunk01 sshd[12964]: Authorized to aduser, krb5 principal aduser (ssh_gssapi_krb5_cmdok)
Feb 19 14:25:41 splunk01 sshd[12964]: Accepted gssapi-with-mic for aduser from x.x.x.x port 52353 ssh2
Feb 19 14:25:41 splunk01 sshd[12964]: pam_unix(sshd:session): session opened for user aduser by (uid=0)

Feb 19 14:29:25 splunk01 sshd[13053]: error: AuthorizedKeysCommand /usr/bin/sss_ssh_authorizedkeys returned status 1
Feb 19 14:30:40 splunk01 sshd[13053]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=sdecent-mgmt01.ad-company.com  user=aduser
Feb 19 14:30:41 splunk01 sshd[13053]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=sdecent-mgmt01.ad-company.com user=aduser
Feb 19 14:30:41 splunk01 sshd[13053]: pam_sss(sshd:auth): received for user aduser: 4 (System error)
Feb 19 14:30:43 splunk01 sshd[13053]: Failed password for aduser from y.y.y.y port 64921 ssh2
Feb 19 14:30:54 splunk01 sshd[13053]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=sdecent-mgmt01.ad-company.com user=aduser
Feb 19 14:30:54 splunk01 sshd[13053]: pam_sss(sshd:auth): received for user aduser: 4 (System error)
Feb 19 14:30:56 splunk01 sshd[13053]: Failed password for aduser from y.y.y.y port 64921 ssh2

What is wrong here? Must something changed in sssd.conf?

TIA,
Silvio

Comment 2 Petr Vobornik 2016-02-19 15:12:56 UTC
sssd issue, changing component

Comment 3 Jakub Hrozek 2016-02-19 15:39:20 UTC
Please follow https://fedorahosted.org/sssd/wiki/Troubleshooting to generate the sssd logs and attach them to this bugzilla.

Thanks!

Comment 4 Lukas Slebodnik 2016-02-19 15:42:14 UTC
Can you reproduce with client on rhel 7.2?
There should be sssd >= 1.13.0-40.el7_2.1

Comment 5 Silvio Wanka 2016-02-19 16:17:59 UTC
(In reply to Lukas Slebodnik from comment #4)

Only both IdM servers are 7.2 and there the logon is also not possible but also not with kerberos (maybe on this servers sshd config not changed).

Comment 6 Lukas Slebodnik 2016-02-22 08:34:47 UTC
(In reply to Silvio Wanka from comment #5)
> (In reply to Lukas Slebodnik from comment #4)
> 
> Only both IdM servers are 7.2 and there the logon is also not possible but
> also not with kerberos (maybe on this servers sshd config not changed).

You wrote in description of this bug:
   "the client is still 7.1 (ipa 4.1)"

Please provide version of sssd on machine where you want to connect
and also log files. @see Comment 3

Comment 7 Silvio Wanka 2016-02-22 15:21:00 UTC
Created attachment 1129360 [details]
debug_level=6 for the most sssd.conf sections

You can find two logons, first from a Windows Desktop there I was logged on as same user

   siwa   pts/1        172.21.1.21      Mon Feb 22 13:52"

The second at Feb 22 13:54 from a server there I was logged on as different user. I have typed my password 4 times but no success.

Comment 8 Silvio Wanka 2016-02-22 15:52:14 UTC
$ rpm -qa | fgrep sssd
sssd-client-1.12.2-58.el7_1.18.x86_64
sssd-ipa-1.12.2-58.el7_1.18.x86_64
sssd-ldap-1.12.2-58.el7_1.18.x86_64
sssd-common-pac-1.12.2-58.el7_1.18.x86_64
sssd-proxy-1.12.2-58.el7_1.18.x86_64
python-sssdconfig-1.12.2-58.el7_1.18.noarch
sssd-common-1.12.2-58.el7_1.18.x86_64
sssd-krb5-1.12.2-58.el7_1.18.x86_64
sssd-krb5-common-1.12.2-58.el7_1.18.x86_64
sssd-ad-1.12.2-58.el7_1.18.x86_64
sssd-1.12.2-58.el7_1.18.x86_64

Comment 9 Lukas Slebodnik 2016-02-22 16:21:08 UTC
(In reply to Silvio Wanka from comment #8)
> $ rpm -qa | fgrep sssd
> sssd-client-1.12.2-58.el7_1.18.x86_64
> sssd-ipa-1.12.2-58.el7_1.18.x86_64
> sssd-ldap-1.12.2-58.el7_1.18.x86_64
> sssd-common-pac-1.12.2-58.el7_1.18.x86_64
> sssd-proxy-1.12.2-58.el7_1.18.x86_64
> python-sssdconfig-1.12.2-58.el7_1.18.noarch
> sssd-common-1.12.2-58.el7_1.18.x86_64
> sssd-krb5-1.12.2-58.el7_1.18.x86_64
> sssd-krb5-common-1.12.2-58.el7_1.18.x86_64
> sssd-ad-1.12.2-58.el7_1.18.x86_64
> sssd-1.12.2-58.el7_1.18.x86_64

I can see in krb5_child.log that sssd tried to use enterprise
principal for AD user. And it's  known issue.

[unpack_buffer] (0x0100): cmd [241] uid [142603268] gid [142603268] validate [true] enterprise principal [false] offline [false] UPN [siwa]
[get_and_save_tgt] (0x0400): Attempting kinit for realm [COMPANY.COM] 
[get_and_save_tgt] (0x0020): 996: [-1765328230][Cannot find KDC for realm "COMPANY.COM"]
[map_krb5_error] (0x0020): 1065: [-1765328230][Cannot find KDC for realm "COMPANY.COM"]
[k5c_send_data] (0x0200): Received error code 1432158209
[main] (0x0400): krb5_child completed successfully

Enterprise principals are not supported in a IPA-AD trust scenario, but one can 
work around that by using:      
    subdomain_inherit = ldap_user_principal
    ldap_user_principal = nosuchattr
and thus tricking sssd into 'deriving' the UPN from the domain name.

But workaround with subdomain_inherit needn't be available on el7.1 (sssd-1.12.2-58.el7_1.18.x86_64) I do not remember it from top of my head.
Please try to upgrade this machine to el7.2 with sssd-1.13.0

Comment 10 Silvio Wanka 2016-02-23 09:11:50 UTC
(In reply to Lukas Slebodnik from comment #9)
> 
> Enterprise principals are not supported in a IPA-AD trust scenario, but one
> can 
> work around that by using:      
>     subdomain_inherit = ldap_user_principal
>     ldap_user_principal = nosuchattr
> and thus tricking sssd into 'deriving' the UPN from the domain name.

THX for your answer, but I have called ipa-client-install and this script has configured sssd. So I must ask: In which sections of sssd.conf must I put this tricky definitions?

Comment 11 Jakub Hrozek 2016-02-23 09:20:56 UTC
(In reply to Silvio Wanka from comment #10)
> (In reply to Lukas Slebodnik from comment #9)
> > 
> > Enterprise principals are not supported in a IPA-AD trust scenario, but one
> > can 
> > work around that by using:      
> >     subdomain_inherit = ldap_user_principal
> >     ldap_user_principal = nosuchattr
> > and thus tricking sssd into 'deriving' the UPN from the domain name.
> 
> THX for your answer, but I have called ipa-client-install and this script
> has configured sssd.

Yes, but we don't handle additional UPN suffixes by default well yet.

> So I must ask: In which sections of sssd.conf must I
> put this tricky definitions?

The domain section.

Comment 12 Silvio Wanka 2016-02-23 10:29:26 UTC
Added to domain section, sssd restarted but does not work. Still "Access denied". An upgrade to 7.2 is no option and if I look into the web there is the same problem.  And I have also tried with an account which does not have an enterprise principal.

But if I look into the manual of sssd.conf then "subdomain_inherit" is an General service configuration not a domain options. And ldap_user_principal is a sssd-ldap option, so this is ok in domain section. In wich service section must I put "subdomain_inherit = ldap_user_principal"? NSS, SSH or what?

And BTW if I change this, what will be happens with IPA domauin users?

Comment 13 Jakub Hrozek 2016-02-23 11:09:45 UTC
(In reply to Silvio Wanka from comment #12)
> Added to domain section, sssd restarted but does not work. Still "Access
> denied".

Then I guess another set of logs is due.

> An upgrade to 7.2 is no option and if I look into the web there is
> the same problem.

Please note you can only upgrade the sssd packages.

>  And I have also tried with an account which does not have
> an enterprise principal.
> 
> But if I look into the manual of sssd.conf then "subdomain_inherit" is an
> General service configuration not a domain options.

Yeah, this is wrong, I'll fix it in the man page.

>  And ldap_user_principal
> is a sssd-ldap option, so this is ok in domain section. In wich service
> section must I put "subdomain_inherit = ldap_user_principal"? NSS, SSH or
> what?
> 

The domain section.

> And BTW if I change this, what will be happens with IPA domauin users?

SSSD would infer the UPN from that domain's realm, so the IPA users should still work.

Comment 14 Silvio Wanka 2016-02-23 11:18:20 UTC
(In reply to Jakub Hrozek from comment #13)
> 
> Then I guess another set of logs is due.

OK, then please specify which debug level should be set in which section.

TIA

Comment 15 Lukas Slebodnik 2016-02-23 11:22:27 UTC
(In reply to Silvio Wanka from comment #14)
> (In reply to Jakub Hrozek from comment #13)
> > 
> > Then I guess another set of logs is due.
> 
> OK, then please specify which debug level should be set in which section.
> 
> TIA

debug_level = 0xfff0
in domain section should be enough. Then reproduce bug and attach sssd_$domain.log and *_child.log

Please ensure you have sssd-1.13 on testing machine.

Comment 16 Silvio Wanka 2016-02-23 16:09:55 UTC
Created attachment 1129836 [details]
the requested new logs

debug_level = 0xfff0 in domain section
sssd_$domain.log and *_child.log

Comment 17 Lukas Slebodnik 2016-02-23 16:30:48 UTC
I can see in log files that ldap_user_principal has value nosuchattr
but "userPrincipalName" attribute is still downloaded for AD user.

(Tue Feb 23 16:30:37 2016) [sssd[be[ipa.company.com]]] [sdap_get_map] (0x0400): Option ldap_user_principal has value nosuchattr

(Tue Feb 23 16:32:19 2016) [sssd[be[ipa.company.com]]] [get_extra_attrs] (0x4000): Extra attribute [objectSIDString].
(Tue Feb 23 16:32:19 2016) [sssd[be[ipa.company.com]]] [get_extra_attrs] (0x4000): Extra attribute [userPrincipalName].
(Tue Feb 23 16:32:19 2016) [sssd[be[ipa.company.com]]] [get_extra_attrs] (0x4000): Extra attribute [adUserAccountControl].
(Tue Feb 23 16:32:19 2016) [sssd[be[ipa.company.com]]] [get_extra_attrs] (0x4000): Extra attribute [originalDN].

that's weird.

Comment 18 Jakub Hrozek 2016-02-23 16:44:23 UTC
And I don't see the inherit options being used at all.

1) What is the exact output of rpm -q sssd on both client and server
2) Can we see the sssd.conf files from both client and server?

Comment 19 Sumit Bose 2016-02-24 08:22:54 UTC
The logs are from a client 'Option ipa_server_mode is FALSE'. IPA clients get the attributes of AD users and groups from the IPA server. The options from comment #9 (subdomain_inherit = ldap_user_principal and ldap_user_principal = nosuchattr) must be applied on the IPA servers. This will prevent that the IPA servers will read the userPrincipal attribute from AD and as a consequence it cannot be delivered to the IPA clients.

Comment 20 Silvio Wanka 2016-02-24 16:02:16 UTC
Created attachment 1130249 [details]
sssd.conf from both servers and client

I have now removed (subdomain_inherit = ldap_user_principal and ldap_user_principal = nosuchattr) on the client and added on both idm servers. SSSD restarted on both servers and after that on the client. Logon still not possible with ad-user+password on the client but now on both IdM servers.

rpm -q sssd:

idm-srv1: sssd-1.13.0-40.el7_2.1.x86_64
idm-srv2: sssd-1.13.0-40.el7_2.1.x86_64
idm-client: sssd-1.13.0-40.el7_2.1.x86_64

So must I again add the options also on the client or must I delete the sssd cache or which logs with which debug level will be needed now. On the IdM servers I do not really need ad-user logins ;-).

Comment 21 Sumit Bose 2016-02-24 16:41:43 UTC
Yes, the cache in the client must be removed or at least invalidated because it might still contain the non-working principal. Please try 'sss_cache -E' on the client first and if this does not help remove the cache with rm.

Comment 22 Silvio Wanka 2016-02-25 11:21:32 UTC
Neither 'sss_cache -E' nor 'systemctl stop sssd;rm -f /var/lib/sss/db/*;systemctl start sssd' solves the problem. On the IPA client still not possible to logon as AD user using password.

Only FYI:
In meantime I have changed the ssd.conf on the servers (domain section):

subdomain_inherit = ldap_user_principal, ignore_group_members
ldap_user_principal = nosuchattr
ignore_group_members = True

because it takes often to long time if I try "id ad-user@ad-comain" and sometimes it hung on the client but during the second try there it should be already in the client cache. Terrible. Now it is speedy enough.

Comment 23 Sumit Bose 2016-02-25 11:52:18 UTC
Can you send new logs from the client which include an authentication request?

Comment 24 Silvio Wanka 2016-02-25 12:31:24 UTC
Created attachment 1130530 [details]
Re: Comment #23 from Sumit Bose

Comment 25 Sumit Bose 2016-02-25 12:47:27 UTC
Do you have

  dns_lookup_kdc = true

set in the [libdefaults] section of /etc/krb5.conf on the client?

Does

  dig SRV _kerberos._udp.AD-COMPANY.COM

return a list of DCs from AD-COMPANY.COM?

Comment 26 Silvio Wanka 2016-02-25 14:40:40 UTC
Hi,

dig SRV _kerberos._udp.AD-COMPANY.COM works, this was already tested for creating the trust and the client is in the same network and uses the IdM servers as DNS server in resolv.conf.

First I have rebooted the client because "id admin" returns "no such user" :-(
Now this works again. I have now compared two IdM clients. Both are deployed using the same template which is based on 7.1. After deployment ipa-client was installed and the same call of "ipa-client-install" was done. But the client which I have used for testing was 2 days ago updated to 7.2. I'm really irritated because the krb5.conf is totally different although nobody has changed these files manually. On my 7.2 test machine:

        #File modified by ipa-client-install

        includedir /var/lib/sss/pubconf/krb5.include.d/

        [libdefaults]
          default_realm = IPA.COMPANY.COM
          dns_lookup_realm = false
          dns_lookup_kdc = false
          rdns = false
          ticket_lifetime = 24h
          forwardable = yes
          udp_preference_limit = 0
          default_ccache_name = KEYRING:persistent:%{uid}

        [realms]
          IPA.COMPANY.COM = {
            kdc = dedcs1-idm01.ipa.company.com:88
            master_kdc = dedcs1-idm01.ipa.company.com:88
            admin_server = dedcs1-idm01.ipa.company.com:749
            default_domain = ipa.company.com
            pkinit_anchors = FILE:/etc/ipa/ca.crt
          }

        [domain_realm]
          .ipa.company.com = IPA.COMPANY.COM
          ipa.company.com = IPA.COMPANY.COM

on the other (7.1) client it looks so:

        #File modified by ipa-client-install

        includedir /var/lib/sss/pubconf/krb5.include.d/

        [libdefaults]
          default_realm = IPA.COMPANY.COM
          dns_lookup_realm = true
          dns_lookup_kdc = true
          rdns = false
          ticket_lifetime = 24h
          forwardable = yes
          udp_preference_limit = 0
          default_ccache_name = KEYRING:persistent:%{uid}

        [realms]
          IPA.COMPANY.COM = {
            pkinit_anchors = FILE:/etc/ipa/ca.crt
          }

        [domain_realm]
          .ipa.company.com = IPA.COMPANY.COM
          ipa.company.com = IPA.COMPANY.COM

On this client it is possible to logon using ad-user & password. So does the ipa-client upgrade from 4.1 to 4.2 which is part of the upgrade from RH 7.1 to 7.2 change the krb5.conf wrongly? The files under /var/lib/sss/pubconf/krb5.include.d/ are the same except that domain_realm_<ipa-domain> under 7.2 includes this additional section but only after the last reboot:

[capaths]
AD-COMPANY.COM = {
  IPA.COMPANY.COM = AD-COMPANY.COM
}
IPA.COMPANY.COM = {
  AD-COMPANY.COM = AD-COMPANY.COM
}

Would help an ipa-client-install --uninstall and after that to call ipa-client-install again? I can of course copy the working krb5.conf to the other server but still not clear why this was happens. The next upgrade must be done with saved krb5.conf.

Comment 27 Sumit Bose 2016-02-25 16:17:21 UTC
During updates of the ipa-client package /etc/krb5.conf should only be touched if it does not contain the include directive. The include directive will be added to the start of the file but otherwise it will not be changed.

The '[capaths]' section in the include file is expected after the upgrade to 7.2, it will be created by SSSD.

An uninstall-install cycle should fix /etc/krb5.conf. Please note that you might need the --force-join option when running install for a second time.

The krb5.conf with 'dns_lookup_kdc = false' looks like ipa-client-install was either run with the --force option or there where some DNS issues during installation and you entered a server name directly. If you still have the logs you might want to check /var/log/ipaclient-install.log for details.

Comment 28 Silvio Wanka 2016-02-26 11:46:40 UTC
THX!!! Now it works.

Comment 29 Sumit Bose 2016-02-26 11:59:48 UTC
Great. I'll close the ticket. Fell free to open it again if the issue shows up again.