+++ This bug was initially created as a clone of Bug #1839805 +++

Description of problem: Yesterday's sssd upgrade [1] regressed authentication against AD. This was spotted by our fedora-testing image refresh in Cockpit [2], unfortunately it came a day too late to block the bodhi update.

Version-Release number of selected component (if applicable):

sssd-client-2.3.0-1.fc31.x86_64
krb5-libs-1.17-46.fc31.x86_64
adcli-0.8.2-7.fc31.x86_64

How reproducible: Always


Steps to Reproduce:
1. Join an AD domain (we test against Samba):

# realm discover
cockpit.lan
  type: kerberos
  realm-name: COCKPIT.LAN
  domain-name: cockpit.lan
  configured: no
  server-software: active-directory
  client-software: sssd
  required-package: oddjob
  required-package: oddjob-mkhomedir
  required-package: sssd
  required-package: adcli
  required-package: samba-common-tools

# realm join 
[... succeeds]

2. Validate domain user resolution:

# id Administrator
uid=766000500(administrator) gid=766000513(domain users) groups=766000513(domain users),766000519(enterprise admins),766000520(group policy creator owners),766000518(schema admins),766000572(denied rodc password replication group),766000512(domain admins)

3. Try to use it for authentication:

# ssh -l Administrator localhost
Administrator@localhost's password: 
Connection closed by ::1 port 22


Journal output during ssh:

Mai 25 11:21:30 x0.cockpit.lan sshd[9242]: pam_sss(sshd:auth): authentication success; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1 user=Administrator
Mai 25 11:21:30 x0.cockpit.lan audit[9242]: USER_AUTH pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_succeed_if,pam_succeed_if,pam_sss acct="Administrator" exe="/usr/sbin/sshd" hostname=::1 addr=::1 terminal=ssh res=success'
Mai 25 11:21:30 x0.cockpit.lan kernel: audit: type=1100 audit(1590420090.634:523): pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_succeed_if,pam_succeed_if,pam_sss acct="Administrator" exe="/usr/sbin/sshd" hostname=::1 addr=::1 terminal=ssh res=success'
Mai 25 11:21:31 x0.cockpit.lan sshd[9242]: pam_sss(sshd:account): Access denied for user Administrator: 4 (System error)
Mai 25 11:21:31 x0.cockpit.lan sshd[9242]: Failed password for Administrator from ::1 port 48764 ssh2
Mai 25 11:21:31 x0.cockpit.lan sshd[9242]: fatal: Access denied for user Administrator by PAM account configuration [preauth]
Mai 25 11:21:31 x0.cockpit.lan kernel: audit: type=1101 audit(1590420091.564:524): pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=? acct="Administrator" exe="/usr/sbin/sshd" hostname=::1 addr=::1 terminal=ssh res=failed'
Mai 25 11:21:31 x0.cockpit.lan audit[9242]: USER_ACCT pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=? acct="Administrator" exe="/usr/sbin/sshd" hostname=::1 addr=::1 terminal=ssh res=failed'
Mai 25 11:21:31 x0.cockpit.lan audit[9242]: CRYPTO_KEY_USER pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:fd:9e:7c:fc:58:c0:cf:77:3f:8a:ff:1f:40:68:c0:e1:c5:d1:a9:68:ae:49:a4:65:29:4b:c2:14:84:12:7a:11 direction=? spid=9243 suid=74  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
Mai 25 11:21:31 x0.cockpit.lan kernel: audit: type=2404 audit(1590420091.565:525): pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:fd:9e:7c:fc:58:c0:cf:77:3f:8a:ff:1f:40:68:c0:e1:c5:d1:a9:68:ae:49:a4:65:29:4b:c2:14:84:12:7a:11 direction=? spid=9243 suid=74  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
Mai 25 11:21:31 x0.cockpit.lan audit[9242]: CRYPTO_KEY_USER pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=session fp=? direction=both spid=9243 suid=74 rport=48764 laddr=::1 lport=22  exe="/usr/sbin/sshd" hostname=? addr=::1 terminal=? res=success'
Mai 25 11:21:31 x0.cockpit.lan kernel: audit: type=2404 audit(1590420091.565:526): pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=session fp=? direction=both spid=9243 suid=74 rport=48764 laddr=::1 lport=22  exe="/usr/sbin/sshd" hostname=? addr=::1 terminal=? res=success'
Mai 25 11:21:31 x0.cockpit.lan audit[9242]: CRYPTO_KEY_USER pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:fd:9e:7c:fc:58:c0:cf:77:3f:8a:ff:1f:40:68:c0:e1:c5:d1:a9:68:ae:49:a4:65:29:4b:c2:14:84:12:7a:11 direction=? spid=9242 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
Mai 25 11:21:31 x0.cockpit.lan kernel: audit: type=2404 audit(1590420091.574:527): pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:fd:9e:7c:fc:58:c0:cf:77:3f:8a:ff:1f:40:68:c0:e1:c5:d1:a9:68:ae:49:a4:65:29:4b:c2:14:84:12:7a:11 direction=? spid=9242 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
Mai 25 11:21:31 x0.cockpit.lan kernel: audit: type=1112 audit(1590420091.574:528): pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="Administrator" exe="/usr/sbin/sshd" hostname=? addr=::1 terminal=ssh res=failed'
Mai 25 11:21:31 x0.cockpit.lan audit[9242]: USER_LOGIN pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="Administrator" exe="/usr/sbin/sshd" hostname=? addr=::1 terminal=ssh res=failed'

I tried "dnf downgrade sssd-ad", but that's just causing sssd.service to fail (due to a version check in /var). But on last week's image with sssd 2.2.3 this works fine.


[1] https://bodhi.fedoraproject.org/updates/FEDORA-2020-803e9e021c
[2] https://github.com/cockpit-project/bots/pull/893

--- Additional comment from Alexander Bokovoy on 2020-05-25 15:39:19 UTC ---

Make sure your AD environment is configured to use AES encryption types for users. System-wide crypto policy in Fedora 32 and above disables RC4-HMAC.

--- Additional comment from Martin Pitt on 2020-05-25 15:52:59 UTC ---

@Alexander: So should this be reassigned to realmd then? I'd expect `realm join` to work. Besides, this is Fedora 31, does the changed crypto policy apply to that as well?

--- Additional comment from Alexander Bokovoy on 2020-05-25 16:05:18 UTC ---

To say anything concrete, we need to see SSSD logs.

Set 

debug_level = 9

in sssd.conf

for [domain/*], [nss], and [pam] sections, restart SSSD and collect /var/log/sssd directory.

Alternatively, you might want to call 'sssctl debug-level 9' before the test if none of applications in the test restarts sssd, this setting will not survive restart.

To pick up sssd logs, you can also do

sssctl logs-fetch /path/to/file.tar

and then attach the file.

--- Additional comment from Martin Pitt on 2020-05-25 18:00:17 UTC ---

Thanks Alexander for the hints -- I keep forgetting that sssd doesn't log to the journal. The debug logs are quite a mouthful, so I'm not sure which useful extracts to provide. I naïvely grepped for

# grep -ri rc4.*hmac /var/log/sssd/
/var/log/sssd/krb5_child.log:(2020-05-25 13:55:27): [krb5_child[9501]] [sss_child_krb5_trace_cb] (0x4000): [9501] 1590429327.118054: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, rc4-hmac, camellia128-cts, camellia256-cts

but that doesn't seem to be very useful.

--- Additional comment from Martin Pitt on 2020-05-25 18:10:01 UTC ---

> Make sure your AD environment is configured to use AES encryption types for users

Do you happen to have some links for that? A Google search isn't very fruitful, and https://wiki.samba.org/index.php/Samba_Security_Documentation seems to suggest that it's supported already.

I checked the entire /etc in the Samba 4.3.11 container for `grep -ri aes` and `grep -ri rc4`, that is the only hit:

krb5kdc/kdc.conf:        supported_enctypes = aes256-cts:normal arcfour-hmac:normal des3-hmac-sha1:normal des-cbc-crc:normal des:normal des:v4 des:norealm des:onlyrealm des:afs3

--- Additional comment from Alexander Bokovoy on 2020-05-25 18:21:08 UTC ---

there is a failure in retrieving GPOs.

in PAM log:
(2020-05-25 13:55:27): [pam] [pam_dp_send_req] (0x0100): Sending request with the following data:
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): command: SSS_PAM_ACCT_MGMT
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): domain: cockpit.lan
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): user: Administrator
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): service: sshd
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): tty: ssh
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): ruser: not set
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): rhost: ::1
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): authtok type: 0
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): newauthtok type: 0
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): priv: 1
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): cli_pid: 9499
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): logon name: Administrator
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): flags: 0
(2020-05-25 13:55:27): [pam] [pam_dom_forwarder] (0x0100): pam_dp_send_req returned 0
(2020-05-25 13:55:28): [pam] [sbus_dispatch] (0x4000): Dispatching.
(2020-05-25 13:55:28): [pam] [pam_dp_send_req_done] (0x0200): received: [4 (System error)][cockpit.lan]
(2020-05-25 13:55:28): [pam] [pam_reply] (0x4000): pam_reply initially called with result [4]: System error. this result might be changed during processing
(2020-05-25 13:55:28): [pam] [filter_responses] (0x0100): [pam_response_filter] not available, not fatal.
(2020-05-25 13:55:28): [pam] [pam_reply] (0x0200): blen: 28
(2020-05-25 13:55:28): [pam] [pam_reply] (0x0200): Returning [4]: System error to the client

from domain log:
(2020-05-25 13:55:28): [be[cockpit.lan]] [read_pipe_handler] (0x0400): EOF received, client finished
(2020-05-25 13:55:28): [be[cockpit.lan]] [gpo_cse_done] (0x0020): ad_gpo_parse_gpo_child_response failed: [22][Invalid argument]
(2020-05-25 13:55:28): [be[cockpit.lan]] [ad_gpo_cse_done] (0x0400): gpo_guid: {31B2F340-016D-11D2-945F-00C04FB984F9}
(2020-05-25 13:55:28): [be[cockpit.lan]] [ad_gpo_cse_done] (0x0040): Unable to retrieve policy data: [22](Invalid argument}
(2020-05-25 13:55:28): [be[cockpit.lan]] [ad_gpo_access_done] (0x0040): GPO-based access control failed.

from GPO child log:
(2020-05-25 13:55:27): [gpo_child[9502]] [main] (0x0400): performing smb operations
(2020-05-25 13:55:27): [gpo_child[9502]] [copy_smb_file_to_gpo_cache] (0x0400): smb_uri: smb://f0.cockpit.lan/sysvol/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/GPT.INI
(2020-05-25 13:55:28): [gpo_child[9502]] [copy_smb_file_to_gpo_cache] (0x4000): smb_buflen: 20
(2020-05-25 13:55:28): [gpo_child[9502]] [prepare_gpo_cache] (0x4000): smb_path_with_suffix: /cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/GPT.INI
(2020-05-25 13:55:28): [gpo_child[9502]] [prepare_gpo_cache] (0x0400): Storing GPOs in /var/lib/sss/gpo_cache/cockpit.lan
(2020-05-25 13:55:28): [gpo_child[9502]] [prepare_gpo_cache] (0x0400): Storing GPOs in /var/lib/sss/gpo_cache/cockpit.lan/Policies
(2020-05-25 13:55:28): [gpo_child[9502]] [prepare_gpo_cache] (0x0400): Storing GPOs in /var/lib/sss/gpo_cache/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}
(2020-05-25 13:55:28): [gpo_child[9502]] [unique_filename_destructor] (0x2000): Unlinking [/var/lib/sss/gpo_cache/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/GPT.INIDQTKDJ]
(2020-05-25 13:55:28): [gpo_child[9502]] [unlink_dbg] (0x2000): File already removed: [/var/lib/sss/gpo_cache/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/GPT.INIDQTKDJ]
(2020-05-25 13:55:28): [gpo_child[9502]] [ad_gpo_parse_ini_file] (0x0400): ini_filename:/var/lib/sss/gpo_cache/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/GPT.INI
(2020-05-25 13:55:28): [gpo_child[9502]] [perform_smb_operations] (0x0400): sysvol_gpt_version: 0
(2020-05-25 13:55:28): [gpo_child[9502]] [copy_smb_file_to_gpo_cache] (0x0400): smb_uri: smb://f0.cockpit.lan/sysvol/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/Machine/Microsoft/Windows NT/SecEdit/GptTmpl.inf
(2020-05-25 13:55:28): [gpo_child[9502]] [copy_smb_file_to_gpo_cache] (0x0400): smb_uri: smb://f0.cockpit.lan/sysvol/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/MACHINE/Microsoft/Windows NT/SecEdit/GptTmpl.inf
(2020-05-25 13:55:28): [gpo_child[9502]] [copy_smb_file_to_gpo_cache] (0x0020): smbc_getFunctionOpen failed [2][No such file or directory]
(2020-05-25 13:55:28): [gpo_child[9502]] [perform_smb_operations] (0x0020): copy_smb_file_to_gpo_cache failed [2][No such file or directory]
(2020-05-25 13:55:28): [gpo_child[9502]] [main] (0x0020): perform_smb_operations failed.[2][No such file or directory].
(2020-05-25 13:55:28): [gpo_child[9502]] [main] (0x0020): gpo_child failed!


So, basically, libsmbclient-based utility in SSSD cannot talk to your AD DC over SMB protocol for specific files (smb://f0.cockpit.lan/sysvol/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/MACHINE/Microsoft/Windows NT/SecEdit/GptTmpl.inf) but can do that for others (smb://f0.cockpit.lan/sysvol/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/GPT.INI). Note that SSSD tried the path twice, with /Machine/ and /MACHINE/ component replacement. This is expected, SSSD has the following comment in this place:

            /*
             * DCs may use upper case names for the main folder, where GPTs are
             * stored. libsmbclient does not allow us to request case insensitive
             * file name lookups on DCs with case sensitive file systems.
             */


This looks to me like a problem on the server side -- we have GPT.INI in the policy but no machine policy in place, thus no way to retrieve the policy. The corresponding code in SSSD did not change since 2017 aside from replacing debug string prefix to [gpo_child[%pid]].

--- Additional comment from Alexander Bokovoy on 2020-05-25 18:25:24 UTC ---

(In reply to Martin Pitt from comment #5)
> > Make sure your AD environment is configured to use AES encryption types for users
> 
> Do you happen to have some links for that? A Google search isn't very
> fruitful, and https://wiki.samba.org/index.php/Samba_Security_Documentation
> seems to suggest that it's supported already.
> 
> I checked the entire /etc in the Samba 4.3.11 container for `grep -ri aes`
> and `grep -ri rc4`, that is the only hit:
> 
> krb5kdc/kdc.conf:        supported_enctypes = aes256-cts:normal
> arcfour-hmac:normal des3-hmac-sha1:normal des-cbc-crc:normal des:normal
> des:v4 des:norealm des:onlyrealm des:afs3

No, my comment is unrelated. Samba DC should be able to prefer AES encryption types. I thought you are using Microsoft Windows Server.

--- Additional comment from Nicolas Pöhlmann on 2020-05-26 00:22:50 UTC ---

We run into this bug also on Fedora 32 in V2.3.0, so it's not only restricted to F31.

For a temporary fix, you can downgrade to V2.2.3 with deleting the sss-DB files and rejoining the machine to AD.

Overall it's harder to debug the exact case on F32 right now, because of the AD 'dot' username problem in systemd V245.4 [see: https://ask.fedoraproject.org/t/gui-login-with-sssd-managed-users-fails/6606]. So for a full functional F32 AD integration every one needs sssd V2.2.3 and systemd V245.5 from rawhide which I think isn't the best circumstance to post any log here. But it's the same PAM error on F32.

--- Additional comment from Sumit Bose on 2020-05-26 05:23:59 UTC ---

Hi,

given the failure in the GPO lookup the issue is most probably related to the changes added by https://pagure.io/SSSD/sssd/issue/3324.

As a workaround you can set 'ad_gpo_access_control' to 'disabled' or 'permissive' in the [domain/...] section of sssd.conf to skip the GPO based access control.

Martin, would it be possible to attach SSSD logs from an older version as well so that I can compare what is the effect of the changes in your environment.

bye,
Sumit

--- Additional comment from Martin Pitt on 2020-05-26 05:49:49 UTC ---

(In reply to Sumit Bose from comment #9)

> As a workaround you can set 'ad_gpo_access_control' to 'disabled' or
> 'permissive' in the [domain/...] section of sssd.conf to skip the GPO based
> access control.

That works very nicely indeed, thanks!

> Martin, would it be possible to attach SSSD logs from an older version as
> well so that I can compare what is the effect of the changes in your
> environment.

Gladly -- attached.

Thanks!

--- Additional comment from Sumit Bose on 2020-05-26 11:08:14 UTC ---

Hi,

thanks for the logs. In the old version the GPO is considered as 'not applicable' and hence SSSD does not try to download it. I the new version, due to changes from https://pagure.io/SSSD/sssd/issue/3324, it now is considered applicable and while trying to load the related data SSSD fails.

I wonder if you can send me some raw data as well. First the LDAP object:

    kinit -k 'X0$@COCKPIT.LAN'
    ldapsearch -H ldap://f0.cockpit.lan -Y GSSAPI -b 'CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit,DC=lan'

and second the sysvol content:

    smbclient -k '\\f0.cockpit.lan\sysvol'
    smb: \> ls cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\
    smb: \> ls cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\Machine\

I would expect that the Machine directory is empty, if not, please follow the path as long as possible.

Thanks.

bye,
Sumit

--- Additional comment from Martin Pitt on 2020-05-26 11:34:48 UTC ---

@Sumit: If you prefer, I'm happy to give you ssh access to a running instance that has everything set up. Just send me your public SSH key.

# ldapsearch -H ldap://f0.cockpit.lan -Y GSSAPI -b 'CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit,DC=lan'
SASL/GSSAPI authentication started
SASL username: X0$@COCKPIT.LAN
SASL SSF: 256
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit,DC=lan> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# {31B2F340-016D-11D2-945F-00C04FB984F9}, Policies, System, cockpit.lan
dn: CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit
 ,DC=lan
objectClass: top
objectClass: container
objectClass: groupPolicyContainer
cn: {31B2F340-016D-11D2-945F-00C04FB984F9}
instanceType: 4
whenCreated: 20200526112825.0Z
whenChanged: 20200526112825.0Z
displayName: Default Domain Policy
uSNCreated: 3585
uSNChanged: 3585
showInAdvancedViewOnly: TRUE
name: {31B2F340-016D-11D2-945F-00C04FB984F9}
objectGUID:: fVb/256RAUCk0sih9OPuig==
flags: 0
versionNumber: 0
systemFlags: -1946157056
objectCategory: CN=Group-Policy-Container,CN=Schema,CN=Configuration,DC=cockpi
 t,DC=lan
isCriticalSystemObject: TRUE
gPCFunctionalityVersion: 2
gPCFileSysPath: \\cockpit.lan\sysvol\cockpit.lan\Policies\{31B2F340-016D-11D2-
 945F-00C04FB984F9}
gPCMachineExtensionNames: [{35378EAC-683F-11D2-A89A-00C04FBBCFA2}{53D6AB1B-248
 8-11D1-A28C-00C04FB94F17}][{827D319E-6EAC-11D2-A4EA-00C04F79F83A}{803E14A0-B4
 FB-11D0-A0D0-00A0C90F574B}][{B1BE8D72-6EAC-11D2-A4EA-00C04F79F83A}{53D6AB1B-2
 488-11D1-A28C-00C04FB94F17}]
gPCUserExtensionNames: [{3060E8D0-7020-11D2-842D-00C04FA372D4}{3060E8CE-7020-1
 1D2-842D-00C04FA372D4}][{35378EAC-683F-11D2-A89A-00C04FBBCFA2}{0F6B957E-509E-
 11D1-A7CC-0000F87571E3}]
distinguishedName: CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=Sy
 stem,DC=cockpit,DC=lan

# User, {31B2F340-016D-11D2-945F-00C04FB984F9}, Policies, System, cockpit.lan
dn: CN=User,CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC
 =cockpit,DC=lan
objectClass: top
objectClass: container
cn: User
instanceType: 4
whenCreated: 20200526112825.0Z
whenChanged: 20200526112825.0Z
uSNCreated: 3586
uSNChanged: 3586
showInAdvancedViewOnly: TRUE
name: User
objectGUID:: BOTZGXz8RU6WORiszMkG+A==
systemFlags: -1946157056
objectCategory: CN=Container,CN=Schema,CN=Configuration,DC=cockpit,DC=lan
isCriticalSystemObject: TRUE
distinguishedName: CN=User,CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Polici
 es,CN=System,DC=cockpit,DC=lan

# Machine, {31B2F340-016D-11D2-945F-00C04FB984F9}, Policies, System, cockpit.la
 n
dn: CN=Machine,CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System
 ,DC=cockpit,DC=lan
objectClass: top
objectClass: container
cn: Machine
instanceType: 4
whenCreated: 20200526112825.0Z
whenChanged: 20200526112825.0Z
uSNCreated: 3587
uSNChanged: 3587
showInAdvancedViewOnly: TRUE
name: Machine
objectGUID:: GOx13irBnEKIjwCn6yaNOg==
systemFlags: -1946157056
objectCategory: CN=Container,CN=Schema,CN=Configuration,DC=cockpit,DC=lan
isCriticalSystemObject: TRUE
distinguishedName: CN=Machine,CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Pol
 icies,CN=System,DC=cockpit,DC=lan

# search result
search: 4
result: 0 Success

# numResponses: 4
# numEntries: 3


smb: \> ls cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\
NT_STATUS_OBJECT_NAME_INVALID listing \cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\

smb: \> ls cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\Machine\
NT_STATUS_OBJECT_NAME_INVALID listing \cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\Machine\

That didn't seem expected, so I went upwards instead:

smb: \> ls cockpit.lan\Policies\
NT_STATUS_OBJECT_NAME_INVALID listing \cockpit.lan\Policies\
smb: \> ls cockpit.lan\Policies
  Policies                            D        0  Tue May 26 07:28:26 2020

		10258432 blocks of size 1024. 4190900 blocks available

It seems to work better with a "*" at the end (NB I don't really know what I'm doing here):

smb: \> ls cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\*
  .                                   D        0  Tue May 26 07:28:26 2020
  ..                                  D        0  Tue May 26 07:28:26 2020
  GPT.INI                             N       20  Tue May 26 07:28:26 2020
  MACHINE                             D        0  Tue May 26 07:28:26 2020
  USER                                D        0  Tue May 26 07:28:26 2020

		10258432 blocks of size 1024. 4190968 blocks available
smb: \> ls cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\Machine\*
  .                                   D        0  Tue May 26 07:28:26 2020
  ..                                  D        0  Tue May 26 07:28:26 2020

		10258432 blocks of size 1024. 4190968 blocks available

So Machine/ dir is indeed empty.

--- Additional comment from Sumit Bose on 2020-05-26 12:53:33 UTC ---

Hi,

thanks for the offer to access you system directly. If you don't mind I'd prefer to continue the ping-pong here because the steps recorded here might help others to debug GPO related issues as well. But please let me know if you prefer otherwise.

I forgot that the SIDs are not added automatically to the ldapsearch output, can you re-run ldapsearch with

    kinit -k 'X0$@COCKPIT.LAN'
    ldapsearch -H ldap://f0.cockpit.lan -Y GSSAPI -b 'CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit,DC=lan' cn flags gPCFunctionalityVersion gPCFileSysPath gPCFileSysPath nTSecurityDescriptor

Thanks.

bye,
Sumit

--- Additional comment from Martin Pitt on 2020-05-26 13:47:22 UTC ---

# ldapsearch -H ldap://f0.cockpit.lan -Y GSSAPI -b 'CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit,DC=lan' cn flags gPCFunctionalityVersion gPCFileSysPath gPCFileSysPath nTSecurityDescriptor
SASL/GSSAPI authentication started
SASL username: X0$@COCKPIT.LAN
SASL SSF: 256
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit,DC=lan> with scope subtree
# filter: (objectclass=*)
# requesting: cn flags gPCFunctionalityVersion gPCFileSysPath gPCFileSysPath nTSecurityDescriptor 
#

# {31B2F340-016D-11D2-945F-00C04FB984F9}, Policies, System, cockpit.lan
dn: CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit
 ,DC=lan
cn: {31B2F340-016D-11D2-945F-00C04FB984F9}
flags: 0
gPCFunctionalityVersion: 2
gPCFileSysPath: \\cockpit.lan\sysvol\cockpit.lan\Policies\{31B2F340-016D-11D2-
 945F-00C04FB984F9}

# User, {31B2F340-016D-11D2-945F-00C04FB984F9}, Policies, System, cockpit.lan
dn: CN=User,CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC
 =cockpit,DC=lan
cn: User

# Machine, {31B2F340-016D-11D2-945F-00C04FB984F9}, Policies, System, cockpit.la
 n
dn: CN=Machine,CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System
 ,DC=cockpit,DC=lan
cn: Machine

# search result
search: 4
result: 0 Success

# numResponses: 4
# numEntries: 3

wrt. remote debugging, that makes sense -- I'm happy either way, I just know from myself that I prefer local/direct debugging.

Thanks!

--- Additional comment from Andrew Gunnerson on 2020-05-27 21:11:20 UTC ---



--- Additional comment from Sumit Bose on 2020-05-29 16:47:24 UTC ---

Hi,

thanks for providing access to a test system.

The important items are that a Samba DC was used here and that at least in the initial configuration of the Samba DC the GPO related files for the default domain policy {31B2F340-016D-11D2-945F-00C04FB984F9} are empty.

I'm currently discussing with Samba developers what would be the best way to solve this. However, since in this setup the GPO are empty and as a result using them for access control will give ot benefit using 'ad_gpo_access_control = disabled' is not only a workaround but makes sense.

I keep the ticket open in case the result of the discussion is that SSSD has to deal with it. In this case we have to add code to detect this case and handle it more gracefully.

bye,
Sumit

--- Additional comment from Martin Pitt on 2020-05-30 05:52:16 UTC ---

Thanks Sumit for your debugging efforts!

Naively it feels like manually setting ad_gpo_access_control=disabled on the client side is a hack/workaround. We shouldn't have "unbreak my computer" options to think about. Of course we could add that to our test suite, but we want our testing to represent reality, not being arbitrarily tweaked to work.

The empty machines policy on the server side sound like a more reasonable root cause -- either this is a valid config -- and sssd then should ignore an empty policy; or it is not a valid config, then we should fix our server configuration, and we can file an issue against https://github.com/Fmstrat/samba-domain/blob/master/init.sh to do the right thing by default. That samba setup script doesn't deal with any policy stuff, so at least the default config in Samba 4.3.8 seems wrong. Possibly this is fixed in newer versions, I'm happy to give that a try.

I didn't find some good docs yet how to manage these GPOs -- I found https://wiki.samba.org/index.php/Managing_local_groups_on_domain_members_via_GPO_restricted_groups and https://wiki.samba.org/index.php/Implementing_System_Policies_with_Samba but they both talk about interactively editing the policies with a Windows client. That's not an option for our automated image builds, but hopefully it can be done on the CLI somehow. Does anyone happen to have a pointer here?

Or would it help to entirely remove the Machine\ directory, so that no GPOs are applied? 

Thanks!

--- Additional comment from Sumit Bose on 2020-06-02 09:22:48 UTC ---

(In reply to Martin Pitt from comment #17)
> Thanks Sumit for your debugging efforts!
> 
> Naively it feels like manually setting ad_gpo_access_control=disabled on the
> client side is a hack/workaround. We shouldn't have "unbreak my computer"
> options to think about. Of course we could add that to our test suite, but
> we want our testing to represent reality, not being arbitrarily tweaked to
> work.
> 
> The empty machines policy on the server side sound like a more reasonable
> root cause -- either this is a valid config -- and sssd then should ignore
> an empty policy; or it is not a valid config, then we should fix our server
> configuration, and we can file an issue against

Hi,

since this is the default Samba DC configuration and I would expect the Windows clients do not have an issue with the missing file, SSSD should at least ignore the missing GPO file for the default domain policy in this case. 


> https://github.com/Fmstrat/samba-domain/blob/master/init.sh to do the right
> thing by default. That samba setup script doesn't deal with any policy
> stuff, so at least the default config in Samba 4.3.8 seems wrong. Possibly
> this is fixed in newer versions, I'm happy to give that a try.
> 
> I didn't find some good docs yet how to manage these GPOs -- I found
> https://wiki.samba.org/index.php/
> Managing_local_groups_on_domain_members_via_GPO_restricted_groups and
> https://wiki.samba.org/index.php/Implementing_System_Policies_with_Samba but
> they both talk about interactively editing the policies with a Windows
> client. That's not an option for our automated image builds, but hopefully
> it can be done on the CLI somehow. Does anyone happen to have a pointer here?
> 

Yes, the afaik the currently expected way is to use gpmc on a Windows client to create the GPO file.

David Mulder from S.u.S.E. pointed out that you can create the file Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\MACHINE\Microsoft\Windows NT\SecEdit\GptTmpl.inf
manually with the content:

-------- snip --------
[Unicode]
Unicode=yes
[Version]
signature="$CHICAGO$"
Revision=1
-------- snip --------

which represents just an empty policy file with a valid header. The UUID/GUID 31B2.... is always the same and represents the default domain policy.


> Or would it help to entirely remove the Machine\ directory, so that no GPOs
> are applied?

No, this won't work, you have to modify or remove entries in the Samba DC LDAP tree but I think creating the file as described above would be easier.

HTH

bye,
Sumit

> 
> Thanks!

--- Additional comment from Martin Pitt on 2020-06-02 14:24:59 UTC ---

Amazing, thanks Sumit! Creating that file with specified  contents works great. That's a nice workaround for the time being, and better than skipping the tests.

--- Additional comment from Charles Butterfield on 2020-06-03 18:02:49 UTC ---

I'm having the same problem on Fedora-32.  Just upgraded to sssd 2.3.0-1.fc32 and my AD authentication stopped working.  The workaround described above also works for me, namely

/etc/sssd/sssd.conf
[domain WHATEVER]
ad_gpo_access_control=disabled

--- Additional comment from Sumit Bose on 2020-06-04 05:29:51 UTC ---

(In reply to Charles Butterfield from comment #20)
> I'm having the same problem on Fedora-32.  Just upgraded to sssd
> 2.3.0-1.fc32 and my AD authentication stopped working.  The workaround
> described above also works for me, namely
> 
> /etc/sssd/sssd.conf
> [domain WHATEVER]
> ad_gpo_access_control=disabled

Hi,

I've seen that you commented on https://bugzilla.redhat.com/show_bug.cgi?id=1840908 as well. Are you using Samba as a domain controller or a Windows AD DC. In the Samba case this ticket here is the right on in the other case you most probably see the issue described in https://bugzilla.redhat.com/show_bug.cgi?id=1840908.

bye,
Sumit